The Associated Press Disaster

Or, how to kill 4,000 satellite receivers in just 1 second..
Notice the 'FAULT' Lamp ? 4,000 affiliates did !!
The Associated Press has long used KU band satellite receivers to distribute their news, spots, and other audio clips.

One of the improvements in the new receiver models is the ability to update the operating system through the satellite. In other words, the instructions that tell the receiver just what to do can be rewritten without touching the radio itself.
On August 23, 2000, one of the new programs was to be loaded into the receivers. No big deal , it had been done before without any problems.

But this time, somehow, the WRONG file was sent up to the satellite. It was dutifully beamed down to some 4,000 A.P. receivers. And it locked them up. Cold. Even unplugging them and then plugging them back in wouldn't unlock the program and turn off the fault light.
Where do you start untangling a mess like this ?
Fortunately, A.P. rose to the occasion by sending out teams of technicians that would open up the receiver, stack a printed circuit card on top of the one inside, and reprogram the receiver to work again.
The whole procedure took less than 10 minutes for each receiver.
But it took 5 days and 4,000 procedures...

The amount this must have cost A.P. can only be speculated at, but can you imagine the way the tech felt that sent up the 'death code'?


Copyright 2005 by Vital Sounds - ALL RIGHTS RESERVED