Your Worst Day
Like most of us, St. Johns Health Center in Santa Monica, Calif. has a whole file of Worst Days to draw from. Their latest happened on Labor Day weekend. St. Johns is a 224 bed medical facility on Historic Route 66. An established medical center, it is also the home of the John Wayne Cancer Institute.
At 6:15 a.m. on Sunday, September 3, 2000, the fire alarm activated and there was a great deal of smoke in the basement of the south wing where primary power distribution for the hospital is located. Immediately, the entire emergency power distribution system failed. St. Johns has a one megawatt cogeneration plant in addition to its emergency generators. The cogeneration plant dropped off line upon system fault. The life safety branch was then lost, causing loss of emergency lighting and fire alarm systems.
The problem started with the failure of a 1,200 Amp normal power breaker in the emergency transfer switchgear enclosure. When Southern California Edison restored power to the switchgear, power to the emergency equipment and life safety branches did not return. The failure and fire of the normal side also caused an open on the emergency power side because the emergency breaker was mounted above the normal breaker in the same enclosure. There was concern about resetting the emergency breaker because of heat damage, which had also blown two 1,000 Amp fuses on the life safety branch circuits.
All normal power, or White Plugs, were back in operation. However, the entire primary emergency power distribution system, including the Red Plugs, was still off. This disabled ventilation, vertical transportation and major clinical equipment such as radiology, surgery, laboratory, clinical freezers and UPS systems. Communications were running on UPS batteries and by 11:00 a.m, failed.
The damaged wiring and fuses were replaced to restore power to the life safety branch. Power could not be restored to the emergency equipment branch because of the fire-damaged breaker. Trying to remove it from the live power buss was considered too dangerous, so it was necessary to kill all emergency power. The clinical staff was notified that critical devices would have to be connected to normal circuits by extension cords or operated from battery.
When the breaker was removed, not only was it destroyed, the heat-damaged buss behind it needed to be replaced. Sunday afternoon on Labor Day weekend, the hunt started and a supplier was found who could fabricate a 1,200 Amp circuit breaker and buss.
While parts were being procured, emergency power was temporarily restored and a connection was made to the UPS for the telephone switch. The biomed staff was busy connecting other emergency equipment into normal outlets. Hard-wired emergency services such as ventilation, elevators, radiology equipment, freezers and other fixed systems were still without power. The clinical and medical staff stayed at bedsides to monitor and reassure their patients.
Emergency power circuits were opened again and the final repairs were completed at 10:30 p.m. Some circuits had been up and down three times during the day, others had been down over 15 hours. The only medical imaging available was ultrasound and mobile X-ray. St. Johns has a Siemens PACS system with Fuji CR components, which were all out of commission. Even though the mobile X-ray machines worked, they couldnt be moved from floor to floor and the CR cassettes could not be processed.
Analyzing the event, the breaker was within its PM schedule, had been recently thermoscanned without indication and was typically carrying only 700 Amps. There was no prior indication of any problems.
A hospital visitor who fell trying to jump out of an unleveled elevator car suffered the only injury.
The largest take-home lesson is the importance of communications. Keeping the staff informed was difficult. Radio repeaters were on the emergency power grid and large parts of the hospital were without coverage. Many radios were not fully charged, and most of the clinical staff did not understand frequency allocations. In the final analysis, runners were needed to deliver messages about what was happening and when things would be restored. Unchanged by 4,000 years of history and technology. Dont shoot the messenger! still applies.
This was not St. Johns Worst Worst-Day. That was January 17, 1994 when they were on a focal point of the Northridge earthquake. That closed the hospital for nine months. The entire facility is currently being rebuilt with a new campus that will be completed in 2005. The new facility is a design model for earthquake and emergency preparedness. The staff is already prepared.
We want to thank Terry A. Muldoon, Vice President of St. Johns Health Center and his staff for sharing their experience. We hope that their experience helps you prepare for your Worst Day.
C. Wayne Hibbs is a contributing editor to 24×7 and president of Hibbs & Associates, of Dallas, Texas.