Acts of God and other fun things
Thursday august 23rd, 2007, at approximately 11 pm, our backup power generator was unable to cope with a lightning strike to our data center and shut itself down. A technician was dispatched from the generator manufacturer, CAT, and was on site in 15 minutes after the outage. Battery power allowed our servers to keep working, however, the air conditioning units were disabled until generator power could be restored. After approximately 20 minutes of battery power a decision was made to bring down the equipment in order to avoid heat damage. Approximately at the same time as the network was being brought down, two mobile power generators capable of producing 1 Megawatt each, arrived on site on flatbed trucks.
After the two mobile generators started to feed the air conditioners and our main building generator kicked in, all servers were turned up to full service and were scanned for array problems. All together, most customers experienced about 30 minutes of outage.
In the last 24 hours, the Chicago area experienced floods, severe thunderstorms, and a tornado. Trees are broken everywhere and one of our staff lost his car to a flood. All things considered, we are very lucky that the damage was not any worse.
To head off any questions, yes we do have backup batteries, yes we do test the generator every month, and yes, we are prepared and have survived times this type of situation many times. We apologize for the outage and will be happy to speak with you on an individual basis about how this effected your service and what we can do to help.
Artur
August 24, 2007 