Bad Things Can Happen…Twice
So why would I be taking time to write a post about us having redundancy and
it actually working right? Well, it's pretty simple really. In the
hosting business, there have been several incidents over the years where all of
the promised redundancies did not work as they should and customers were left
offline for extended periods. Times where transformers blew up and there
was no backup power source, generators sat with their diesel engines locked up
due to no one test firing them on a regular schedule, or UPS systems that were
overloaded and failed as other back up systems attempted to come online.
Last week, it was very satisfying to watch all of the systems we tout here at
MaximumASP work exactly as designed, and you the customer experienced absolutely
no downtime.
On Thursday, June 14th, at approximately 12:15PM EST the utility supplier to
Louisville, Kentucky experienced a significant transformer failure in a
substation that is in the same grid as the MaximumASP data center. This
failure forced other parts of the grid to attempt to carry the load of failed
section. This was too much for the remaining sections to handle and they
began to cascade into a blackout condition. As this reached the MaximumASP
data center, the UPS systems carried the full data center load as the power
bucked 3 times in a 10 second span. As the UPS carried the load, the
Automatic Transfer Switch (ATS) sensed the supply change and fired up the 2
megawatt generator. As the generator synchronized cycles with the existing
electrical load, the ATS shifted power to the generator. We remained in
this state for almost 1 full hour, until power was restored from the utility.
Less than 1 hour after power was restored by the utility, the transformer
failed again. Like clockwork, the UPS's, ATS and generator engaged, and we
switched the generator to manual mode allowing us to run it continuously until
we were confident that the problem had been completely rectified. We
actually ran the unit until 6PM EST that evening making our own power, and
allowing the diesel refueling trucks to come in and replenish the fuel tanks as
the generator was running.
All is normalized now, but I thought that some of you might be interested in
all that went on behind the scenes on Thursday as your servers, switches, SANS,
and disk arrays hummed along without skipping a beat. It is certainly nice
to see these complex support systems work as they were designed to, and
hopefully this provides further reassurance that you made the right choice in
selecting MaximumASP as your hosting provider.
June 18th, late-afternoon