Far be it from me to quote Scripture, but as I write this column, the images of the devastation wrought by Frankenstorm Sandy are still filling the 24-hour cable news and they look like something out of those ’50s Bible epics.
Undoubtedly, by the time this piece goes to press, there will be an official calculation of the financial costs of the event, and most certainly they will be extraordinary, setting new records. This is actually a pretty easy prediction to make, given the high real estate values in New Jersey, New York, and Connecticut prior to the storm, and the hourly cost of downtime for organizations that have entered that dystopian reality reminiscent of “Mad Max” in the days and weeks—and perhaps months—following Sandy.
I find my thoughts returning to Sunday school lessons and to the verses from Matthew 7:24 - 27: “Therefore everyone who hears these words of mine and puts them into practice is like a wise man who built his house on the rock. (25) The rain came down, the streams rose, and the winds blew and beat against that house; yet it did not fall, because it had its foundation on the rock. (26) But everyone who hears these words of mine and does not put them into practice is like a foolish man who built his house on sand. (27) The rain came down, the streams rose, and the winds blew and beat against that house, and it fell with a great crash.”
Aside from the religious meaning, there’s quite a bit of in-your-face practical wisdom in those verses. Obviously, building on a solid foundation is a proactive way to ensure your ability to weather calamity; building on sand, on the other hand, is foolhardy, shortsighted, and ultimately inviting of disaster. Even if the authors of the Good Book weren’t teaching civil engineering or architecture—let alone, information systems architecture and design—there are some important takeaways for practitioners of these arts.
First, we need to learn more about sand and rock and what these are in the context of information technology. Sandy probably showed a lot of companies they were building their IT services on sand. For one thing, the proximity of data centers for production operations and for recovery was a problem for many firms. Granted, the low likelihood of “once-in-a-hundred-years” events might have encouraged some firms to place their data center assets (including cloud services, by the way) within 50 kilometers of each other without too much concern about outages that impact both the primary and redundant sites. However, only about a year before Sandy, the area had another close encounter with a once-in-a-hundred-years weather event with a broad geographical footprint that should have called into question the efficacy of data center redundancy strategies. In fact, the work on the potential impact of a dirty bomb following the 9-11 attacks should have seen a lot of companies moving their secondary sites further away from the NY-NJ-CT metroplex. To do otherwise was to build your DR strategy on sand.
Of course, it will likely emerge that some of the problems run much deeper than the separation distance of data centers. In many cases, firms have stopped doing DR planning altogether. PriceWaterhouseCooper’s study last year showed that the number of firms worldwide undertaking DR had dropped to 39 percent, down from 44 percent in 2011, and well below the 50 percent rule of thumb we were all quoting prior to the recession. Part of the reason for the decline, of course, is the economy—hard budgetary decisions and all that.
Part of the decline, however, is due to marketing woo from the x86 server virtualization crowd, and maybe the “virtual-tape-library-with-deduplication” vendors: In case you haven’t heard, DR planning and tape backup have been done away with by high availability strategies levering “vmotioning” of workload from rig to rig and WAN-based disk-to-disk data replication. (The latter is what’s really forcing firms to keep their redundancy assets within 50 kms, by the way.)
The “DR/tape backup is dead” noise is overwhelming and eminently believable until that one-in-a-hundred-years event occurs that reveals it for the sand it’s built upon. We’ve now had two such events in about as many years and the climate change scientists are warning that we can expect more. Maybe 2013 is when we rediscover the importance and value of building on rock-solid technology.