IT Management

The subject for this column developed on its own after I returned from a long weekend filled with numerous scattered thunderstorms and predictions for a difficult Monday, weather-wise.

As soon as I walked into our offices, one of the IT technicians, with a very sad look on his face, informed me that the power distribution to the building had been fluctuating wildly for more than two hours. Short-term outages and voltage variances were easily handled by the data center UPS, and most desktops had smaller desk-side units.

The IT tech then asked me, “But do you remember that temporary NAS unit … ?”

As soon as I heard the “But,” I knew it was going to be a bad day.

A few months earlier, we installed a small RAID-5 NAS unit for use during a data migration project. It was designated for non-critical, temporary data. Upon project completion, it was supposed to be decommissioned but somehow became a handy repository for an unrelated project. Its use grew, and before long, this ad hoc repository contained important data used by several groups.

But this device wasn’t protected by any IT resources normally used for critical data protection. And it was this very resource that was now dormant on the shelf, with the blinking “LED of death” the only activity.

The diagnosis was grim: a loss of the RAID-5 configuration and nothing available from the manufacturers for the recovery of user data. Worse was the determination that the most recent backup was two months old, and several critical files had been placed on the device in the interim.

If the lack of a UPS caused the snowball to begin rolling down this proverbial snow-covered hill, the lack of any backup for the missing data certainly allowed that snowball to blossom into a full-fledged avalanche.

We sent the unit to a data recovery specialist, and $7,000 later, we had all the missing data recovered and peace once again reigned throughout the land. There’s a lesson to be learned from this mistake; hence, my decision to make it the subject of this column. 

System security means so much more than firewalls, strong passwords, and intrusion detection appliances. It also extends to physical security, environmental protections, disaster recovery, training, and more. We violated our own rules by failing to protect an asset. In our case, the intrinsic value of the storage unit was significantly higher than expected due to the unexpected data stored within. Regardless of company size, you must protect all assets, at all times. For us, a $200 portable UPS would have saved us from a seven-day down time and the cost for specialized data recovery services. Most of our technology is protected behind a robust data center environment, but ad hoc projects and good intentions sometimes go awry and expose the enterprise to risk.

It’s all about risks. System security is all about identifying, managing, and mitigating them. You should be evaluating your environment by addressing all risk factors and following some authoritative process. The Special Publications series of NIST documents are a great starting point, regardless of whether or not you’re commercial or federal. They’re available at www.nist.gov (start with SP 800-30 and SP 800-53).

The NAS was implemented as a temporary tool but unwittingly became an ad hoc production device. Originally, the security categorization was “Low” for confidentiality, “Low” for availability, and “Low” for integrity, so a lack of supporting resources was justifiable. But as soon as it started to contain critical production data, the ratings should have been increased to “Moderate,” if not arguably, “High.”

Based on this new, higher categorization, the asset should have been treated differently. It shouldn’t have been placed into a general population area (environmental protections such as power, temp, theft/damage, fire); strike one. It should have received frequent backups (operational protections for disaster recovery); strike two. And finally, the change in use should have been communicated to management for review and approval (management protections for planning, coordination and control); strike three.

Think about it. If it was this easy for an experienced company to make a costly mistake, then it’s safe to speculate that some cross-section of all data centers are at risk with unknown exposed assets. What can you afford to lose?