Tier-one storage requirements put a significant amount of fiscal strain on organizations as a result of government-instituted compliance regulations. Naturally, companies seek to minimize their reliance on this costly storage media, but often find that the sacrifice in performance and data availability is too great to transition the data to tier-two storage. Sarbanes-Oxley, HIPAA, and the Securities and Exchange Commission not only require companies to ensure quick and reliable access to information, but also dictate the amount of time certain data must be kept. This is heavily contributing to the amount of storage capacity needed.
Tier-one storage is built to provide the highest levels of access, security, and availability. Widespread deployment throughout an enterprise is impractical, so it’s typically reserved for mission-critical business functions such as Online Transaction Processing (OLTP), and can handle transaction numbers into the hundreds-of-thousands per second. Many CIOs and data center managers struggle with the challenge of balancing tier-one storage costs against federally regulated archival requirements.
In industries such as banking and investment, every transaction, timestamp, email, etc. must be kept. This holds true for all publicly traded companies as well. While some data will be frequently accessed for a period of time, all the information is required to be archived and available for any future needs, whether it’s for fraud investigation or portfolio history.
Typically, institutions establish proprietary algorithms to determine when data can be rolled off tier-one and onto tape. Once the data is relegated to tape, queries require offline batch processes that are costly and time-consuming, so it’s important for these institutions to find a balance between the costs of maintaining high-performance, tier-one platters and offline data access.
Finding this balance is only half the battle, unfortunately. Once data has been successfully transitioned from top-tier storage, the question becomes, “What’s going to happen to this data over the long run?” On the short end of things, some compliance regulations require data to be held for only 10 years. On the other end of the spectrum (such as medical records and patient history in the healthcare field), archival data is required to be stored and maintained for a much greater period of time. Not all storage medium in existence today has that kind of longevity, so a technology transition and maintenance plan must be implemented and followed.
Today, I still talk to companies with tapes that are 30 years old. They’ve diligently stored and protected the tapes per HIPAA standards so they can review the data if necessary for fraud, medical malpractice, trends, symptoms, and exposure. But what are the chances of ever being able to read that data? Unfortunately, it’s very low. The estimated life expectancy of some of these tapes is eight years. When taking into account the current innovation cycle, it isn’t hard to understand why companies are struggling to maintain three to four generations of tape technology to satisfy potential reporting requirements.
Additionally, the catch-22 that many people run into with data archiving when balancing tier-one loads and tier-two access is that the needs for electronic discovery of older information, such as in a lawsuit, could result in a substantial cost to gather all the data from various tape cartridges and may not be fast enough to satisfy the request. Lower-cost media means higher stakes in the compliance gamble and can potentially be more costly at a single point in time when compared to high-performance, top-tier storage media.
With so many angles to consider with regard to data archiving, and the rapid evolution of compliance requirements, it might be a good time to check your current plan to make sure you’ve optimized costs, performance, and compliance. Ask yourself: “Have my storage needs significantly increased in the past two years? Are compliance rules changing, and if so, am I able to meet them in my current environment?”
Several technology alternatives do exist to using traditional mainframe tape or optical media for data archiving, including data migration onto content-addressable storage; however, storage needs will always be dictated by federal regulatory compliance mandates. First, understand what your needs are and how they can potentially change, and then put in place a scalable process to balance these needs with your bottom-line costs.