Have you ever seen the YouTube videos of the soda and candy experiment, where someone drops several pieces of a certain mint-flavored candy into a bottle of diet carbonated soda? Suddenly, you see mass amounts of soda erupting and exploding in every direction. You might say that, over the past several years, data has exploded in a similar way in terms of volume, types, and content. In addition, there’s a greater demand to access data quickly.
But how do you manage the mass volumes of data that keep growing? The established methodologies for managing data and databases probably aren’t going to be effective in the future. Further, data plays a new role in today’s business, and the IT department needs to successfully manage to that new role by quickly and effectively creating information from the data. One example of turning data into information is an application that reconfigures data to effectively display it on a mobile device. Another is a business intelligence solution that mines and analyzes stored data. So how do you take that data in its raw form and process, arrange, and configure it so you can present it in a way that provides information or knowledge?
Five Data Management Challenges
With the skyrocketing volume of data, companies are facing major challenges to remain competitive and continue to meet customer demands. The following examines five such challenges and how you can tackle them:
1. Extract, Transform, and Load (ETL): ETL goes hand-in-hand with application modernization. It’s all about companies providing services in a simplified way to their customers—modernizing legacy applications, the user interfaces, and the middleware that ties them all together from an enterprisewide perspective. But sometimes it isn’t enough to merely modernize the applications. Sometimes the data itself needs to be transformed into a more appropriate format and then reloaded for the updated application to use.
If your company requires ETL, you don’t need to build it yourself. A variety of ETL solutions exist, but be sure to select one that meets your unique business requirements. For example, if you’re considering moving some applications to the cloud, ETL is key to ensuring that move is effective. In many cases, you will be looking at large amounts of data that need to be extracted, transformed, and quickly reloaded into the cloud. So make sure the ETL vendor is able to support you in this regard. Moreover, performance is particularly critical for many applications in any virtualized or non-virtualized environment. If it takes an hour to extract, transform, and load a terabyte of data, this may be too long to meet your business requirements.
Look for an ETL solution that’s robust, efficient, and scalable. Even if you have only a small amount of data to manage today, keep in mind that the volume of data is continually growing. Be sure to select a solution that can meet your future needs.
2. Data integration: This involves allowing data from multiple sources, multiple data stores, and multiple types of databases to be translated into a common data structure or architecture. This makes it easier to write and access the data and turn it into information. Data doesn’t always exist in the same form.
For example, consider a scenario where you’ve just refinanced your mortgage with the financial institution where you hold a checking account. You may, in fact, receive a new offer to refinance the week after you sign off on your new mortgage. This could happen if this financial institution has separate databases for general customers and for mortgage customers. If the data were integrated, the company could save the time and effort of mailing mortgage refinance offers to customers who’ve just refinanced. Sending these offers out just after someone has refinanced can lower the bank’s credibility and confuse the customer. It can also cost money.
Data integrated just for the sake of integrating it isn’t very useful. Be sure to understand the business requirements for integrating data. Also, determine if the data is brought together logically into a federated database, pulled into a virtual database, or brought together physically. To a large extent, effective integration requires recognition of the key structures that can be used to bring the data together in an effective, useful fashion—and it’s absolutely essential to have an understanding of the data content. This content is critical to creating useful information.