According to technology leaders, it was recently estimated that the world creates several quintillion bytes of data every day—so much information that 90 percent of the world’s data today has been created in just the last few years. This data comes from things such as digital sensors, social media, digital pictures and videos, online purchase information, Internet text and indexing, medical records, mobile devices and online stock trading to name a few.
While definitions vary, this is typically considered Big Data. However, when organizations talk about Big Data, they aren’t just talking about the size or volume of their data. These remarkable new sources of information are being added at extraordinary rates compared to legacy, transactional data. Big Data represents new risks and challenges to process, manage and sustain usable information. This isn’t an easy thing to do using current methodologies, people and tools. Business users and IT professionals must understand the potential value and then work together in harmony to even understand what data is worth analyzing and how the project needs to be implemented.
Big Data utilization requires talented staffs with the proper knowledge and skills to identify, capture, store, manage, secure and analyze this valuable information to provide meaningful business insights. If you’ve found yourself wondering how to make Big Data systems work for your organization, then that’s good. Depending on the kind of business you’re in, there’s a strong chance you can create access to untapped, critical and revolutionary corporate information. But new thinking is required.
The breadth and depth of Big Data can be staggering. Here are just a few anecdotal examples:
• Turn terabytes of information generated in Tweets and chats to analyze customers and product issues
• Convert millions, even billions, of meter readings to analyze electrical power, natural gas and water usage for both near-term and predictive analysis
• Use previously unavailable Big Data results to confirm and substantiate “gut feelings” and create trust in what you’re seeing from overall analytics efforts
• See information in near-real-time fashion and with new granularity, something that can be particularly useful in fraud detection—a high-reward, constantly running, real-time analysis
• Exploit and interpret information hidden in photos, video (including surveillance video) and other unstructured documents
• Analyze 1 million customer transactions per hour generated by one of the world’s largest retailers, which is then imported into a database of more than 2.5 Petabytes (PB) of data (2,500 TB)
• Imagine using Big Data strategies to work with the NASA Center for Climate Simulation and their 32 PB of climate observation and simulation data.
The technologies and interest in making Big Data projects feasible and valuable are definitely on executives’ minds these days. Gartner helped add some interest to the Big Data discussion as well when they stated in October 2012 that Big Data will add more than 1.9 million jobs in the U.S. by 2015. While that seems to be likely overstated, it’s obvious something “big” is happening.
As you begin to look at possible Big Data projects, make sure your organization has the staff, or the ability to bring in outside help, to:
• Create an overall Big Data strategy, assessment and roadmap of ideas, recommendations and tasks
• Integrate all types of data to make the most of existing and planned information assets
• Update infrastructure to facilitate storing, searching, cataloging and indexing of unstructured data
• Focus on how, when and what technologies are needed to process very large data volumes
• Create a platform where intelligent decisions can be made by using advanced types of analytics made possible with Big Data.
Big Data isn’t just another “big hype.” It’s an opportunity to see business metrics in new and evolving ways based on the new ways business works—and at the volumes it operates. Systems, technology and people now exist to make this data harvest work, and the results will be fascinating.