It’s easy for today’s mainframers to understand that Business Intelligence (BI) is a hot topic; the most important thing for them to also understand is that the emphasis is on the BI, not the mainframe. To deliver the best organizational value-add, IT needs to fit the mainframe effectively into an overall strategy for leveraging the new types of BI, not do everything on the mainframe. Fortunately, the three most important new trends in BI—analytics, Big Data, and agile BI—are areas in which the mainframe is well-equipped to play a key supporting role in implementation/operation.
So let’s consider each of these three new trends, then suggest some ways the mainframe can drive these initiatives forward and support the CIO as well as the CEO. Yes, it’s that important.
The old BI focused primarily on regularly scheduled enterprise reporting, plus data mining, done in the IT department, using data combined in data warehouses and data marts as information sources. Analytics is the new BI, in which deeper, ad hoc, and/or near-real-time analyses and alerting occur on data inside and outside the data warehouse, or where these actions are embedded in business software inside and outside IT. Analytics may mean the CFO doing more in-depth analysis of activity-cost allocation from last quarter, or the Chief Marketing Officer (CMO) doing what-if scenarios and customer-of-one strategy testing, or software checking the enterprise network to detect usage patterns and react semi-automatically before performance slows too much.
With analytics, the BI emphasis for IT shifts from carrying out tasks delegated by the rest of the business (particularly corporate) to supporting corporate and lines of business in carrying out strategic tasks. Embedded analytics also means administrators have to “up their game” to learn the business meaning and impact of the new information about enterprise computing. For mainframers, it isn’t just a matter of applying the mainframe’s superior power to a new application; the need is to change the mainframe administrator’s skillset and reach out to corporate to ensure that the mainframe (usually together with other platforms) is providing the necessary support.
The key focus of the mainframer, in all likelihood, should be to ensure the data warehouse keeps pace with new demands. For most large enterprises (and, eventually, for most small and medium-sized businesses now in the early stages of implementing analytics), the data warehouse is a necessity. It provides the most in-depth analysis of the largest amount of pre-cleaned, pre-categorized data. Getting the right mix between the fast analysis “on the edge” and the deeper analysis in data warehouse BI is vital to achieving a competitive advantage, according to recent IBM surveys of CEOs and CMOs. The mainframer should know and proactively support the software tools for such things as master data management, near-real-time querying across operational or outside-the-data center and data-warehouse data sources, and embedded enterprise-architecture or private-cloud analytics.
While vendors tend to talk about Big Data as a mammoth amount of any non-structured data, the new BI trend is to deliver intelligence on Web-oriented, unstructured data (Facebook videos and photos) plus semi-structured data (Twitter tweets) because, increasingly, the enterprise must satisfy the consumer who lives in social media. However, that data typically is super-massive, lives in public clouds, and is often “dirty” or intermittently unavailable.
There’s a common misconception—often fed by vendors, alas—that a handy-dandy software tool will let you dump all this data into a data warehouse where you can apply your near-real-time analytics capabilities to deliver “business as usual” BI. This is an inherently risky approach. Currently, there’s simply no way for any tool to deliver enterprise-quality public cloud Big Data in amounts far greater than what the data warehouse has faced so far, in a consistently near-real-time fashion. There’s also no way for the data warehouse alone to deliver analytics performance on that Big Data comparable to that presently available for enterprise data. The mainframer who tries to do this faces a no-win situation.
A feasible approach may be for the mainframer to support something like a “triple virtual data warehouse.” Here, “virtual warehouse” software in the public cloud will query social media data in real-time via Hadoop (onsite, where possible) and download only the results to the rest of the triple warehouse. A second “staging” virtual warehouse will accept massive amounts of near-real-time dirty data, allowing querying of some of it across the existing and staging data stores. “Clean” new Big Data, plus data cleansed by the staging warehouse, can now be passed as “older” but still valuable data to the existing data warehouse. Note that the mainframe is also a reasonable solution for running the separate staging data warehouse, and especially given its ability to load balance virtual machines within and across systems. Note also, however, that, like analytics, this approach will require querying across systems, architectures (public cloud and enterprise), and physical data warehouses, and therefore will need cross-database querying or metadata management tools and processes.
The concept of agile BI is part value-add and part pure vendor hype; it’s important for the mainframer to understand which is which. As practiced in the real world, agile BI is the practice of using programmers who are following an agile development process (e.g., constant “corrective steering” by interaction with a user and emphasis on rapid implementation of a usable portion of the new solution) to deliver new analytics capabilities rapidly. Vendor hype would suggest, however, that agile BI is using the new analytics tools and Big Data to react instantaneously and correctly to changes in the market, which they call business agility. That’s incorrect.
The most important thing for the mainframer to understand here is the danger of believing the hype. The biggest bang for the enterprise’s buck accrues from increased New Product Development (NPD) agility. This is proactive; it leads as well as quickly tracks the changing market—and not only from faster reaction after the product is out there. Moreover, focusing on after-the-fact analytics often leads to increasing your investment in a fundamentally obsolete business process rather than upgrading or abandoning it. The mainframer should resist the temptation to put a nice red bow on one’s Big Data implementation and call it business agility, and instead focus on the blocking and tackling of applying agile BI programmers to the most strategic analytics projects—such as those involving improving your NPD or changing your business processes fundamentally. Specific steps in the development area might include looking at Scrum, Thoughtworks, and CollabNet rather than just Micro Focus and REXX.
Mainframe Action Items
The message of the new BI trends for the mainframer is easily summarized:
• Shift your focus away from BI that handles delegated corporate tasks and toward BI that supports the rest of the company.
• Support careful integration of existing mainframe data warehouses with new Big Data sources in the public cloud by such measures as global metadata management, the triple virtual data warehouse, and data federation.
• Learn the new agile BI programming processes, especially the part about linking better with corporate users, and integrate them with mainframe ones.
These are baby steps. That’s good. In handling the new BI trends, the mainframer wins by not losing. Avoid being a boat anchor that clings to the old data warehouse. Avoid wasting IT money by helping chase after poorly planned, badly implemented new analytics capabilities. By not falling into such traps, the mainframer allows the platform to take its proper share of the responsibility for the BI that’s at or near the top of the CEO’s list of strategic initiatives linked to enterprise success or failure. Yes. It’s that important.