Callout: Batch processing is such an integral part of business operations that you should approach modernization in phases and with proper planning.
Batch processing has been a core part of business and IT processes for decades. Across industries, businesses depend on the successful execution of their batch or bulk workloads. For example, banks must reconcile all the day’s financial activities before a new business day can begin. Insurance companies must sift through terabytes of data to quantify the risk (and therefore price) of insurance policies. Credit card companies must apply interest calculations to hundreds of thousands of accounts at the end of each billing period. Financial institutions must calculate credit scores, adjust billing rates, deliver statements, and generate reports. Retail stores and manufacturing companies need to do efficient inventory management. Without these activities, the respective businesses couldn’t function. Batch processing is uniquely oriented to efficiently carry out such tasks by providing economies of scale; this makes it a fundamental, mission-critical component of the business and its information technology infrastructure.
Why Modernize Batch Processing?
In the current business environment, with trends of globalization and an “always-on” customer who can be anywhere, the luxury of having a dedicated window for “batch processing” is diminishing. To achieve business agility with efficiency, the notion of real-time and “just in time” access to goods and information in the context of a growing focus on operational efficiency can only be fully realized if we’re able to get an optimized, balanced, concurrent execution of online (single) and bulk (batch) processing throughout the entire day. This is critical in containing costs through consolidation and reducing complexity to enable globally integrated enterprise transformation.
We introduce the concept of “near-real-time response” as a reasonable amount of time the customer is willing to wait for an acceptable response while we process data in bulk. It could be five seconds to 24 hours (or more), but that’s determined by a business contract rather than a technology constraint. We see batch windows becoming more elastic—stretching to run in non-stop mode concurrently with Online Transaction Processing (OLTP) systems—but still limited by the tolerances of the OLTP applications to resolve their contention with shared resources. Batch or bulk processing now starts to take the form shown in Figure 1.
Webster’s Dictionary defines modern as “involving recent techniques, methods, or ideas.” To summarize, the motivation to modernize batch processing is a reaction to several pressures business and IT managers face today:
- Cost: Process more data in batch in shrinking time windows with shared infrastructure, services, and resources to manage cost
- Scale: Handle larger-than-expected volumes because of growth in customers, products and channels, while “throwing more hardware” isn’t viable
- Skills: The pressure to focus development skills around a limited set of languages and tools. Java becomes a focus due to its increasing prevalence.
- Flexibility and reuse: The need to adopt Service-Oriented Architecture (SOA) concepts to gain the benefits of modular reuse and rapid change.
Batch modernization isn’t merely moving batch to cheaper hardware or simply using Java rather than COBOL. It’s a requirement that encompasses the entire information processing environment. That includes considerations about development tooling, integration of traditional batch with newer Java batch processes, and the capabilities of the batch run-times under consideration.
Approaching Batch Modernization
Batch processing is such an integral part of business operations that you should approach modernization in phases and with proper planning. Modernization strategies that suggest an all-at-once approach will carry a higher risk. To achieve the maximum benefits of the target state, there are potential changes in the entire spectrum—from business process modeling to infrastructure. The entry point could vary, depending on your current pain points and existing environment, but here are some best practices to consider: