For more than 20 years, mission-critical data processing has relied on the automated initiation, execution, management, integration, and recovery of batch-mode IT processing—scripts, jobs, tasks, and other non-interactive IT processes that manipulate a specific data set. These processes may be application-oriented, such as accounting, payments, supplier management, purchasing, ordering, fulfillment or data mining; or system-oriented, such as data backup, defragmentation, migration, export, transfer, or load operations. Typically, they process multiple data items (accounts, orders, transactions, database records, etc.) and return results as a summary or detail report. This is in contrast to real-time or online operations, which will typically process one item at a time before returning a single result to a specific online user.
Such processing has many different names, including production management, process automation, workload management, process workflow, and batch execution. However, most people with mainframe experience will know it as job scheduling, and 20 years on, it’s more important than ever. While it has a pedigree longer than most technologies, this isn’t the same old job scheduling of 1980. It’s packed with new features such as Web services Application Program Interfaces (APIs), service-based scheduling, cross-platform and multi-tier compatibility, complex event triggering, and messaging interfaces. It’s being used as a foundation technology for e-commerce, supply chain management, grid and cluster support, and virtualization implementations.
So what’s driving the continued importance of job scheduling? Of course, enterprises are always looking to lower the cost of doing business, and scheduled processing for batch jobs lets a single user process a large volume of like transactions with minimal interaction, accomplishing more work in a day than with interactive, online processing. Job scheduling lets multiple processes execute unattended, reducing the cost of managing data center operations such as submitting these processes, monitoring them, checking that they complete normally, etc.
Business processes, especially long-running but not time-critical processes, are more efficient when executed offline and when executed on high-throughput enterprise servers (such as System z). Users can submit long-running jobs and move immediately onto other work while they execute in the background. It also can reduce load on smaller systems during peak online usage, resulting in better online performance. So cost reduction and business efficiency continue to be strong drivers, as they’ve always been. However, there are a variety of relatively new drivers, too.
For example, packaged business applications, whether on mainframe, Unix, Linux or Windows, invariably need a sophisticated job scheduling infrastructure. Enterprises highlight the inability of built-in schedulers to handle the complexity of a real-life workload, such as end-of-month accounting. Problems include resolving contention between jobs and departments, sequencing jobs properly, adequately detecting and correcting scheduling and process errors, integrating with other application instances or external systems, and managing resource allocation.
For custom applications, job scheduling also plays an important role. Organizations that invest heavily in custom application development view their proprietary IT capability as a competitive differentiator. Most of this value is in unique online components, but some event-driven and timer-based processing is invariably required to manage application infrastructure and perform routine maintenance (e.g., database management, configuration tuning, data archiving, file transfer, etc.), and provide offline business processing. Integrating off-the-shelf job scheduling capabilities lets proprietary development focus on competitive features, and deliver them faster. This is especially true for the newest breed of homegrown applications, developed with newer technologies such as Enterprise JavaBeans (EJBs), Java 2
Enterprise Edition (J2EE), Microsoft .NET , Web services, etc. With these complex, multi-tiered applications, job scheduling can make good applications better, by extending their functionality without complex programming; for example, using a batch-driven Extract, Transform, and Load (ET L) process to feed financial data into a custom reporting facility, or even into more sophisticated Business Intelligence (BI) engines.
With Web services, it’s clear there’s significant opportunity for job scheduling to serve as an extension for the Enterprise Service Bus (ESB). Most vendors don’t currently support a true Service-Oriented Architecture (SO A) environment, but this is likely to change over time. There are two vectors for job scheduling in an SOA environment: as a service used by other applications, and as a service that controls other applications. Applications exposed as services could as easily be controlled in offline mode by a job scheduler, just as they’re controlled in online mode by another application or calling process.
SOA has the potential to make process scheduling more relevant, in the same way Enterprise Resource Planning (ERP) applications did. The expectation for the ES B is to deliver a real-time, dynamic, responsive transport layer for accessing a set of common application services from any online application. The reality is likely to be similar to ERPs: The initial hype and expectation will be online, but eventually will come the realization that offline, batch-mode processing is more efficient for any process that doesn’t need to return results immediately to a user at an online screen. There’ll likely be an online ES B, responding immediately to user events (mostly during online business hours), and an offline ES B, holding user events in batches until they can be processed most efficiently (e.g., overnight or during off-peak periods). At the intersection of these two vectors, job scheduling functions are exposed as services that control other common application services. Effectively, this becomes a scheduled service broker, where an application can ask a job scheduling solution to perform a process at a specific time, based on a specific event, or dependent on specific conditions. The job scheduling solution holds the request until the condition is fulfilled, then executes the service as requested. It may or may not pass results back to the calling service, or indeed to any other service, to form a services-oriented workflow.