Apr 6 ’11
CICS and the Open Transaction Environment
When IBM announced support for running application code in the Open Transaction Environment (OTE), among the selling points was a significant reduction in CPU utilization in CICS-DB2 applications. Unfortunately, the possibility of reducing an application’s CPU requirements by 30 percent, without any coding changes, diverted attention from the OTE’s many other benefits. This article discusses those benefits and related OTE issues.
The OTE and threadsafe provide a significant architectural change to CICS. For the first time, CICS provides a true multi-processing environment running application code under multiple Task Control Blocks (TCBs) within the same CICS region.
Problems Caused by CICS Multi-Processing
In a standard CICS environment, all CICS application code runs under a single CICS TCB known as the Quasi-Reentrant (QR) TCB. (While all application code runs under the QR TCB, CICS maintains several specialty TCBs for use by CICS internal services.) Single-threading all user code lets CICS provide direct access to shared storage areas without requiring serialization. From the time CICS dispatches a task until the time a task issues an EXEC CICS request, the task is the sole owner of the entire CICS address space. For example, program PROG1 could safely read and increment a counter in the Common Work Area (CWA) because no other instances of PROG1 could be running concurrently.
Such single-threading doesn’t exist in the OTE. The CICS dispatcher initiates simultaneous tasks on multiple OTE TCBs whenever possible. The PROG1 program can no longer safely read or increment its CWA counter because multiple instances of PROG1 might also be reading and incrementing the counter simultaneously. PROG1 must be modified to use some manual serialization technique to replace the QR TCB serialization. Even when some user programs are enabled to run on an open TCB, the QR TCB continues to run single-threaded. This doesn’t eliminate shared storage serialization concerns for programs remaining on the QR TCB; programs running on open TCBs may access the shared storage simultaneously as a program on the QR TCB. Serialization must be maintained at the storage area level, not at the program level.
A program that serializes all access to shared storage, or doesn’t rely on any area of shared storage remaining consistent between machine instructions, is described as being threadsafe. Only threadsafe programs run safely in a multi-tasking environment.
Limiting all user code to the QR TCB has many disadvantages, including:
- Restricts use of OS system calls in user code. When code runs on the QR TCB, it controls the CICS region completely. Any OS call that causes a wait will halt the entire region; OS calls that alter the TCB state will alter the entire region state.
- Restricts all user code to run serially on one processor. A single TCB can’t run any faster than the speed of a single processor. In today’s multi-processor environment, a CICS QR TCB is limited to only a small percentage of the processing capacity of the Logical Partition (LPAR) on which it runs. This can result in the seemingly paradoxical event of a production CICS running slower after being moved to a faster machine. If the new machine has a larger total MIPS capacity due to more, but slower, processors, CICS’ one-processor limit will reduce the CICS region’s CPU capabilities.
- CPU-intensive tasks block out the CICS dispatcher. When a user program is dispatched on the QR TCB, it controls the region. After passing control to the user program, the CICS dispatcher doesn’t receive control until the user program issues an EXEC CICS command. While CICS will receive control at every EXEC CICS command, not all commands activate CICS’ dispatching process. Some commands, such as ASKTIME, are processed so quickly that CICS bypasses the dispatcher and returns control directly to the calling program. In this case, a CPU-intensive program will continue to block the dispatcher until it issues an EXEC CICS command that returns control to the dispatcher.
- If the program is CPU-intensive, the next EXEC CICS command won’t be issued for an unusually long time, resulting in what appears to be a regionwide slowdown.
The OTE resolves these problems by accommodating separation of user code from the QR TCB. When a program runs in the OTE:
- The limitation on issuing OS system calls is lifted because any resulting wait only affects the open TCB on which the task is running. The only documented restriction on OS activity by a program running in the OTE is that it must not attempt to initialize the batch Language Environment (LE).
- The inability to exploit multiple processors is removed; z/OS will dispatch open TCBs on any available processor. CICS no longer runs slower on a faster machine.
- CPU-intensive tasks now block only the open TCB they’re running on, leaving all other open TCBs (and the QR TCB) available for useful work, eliminating the slowdown.
Open TCB Types
There are several different open TCB types, depending on the type of processing occurring. Although the J-series of open TCBs is used to run Java programs, this article is only interested in the L-series.
The L type open TCB evolved from the DB2-specific TCB CICS used to offload DB2 processing from the QR TCB. Each CICS transaction accessing DB2 would have one of these TCBs assigned to it. When the application program issued a DB2 call, CICS would spin the task over to its assigned DB2 TCB to run the DB2 system code, then spin it back to the QR TCB when DB2 system processing completed. A complex CICS/DB2 transaction could issue several thousand DB2 calls, resulting in twice as many TCB spins, which added up to a significant amount of CPU time spent swapping TCBs. IBM realized that the number of TCB swaps required could be significantly reduced if the application could simply remain on the DB2 TCB after DB2 processing was completed. The open TCB is the result.
As first introduced, the open TCBs were named L8 TCBs and were accessed by calling a CICS Task Related User Exit (TRUE) initialized with the OPENAPI parameter. Although the “8” refers to the CICS system storage key 8, tasks defined as user key (key 9) remain in user key when running on an L8 TCB.
IBM later enhanced open TCB access by allowing application programs to be defined as OPENAPI. Any program defined as OPENAPI will be initialized on an L-series TCB, and all application code is guaranteed to run on the instance of the L-series TCB on which the program was initialized. Programs defined as OPENAPI and running in USER key will run on an L9 TCB; CICS key programs will run on an L8 TCB.
Accessing the OTE
There are two methods to run application code in the OTE: Define an OPENAPI TRUE and create a TRUE program and stub, or modify the program’s API attribute in its RDO PROGRAM definition to OPENAPI.
You can define an OPENAPI TRUE and create a TRUE program and stub. Because it’s the CALL itself that results in a switch to the L8 TCB, the TRUE program need not actually have any code associated with it other than a BR 14 (see Figure 1). If a program that’s defined as threadsafe issues a call to the OPENAPI TRUE, CICS will move it to an L8 TCB, and it will remain on that L8 TCB until the program either terminates or issues a non-threadsafe EXEC CICS command. Not all EXEC CICS commands are threadsafe. Consult the Application or Systems Programming Guide for a list of commands that are threadsafe in your release.
If a program has some code that should run in the open environment, and some that must run on the QR TCB (e.g., for serialization purposes), a second dummy TRUE can be established that’s initialized without the OPENAPI parm. A call to the non-OPENAPI TRUE will cause CICS to move the task to the QR TCB. In this way, the programmer can control which environment the task is in at any point in processing.
Unfortunately, there’s no way to tell at run-time if your task is currently running on an open TCB. In a complex program, the programmer may not be able to tell what the state of the program will be at a specific paragraph; he would have to issue “just in case” TRUE calls to ensure the proper processing environment.
Any program using a dummy TRUE to access the OTE should be carefully documented; procedures should be applied to ensure that subsequent maintenance doesn’t inadvertently result in an unsuitable TCB switch. For example, a non-threadsafe command inserted between the call to an OPENAPI TRUE and the application code that must run in an open environment would result in this code running on the QR TCB and could de-stabilize the region.
Besides the application coding changes to call the TRUE, and building the TRUE and stub programs, control programs must be created to activate and de-activate the TRUE.
The second option to run application code in the OTE is to modify the program’s API attribute in its RDO PROGRAM definition to OPENAPI. A threadsafe program that’s defined as OPENAPI will be initialized and run on an open TCB—an L8 TCB for CICS key programs, or an L9 TCB for USER key programs—without having to issue any TRUE calls. In this manner, a program can take advantage of the OTE with just a simple RDO change; no system changes or application code changes are required. There’s no way to run part of the code in an OPENAPI program under the QR TCB; if the program issues a non-threadsafe command, CICS returns it to its open TCB before giving control back to the program. Unlike using the dummy TRUE, there’s no concern regarding which environment the program is in at any point because all the application code will run on the open TCB.
There are some performance concerns to consider before defining a program as OPENAPI. First, all OPENAPI programs incur TCB spin overhead whenever a non-threadsafe CICS command is issued. CICS must spin the task from the open TCB to the QR TCB to process the command, but then must immediately spin the task back to its open TCB as soon as the task is completed. If a large number of non-threadsafe commands are issued, CPU utilization can increase dramatically.
Another performance concern is related to OPENAPI programs that run in USER key. USER key OPENAPI programs run on an L9 TCB, but all OPENAPI TRUE activity occurs on an L8 TCB. If an OPENAPI USER key program (L9) issues a DB2 call (L8), it will hold both the L8 and the L9 TCB instances until task termination. You should review TCB limits to ensure a sufficient number of TCBs will be available to meet these requirements; an insufficient number will result in tasks waiting on TCB availability and can produce a deadly embrace. If a task requires an L-series TCB, and the only available L-series TCBs are in the wrong key, CICS will “steal” an available TCB, tear it down, and rebuild it in the needed key, incurring additional overhead. USER key tasks running on an L8 TCB via OPENAPI TRUE call aren’t subject to this overhead. CICS will run USER key application code in key 9 on an L8 series TCB, if called via OPENAPI TRUE.
IBM has been actively enhancing its CICS-based products to take advantage of the OTE through the use of OPENAPI TRUEs (which use L8 TCBs), so this performance issue extends beyond DB2 and covers products such as WebSphere MQ, CICS sockets, and the XML parser. If OPENAPI USER key programs are installed, be careful when upgrading IBM system software to ensure that no unanticipated L8-L9 TCB conflicts arise.
Law of Unintended Consequences and OTE
While the OTE provides tremendous new opportunities to extend and enhance CICS transaction capabilities, you should consider the possible negative side effects of any change. For example, it’s common to find shops running critical CICS regions that are CPU-constrained on the QR TCB. In such situations, even a short delay by z/OS in re-dispatching the QR can result in a significant backup in workload, so the CICS in question is set to an unusually high dispatching priority. This CICS region would receive considerable benefit from the multiple CPU exploitation achieved when running user programs in the OTE. Transaction response time would be improved, obviously, by reducing wait on QR dispatch. However, consider the case of such a situation on a production processor running two Central Processors (CPs). Currently, CICS is taking 60 to 80 percent of one processor, leaving 20 to 40 percent of that processor and 100 percent of the second processor available for everything else. Once application workload moves to the OTE, this scenario changes drastically. In addition to the QR, there are now multiple L8 TCBs competing for available processor, all of which are set to run at a higher dispatching priority than anything else on the system. Non-DB2-related open TCBs always run at the same dispatching priority as the QR.
With the QR CPU bottleneck removed, total CICS CPU utilization can easily increase to 100 to 150 percent of a processor. With such a reduction in CPU available for non-CICS work, the high-priority system services that CICS requires are now crowded out by application tasks running on higher-priority open TCBs. Transaction response time consequently increases. This is avoidable if anticipated and with an adjustment to CICS’ dispatching priority to a reasonable level. However, it’s difficult to anticipate because it’s assumed that CICS CPU utilization can’t exceed 100 percent of a single processor.
Similarly, the OTE supports use of z/OS storage services (GETMAIN, etc.). Because z/OS GETMAIN offers significant advantages over CICS GETMAIN (supporting use of subpools, for example), it’s tempting to consider converting CICS programs to use them. The unanticipated consequence here is that CICS is totally unaware of any z/OS storage your transaction acquires. It isn’t reflected in the SMF 110 performance statistics, doesn’t show in a CICS monitor, isn’t collected in a CICS transaction dump, and must be manually freed at task termination. Moreover, the GETMAIN and FREEMAIN activity doesn’t appear in the CICS trace table. Drawbacks such as these must be considered before implementing programs using non-CICS services production.
Running batch COBOL programs in the OTE also raises LE run-time issues that you should review before implementation. When running in a CICS address space, LE uses its CICS support routines to process requests:
- COBOL DISPLAY statements will be dynamically converted to EXEC CICS WRITEQ TD commands. WRITEQ TD isn’t a threadsafe command.
- Dynamic calls are converted to use CICS services to load the called program and initialize the LE environment.
- Although use of native I/O commands is supported in the OTE, LE doesn’t support COBOL OPEN/CLOSE in CICS.
CICS OTE is a powerful tool that eliminates or minimizes some of the most significant limitations of the CICS run-time environment; however, it’s also a double-edged sword with the potential to cause serious performance problems or unanticipated run-time errors if used improperly. Careful consideration of these issues is critical to a successful implementation of any OTE project.