Jan 1 ’07
CICS TS 3.1
This article provides an overview of the primary options available for CICS problem determination and diagnosis. The various techniques described can apply to both application and systems programming problems as well as to investigation of problems in CICS itself.
The trace (TR) domain handles general CICS tracing. It traces the flow of execution through CICS code and your applications. CICS can help identify the EXEC CICS Application Program Interface (API) commands and resources your applications use. It’s an essential debugging aid.
CICS supports tracing to an internal trace table, held above the 16MB line in the CICS address space, and an auxiliary trace destination. Auxiliary trace is written to a CICS-owned BSAM data set or data sets. Both destinations can fill up as CICS writes trace entries. When this occurs, trace data can wrap and overwrite older trace data. CICS also supports tracing to GTF.
CICS trace points generate CICS trace entries. These are embedded at specific points in CICS code; from these points, trace entries can be written to any currently selected trace destination. Most trace points are used to trace mainline execution of CICS code. Typically, mainline trace points have an associated trace level. Trace levels can vary in value from one to 32, but nearly all mainline trace points have a trace level of one or two.
qLevel-1 trace points are designed to give you enough diagnostic information to fix user errors. They’re located on entry to, and exit from, every CICS domain and major internal domain functions. The information traced includes parameters passed on the call and any output.
Level-2 trace points are situated between level-1 trace points; they provide information that’s likely to be more useful for fixing errors in CICS code itself. You probably won’t want to use level-2 trace points yourself, unless you’re requested to do so by IBM support staff as part of problem investigation.
CICS always performs exception tracing when it detects an exception condition. The exception trace points don’t have an associated ”level” attribute, and trace calls are made from them only when exception conditions occur. You can’t prevent CICS exception tracing even if you have all other CICS tracing turned off. Exception tracing is always written to the CICS internal trace table and, if enabled, to CICS auxiliary trace data set(s). The aim is for First Failure Data Capture (FFDC) and to record data that might be relevant to the exception as soon as possible after it has been detected. Preventing exception trace entries from being disabled ensures the FFDC trace data is always available for analysis.
CICS tracing can be controlled via SIT parameters or interactively with the CICS supplied transaction, CETR. Both methods let you specify the amount of tracing to be recorded, the trace level(s), and trace destination(s). CICS internal tracing can be formatted in a CICS transaction or system dump. CICS auxiliary tracing can be formatted by the auxiliary trace formatting batch program supplied with CICS. For CICS Transaction Server 3.1, this is DFHTU640. The suffix of the name corresponds to the release number of the CICS being used. CICS GTF tracing is an invocation of z/OS GTF trace, and the trace entries are directed to a destination defined to z/OS. This can be either a trace table in main storage or a single external GTF trace data set.
Figure 1 shows an example of the main control panel that’s displayed when the CETR transaction runs.
Other programs besides CICS can use the z/OS GTF trace destination. This lets you integrate CICS trace entries with those from other programs. A single GTF trace data set could, for example, contain trace entries made by CICS and VTAM. Another reason to use the GTF trace destination is that performance overhead may be lower than using CICS auxiliary trace destination.
If the performance overhead caused by CICS tracing is an issue, you may wish to consider limiting the amount of CICS tracing to just one transaction. You can set up this special tracing using the CETR transaction. Once in the first CETR panel, turn off the master system trace flag so the rest of the transactions aren’t traced. Then, turn on special tracing for the transaction you want to trace. Hit PF4 to list the components, set up the special tracing to reflect the tracing you want to do on the transaction, and then press enter to make the changes. Once the special tracing is set up as desired, return to the main CETR screen by hitting PF3. Next, hit PF5 to go to the transaction and terminal options. Enter the transaction ID and set the transaction status to “special.” Then return to the main CETR screen by hitting PF3. On the main CETR screen, set internal trace status and auxiliary or GTF trace status to get started. The special tracing for the selected transaction should be written to the auxiliary or GTF trace and the standard tracing for the rest of the transactions will be suppressed because the master system trace flag is off. This process is described in the CICS Transaction Server 3.1 Problem Determination Guide.
Before you can look at your trace output, you need to format it. Formatting depends on which trace destination you selected. The following examples are for CICS Transaction Server 3.1 and have suffices of 640 (its CICS release number).
The internal trace table can be formatted from a:
- CICS system dump using the CICS dump verbexit, DFHPD640
- Transaction dump using the CICS dump utility program, DFHDU640.
Auxiliary trace can be formatted using the CICS trace utility program, DFHTU640. CICS GTF trace can be formatted using the CICS-supplied routine, DFHTG640. For more details, see the CICS Transaction Server 3.1 Operations and Utilities Guide.
You can specify abbreviated, short, or extended trace formatting to give you varying levels of information and detail in your output. Typically:
- Abbreviated format trace gives you one line of trace per entry
- Short format provides two lines of trace per entry
- Extended format provides many lines of trace per entry.
When reading a CICS trace, it’s often useful to start with the abbreviated trace formatting to give you an overview and identify which trace entries are of interest. Each trace entry has its sequence number at the end of each entry. This number is always enclosed by equal signs in the form of, for example, ‘=000005=.’ If you wished to find equivalent trace entry ‘=000005=’ in the extended trace, then a search on ‘=000005=’ would find the fully interpreted version of that same trace entry. Sequence numbers are unique in a given trace and let you specifically identify each trace entry formatted from the trace destination.
Figure 2 shows an edited example of some abbreviated CICS trace entries. The trace formatter gives the task number (00028 in this example), TCB being executed on, and the domain and identifier of the trace entry, as well as a readable interpretation of the trace entry itself. For example, the trace entry with number =000530= was issued by task 00028, running under the QR TCB, and represents an EXEC CICS command (it has the trace point of AP 00E1).
The CICS trace domain also provides a diagnostic global trap/trace exit program called DFHTRAP. This allows a detailed online investigation, typically without having to stop and restart CICS when wanting to use it. The global trap/ trace exit is used to detect errors that can’t be diagnosed by other methods and is intended to be used only with guidance from IBM service personnel. DFHTRAP is an Assembler language program that can be invoked whenever the trace domain is called to make a trace entry. The trap must be activated before it’s used, either dynamically or at CICS initialization. A skeleton version of DFHTRAP is supplied in both source and load-module forms. For CICS Transaction Server 3.1, the source of DFHTRAP is provided in the CICS640. SDFHMAC library. The source of the skeleton DFHTRAP contains comments explaining its use of registers and DSECTs and the coding necessary to use the exit. Note that the code in DFHTRAP must not use any CICS services, cause the current task to lose control, or change CICS system status.
For more details, see the CICS Transaction Server 3.1 Problem Determination Guide. In addition, the February/March 2005 issue of z/Journal contains the article “DFHTRAP: Assisting the CICS Systems Programmer.”
The CICS transaction dump and CICS system dump can help you with problem determination. The type to use depends on the nature of the problem. In practice, the system dump is often the most useful; it contains more information a problem you suspect is limited to a particular program, a transaction dump may suffice. It will capture transaction-related storage and trace entries. However, a transaction dump might have missed some important information. That would be true if your problem relates to deadlocks between multiple tasks and you need to refer to task storage for several tasks in the system at the time of the deadlock.
Transaction dumps contain transaction- specific information that’s written to a pair of CICS BSAM data sets, with DD names DFHDMPA and DFHDMPB. CICS system dumps contain data from the entire CICS system address space and this is written to z/OS dump data sets. CICS can write only one system dump to any individual dump data set.
CICS provides transaction and system dump tables to let you dynamically control the number and type of dumps produced. For example, revisiting the deadlock scenario between multiple CICS tasks, you may wish that CICS produce a system dump instead of a transaction dump. The following example shows how to use the CICS supplied transaction, CEMT, to make a transaction abend code ASRB produce a CICS system dump:
CEMT SET TRDUMPCODE(ASRB) ADD SYSDUMP
You may wish to produce a CICS system dump when a specific CICS message is issued. You can do this by adding the dump code (constructed by removing the “DFH” prefix from the message number) to the system dump table, and specifying the SYSDUMP option. CICS then causes a system dump to be taken when the message is issued. Here’s an example that will produce a CICS system dump when the DFHXS1111 message is written: CEMT SET SYDUMPCODE(XS1111) ADD You also can limit the number of these dumps taken by setting a maximum limit:
CEMT SET SYDUMPCODE(XS1111) MAX(1)
To determine which messages you can do this for, look in the CICS Messages and Codes manual. If the message you’re interested in has a two-character alphabetic component ID after the “DFH” prefix, and it has either XMEOUT global user exit parameters or a destination of “Terminal User,” you can use it to construct a system dump code to add to the dump table.
When describing the formatting of CICS internal trace data, CICS transaction dumps can be formatted by using the supplied utility program, DFHDU640. See the CICS Operations and Utilities Guide for more details.
You can process system dumps from z/OS dump data sets by invoking the Interactive Problem Control System (IPCS). DFHPD640 is the formatting verbexit you must define to IPCS for CICS Transaction Server 3.1. Once you’ve added your CICS system dump to the IPCS dump inventory and set it as the current default, you can begin to investigate. The component keywords specify areas of CICS you want the verbexit to format dump data for and the level number operand specifies the amount of data you want formatted. Some more commonly used component keywords for investigating a system dump are:
IPCS VERBX DFHPD640 ‘DS=3, AP=3, KE=3, SM=3,TR=3’
Component keyword ‘DS’ will show you the CICS dispatcher domain information that includes a task summary. This gives you the currently active transaction numbers and shows what each of the transactions is waiting or suspended upon. Component keyword AP shows you details of the transaction IDs and their programs, EIB (EXEC Interface Block), and transactional storage.
Component keyword KE is useful for investigating program checks. It includes the kernel error data that contains the failing Program Status Word (PSW) and registers for each program check or (where appropriate) abend. Component keyword SM (Storage Manager) provides details of CICS storage management. This can be useful when looking into a CICS storage violation or Short On Storage (SOS) condition. Finally, component keyword TR formats the CICS internal trace table showing abbreviated, then fully interpreted, trace entries.
For more details, see the CICS Transaction Server 3.1 Operations and Utilities Guide and Problem Determination Guide.
IBM Fault Analyzer for z/OS v6.1 provides another good way to format and investigate a CICS system dump. It lets you use ISPF to perform “interactive re-analysis” of a CICS system dump. This function offers a panel-driven approach to CICS dump investigation that provides a quick way to investigate a CICS system abend. In the case of a storage violation dump, it supplies details of the storage corruption and can give you the option to view the overlaying data. IBM Fault Analyzer attempts to inform you of potential problems it detected during dump analysis. It produces observational messages by severity. It may tell you a task is suspended and provide details of that suspended task. Another useful and time-saving feature is the way in which IBM Fault Analyzer lets you view CICS internal trace data. From a panel, you may choose to view abbreviated, short, or full tracing. This can be selected for all tasks or a single task. The trace is interpreted to show you the name of the calling program and offset based on the return address in each trace entry. This can quickly show which programs were involved before an error, which may be just the clue you need.
For more details, visit www-306.ibm.com/software/awdtools/faultanalyzer/.
CICS has a rich set of informational and diagnostic messages. Most of these are controlled and issued by the message domain, ME. Exceptions include certain messages such as those issued early in CICS start-up or late in CICS shutdown. CICS message formats adhere to the following structure for most of its messages:
- A three-character prefix (DFH)
- A two-character domain identifier (e.g., TS for temporary storage)
- A four-digit message number (e.g., 1310). This message number uniquely identifies the message in the range of messages issued by that particular domain in CICS.
The messages are described in the CICS Transaction Server 3.1 Messages and Codes manual. Every message issued by CICS is documented in this book. The entries explain the reason for the message being issued, the CICS system action as a result of the message, the appropriate user response, and also document the components of CICS that can cause the message to be issued, and the message’s destination. CICS supports messages to be written to a variety of destinations, including the console, terminal user, SYSPRINT, and a variety of transient data queues.
CICS provides its message files in English, Kanji and Simplified Chinese characters. The NATLANG system initialization parameter defines the languages supported in a given run. A variety of other languages can be supported, although an appropriate message file containing the CICS messages translated into these languages must be provided. To facilitate this, CICS provides a message editing utility, DFHMEU, that can be used to change the language or text of CICS messages.
CICS provides a global user exit XMEOUT in the message domain. Messages that can drive the XMEOUT global user exit pass a list of XMEOUT parameters to the exit. These let the XMEOUT exit make decisions such as suppressing or rerouting messages. Message numbers can be used to control CICS system dumps, using entries in the system dump table. For example, entering SM0131 in the system dump table requests that CICS captures a system dump when message DFHSM0131 (i.e., “short on storage below the 16Mb line”) is issued.
The CICS transaction CMAC lets users enter message numbers and abend codes online and read the message descriptions and explanations. CMAC is an online equivalent of the Messages and Codes manual and is kept up-todate as message definitions are modified by PTFs during the lifetime of a CICS release.
Figure 3 shows examples of CICS messages written to the joblog. Note the DFH prefix of the messages followed by the two-character domain identifier for the component of CICS that issued the message. For example, the trace domain issued the messages DFHTR0113 and DFHTR0117 when the CICS auxiliary trace was started and stopped, respectively.
CEDF and CEDX
CICS provides the Execution Diagnostic Facility (EDF) to test application programs. The CICS supplied transaction, CEDF, supports it. Use of CEDF is predicated on a terminal associated with the transaction to be debugged. CICS also lets you invoke EDF through the CEDX transaction. CEDX invokes EDF for testing those applications associated with non-terminal transactions (e.g., back-end or batch-style programs running in CICS, transactions initiated by an EXEC CICS START command, or by a transient data queue trigger-level being reached). The CEDX transaction lets you specify the name of a transaction to be debugged (instead of a terminal, as with CEDF).
EDF lets you intercept application programs:
- At their initiation
- Before and after each EXEC CICS command
- For DFHRMCALs to external resource managers such as DB2
- At program termination.
EDF supports single- and dual-terminal modes. To begin an EDF session with a single terminal, you clear the screen and enter the id of CEDF. The EDF input and output screens with their diagnostic data are then interleaved with those from the transaction. EDF also can be used in dual-screen mode, using one terminal to monitor a transaction running at another terminal. You can follow the sequence of commands in the transaction running at the second terminal as they’re displayed by EDF on the first terminal screen.
EDF allows many options at its interception points. These include the ability to:
- Modify values associated with the command options
- Change the results of a command
- Inspect storage in the CICS address space
- Access the program’s working storage and EIB control block
- Display temporary storage and transient data queue contents
- Abend a task.
There are several restrictions on use of EDF. For more details, refer to the CICS Transaction Server 3.1 Application Programming Guide.
This article reviewed several diagnostic functions and techniques available to assist the CICS systems and application programmer. You can learn more by referencing the various manuals in the CICS bibliography. Z