Nov 13 ’14
SMF: Exits and the Life of a Job
In the August/September 2014 issue of Enterprise Tech Journal, we provided an introduction to one of the most important components of z/OS—the System Management Facilities (SMF) component. Two of the key parts of MVS that write records to SMF are the job scheduling system and the master scheduler. SMF records are written as address spaces are created, executed and terminated. In this article, we’ll describe the life of an address space and when user exits are called and SMF records are written. Future articles will include more detail on SMF parameters, recommendations on SMF file creation and usage, and details about the most important record types.
There are several points during the life of an address space when a user may access the address space information and SMF records using SMF exits. In this section, we’ll describe the exits, and how they are used.
The SMFPRMxx parmlib member provides directions to SMF as to which records should be collected and written to a data set or logstream. Two of the SMFPRMxx keywords that specify the type of records that you want to collect are SYS and SUBSYS. One of the subparameters of these two keywords is EXITS(…). In this subparameter, you code the name of the exits that are to be invoked for the entire system or for certain subsystems. User exits are usually written by the systems programmer, although some are provided as samples by IBM or other vendors and others are offered by other users at no charge.
All of the SMF exits are shown in Figure 1, and are described below in the life of a job/ TSO user/started task/APPC process. You can specify which exits are to be invoked for all records on the SYS parameter or by subsystem on the SUBSYS parameter of SMFPRMxx. Valid IBM subsystems are JES2 or JES3 (for batch jobs), TSO (for TSO logons), STC (for started tasks) and ASCH (for APPC/MVS processes). Further information about exits can be found in the IBM z/OS MVS System Management Facilities (SMF) manual, or the IBM z/OS MVS Installation Exits manual.
Exits are primarily used to provide security, an audit trail or additional reporting. For the added function, you would expect some overhead. The overhead directly depends on the programming ability and efficiency of the exit programmer. Remember, these exits are called for every job, TSO user and started task, so they all need to be written extremely efficiently.
The CPU time used by executing exits will show up in various places in the job’s CPU times. It might show up in the TCB time, the SRB time or the initiator CPU time. An inefficient exit is difficult to identify because the CPU times are distributed across all of the jobs.
Your installation will determine which exits are required based on their chargeback, reporting or audit needs. Our primary recommendation is that you understand the reason for each exit and document it in parmlib member PROGxx, which defines all exits. You’ll still need to maintain SMFPRMxx to indicate which exits are to be called for each subsystem.
If an exit is only needed for one subsystem, then include it on the SUBSYS statement, not the SYS statement. Watch out when IBM adds new exits. As an example, originally there was an exit IEFU83, which was called prior to an SMF record being written. An application would pass a record to SMF using the SMFWTM macro, and SMF would call IEFU83 with the record. The exit could choose to delete the record, modify it or perform some action based on the record’s contents. Then IBM came out with a new macro, SMFEWTM, that could be used by applications to pass records to SMF. If SMFEWTM was called with an option of BRANCH=NO, exit IEFU83 was still called. But if SMFEWTM was called with an option of BRANCH=YES and MODE=XMEM (branch entry and cross-memory mode), then SMF called a different exit, IEFU84. And if the macro was called with an option of BRANCH=YES and without MODE=XMEM (just branch entry), then SMF called a different exit again, IEFU85. When IBM added these new exits, most people didn’t notice, so many SMF records were bypassing the exits. Our SMF Reference Summary at www.watsonwalker.com/references.html shows which of these three exits is called for each record type.
If you don’t want to code your own exits, there are two software products that provide exits (and don’t require Assembler knowledge). They are OS/EM from Trident Services (www.triserv.com) and Easy/Exit from DTS Software (www.dtssoftware.com).
Life of an Address Space
An address space may be one of four types of spaces according to SMF: TSO users, batch jobs, started tasks (including system address spaces) and APPC scheduler (ASCH) address spaces. With the exception of the system address spaces, the lives of the address spaces are recorded from the initiation to the termination of the address space. A type 30 SMF record is the most common record type associated with address spaces, but we’ll also address types 6, 26 and 32 records.
Figure 2 shows the life of an address space in SMF processing and this discussion will continue to refer to that figure. The left-most column indicates the point in the life of the address space, along with the records (if any) that are produced. The items indicated by a ▼ show some of the elapsed times that may be calculated from various timestamps. And the right column identifies SMF exits that are invoked. Although we use the term “job” throughout this article, we use it to mean any type of address space, including TSO users and started tasks.
As you can see from Figure 2, a job is read into the system and the JCL is scanned and converted. During this time, the IEFUJV SMF exit is invoked but no SMF records are written. The IEFUJV (job validation) exit can be used to edit and modify the JCL. The IEFUJV exit is actually called three times: before the line is processed by the converter, after the line is processed by the converter and one time after all lines have been processed. This is the most common exit used to validate job names, security and accounting codes. IBM recommends that you use the IEFUAV exit instead of IEFUJV for APPC/MVS work.
TSO users and started tasks (STCs) are then initiated immediately, while batch jobs wait for a JES initiator. When the address space is initiated, the IEFUJI (job initiation) exit is invoked and an SMF record is written. A type 30, subtype 1 (30.1), record is written containing information from the job card, the job identification and several timestamps. A job is uniquely (hopefully!) identified by the job name (or TSO userid or STC procedure name) and the reader timestamp (the time the job was first recognized by JES). Unfortunately, the timestamp is only precise to a hundredth of a second. As machines have gotten faster, this has proven to be a problem since it is possible for multiple jobs with the same job name to have the same reader timestamp! In a few instances, this has produced invalid results for these jobs. One solution to this has been to use the JES job number to further uniquely identify a job. Just before the SMF 30.1 record is written, the IEFU83, IEFU84 or IEFU85 exit is called.
Almost immediately after the job starts, the first step is initiated. The IEFUSI exit is called prior to beginning execution. No SMF records are written at this point.
As the step begins, there is a period of time spent enqueing the data sets, and then the devices are allocated, after which the program is loaded and execution starts. These timestamps are kept for inclusion in later SMF records. During execution, if the amount of CPU time exceeds the time specified for the job class, the IEFUTL (User Time Limit) exit is called. This exit can be used to extend the time or to cancel the step. Also during execution, if the amount of lines written to spool exceed the number specified for the job class, the IEFUSO (User SYSOUT) exit is called. This exit can be used to extend the number of lines or to cancel the step. The JES2 JOBCLASS parameter can indicate that IEFUSO is not to be called for certain job classes, all started tasks and/or all TSO users.
If INTERVAL (another keyword that we’ll cover in the next issue) was specified in SMFPRMxx for this type of address space, and the job is running longer than the interval, a type 30, subtype 2 record is written, after a call to the IEFU83/84/85 exit. In general, the data in the subtype 2 record contains the resources consumed for just the interval. The one exception is the type 30, subtype 6 record that is written for system address spaces at each interval. The subtype 6 records contain cumulative resource usage.
When a step completes, either normally or abnormally, a type 30 subtype 4 record is written. This record provides the majority of the data used for address space analysis. It contains the total consumption for the major resources: CPU, I/O and storage. If INTERVAL recording was active, a type 30 subtype 3 record is also written and contains the resource usage for the last (partial) interval. The IEFU83/84/85 exit is again called before the SMF record is written. In addition, the IEFACTRT exit is called. This is one of the most popular exits and is used to place step termination information in the JES log. Those of you who see summary information of CPU usage and times at the end of every step probably have an IEFACTRT exit in place.
Actually, the records created at job termination are very similar to those written at step termination. Many installations find that job totals are sufficient for chargeback and capacity planning. However, you should consider the instances where the job record does not get produced. For example, a job may have completed execution of 10 steps during the last eight hours and the system crashes. You’ll never see the job termination record but you probably want to account for the eight hours of usage, so the step termination records are important.
At job termination, the IEFACTRT exit is called again. The exit normally has logic to handle both step termination and job termination in slightly different ways. The exit usually produces output in the JES log showing job totals for resource usage, condition codes and timestamps.
A type 30, subtype 5 record is produced at job termination, with the IEFU83/84/85 exit being invoked before the record is written. Major differences between step termination records and job termination records include:
• CPU totals are job totals, not step totals.
• EXCP totals are job totals, not step totals and include LINKLIST activity, JES2 I/O (for JES), catalog management and OPEN/CLOSE.
• Job termination contains job card info such as account codes, programmer name field, job class, etc. while step termination doesn’t.
After the job terminates, it will remain on the JES spool until all of the output is printed or purged. This could take days or even weeks.
A type 6 record is written for every SYSOUT data set that is printed or routed to another location. The IEFU83/84/85 exit is invoked before the type 6 record is written. The type 6 provides line and page counts, as well as information on form usage. This information is used for charge back to recover printer costs; by capacity planners to determine print volumes by printer, location and form; and by service level management to determine when SYSOUT completed printing. Some studies can also be done on the length of time to print output.
There are five subtypes for the type 6 record, depending on the type of output created:
• 0 - External Writer
• 2 - JES2 Output Writer
• 5 - JES3 Output Writer
• 7 - Print Services Facility (PSF) Output
• 9 - IP Printway Output.
Routing information includes the output route code and the output form number. Several studies can be made on number of lines per form or output location. This can be done for daily or weekly totals, as well as hourly totals. One of the major pieces of information that is not available is when the SYSOUT was released from HOLD status. The problem with service level management and report output is that the user could specify a held SYSOUT class, and not release or delete the data set for days. If you are trying to provide batch turnaround (including printing) within one hour, you’ll never make your objectives due to held SYSOUT. For that reason, most service levels today are based on the time the job is read in until the job terminates (but not until the output has been printed). Another item to note with type 6 records is that they can be produced before the step terminates if FREE=CLOSE was specified on the JCL.
The JES2 JOBCLASS parameter can indicate that a type 6 record is not to be created for certain job classes, all started tasks and/or all TSO users.
When all SYSOUT has been printed, routed or deleted, a type 26 record is written. The IEFUJP exit is invoked, and the IEFU83/84/85 exit is called before the record is written. We haven’t seen much use of the IEFUJP exit, so if you have a good reason for one, please let us know. The JES2 JOBCLASS parameter can indicate that IEFUJP is not to be called for certain job classes, all started tasks and/or all TSO users.
Type 26 is a fantastic record. If it contained total CPU time and account code information, it would be almost perfect! The primary information in the type 26 is significant timestamps in the life of the job: reader start and stop, converter start and stop, execution start and stop and output start and stop. Most service level management information can be obtained in this record alone. Additionally, for systems with NJE, all system IDs are provided here. You can tell that a job was read in on one system, converted on a second system, executed on a third system and printed on a fourth. When this information is used with the timestamps, you can also see the impact on multiple systems by time of day, job class and user.
Another nice feature of the type 26 records is the inclusion of total SYSOUT lines written to spool, along with estimated and actual SYSOUT byte counts. You can use this to let programmers know how close they are to their limits (before abending). The JES2 JOBCLASS parameter can indicate that type 26 records are not to be created for certain job classes, all started tasks and/or all TSO users.
Here’s the bad news about the type 26 record: it doesn’t get written until every SYSOUT has been printed or purged. This might take many days if the output is held. If you want to combine all of the type 30 records with the type 26 record for a single job, you might need to hold the type 30 records for a week or more. The amount of time you allow held SYSOUT to remain on spool determines how soon the type 26 record is written after the job terminates.
TSO Command Recording
The intention of the type 32 record is to provide counts and, as an option, resource information on TSO commands. Even though it doesn’t provide response time, the capability to analyze TSO usage by command is very useful. (TSO response time is best collected in the RMF type 72 record, which is by service class period.)
A type 32 record is written at TSO logoff or interval end containing information about each command. What isn’t useful, however, is that called programs all fall under the “CALL” command and CLISTs fall under the “EXEC” command so that you can’t tell information about specific programs or CLISTS. Another negative when using this record is that aliases aren’t combined (that is, “SE” and “SEND” are reported separately). If you want specific information on CLISTs, you could write a command processor that invokes the CLIST and track activity on the command processor.
The type 32 record can be a useful tool for special studies to help you determine which TSO commands should be placed in PLPA and which should be relegated to the LINKLIB libraries. If DETAIL is specified in SMFPRMxx, an additional 40 bytes of data per command is added to the record to provide TCB, SRB milliseconds, TGETs, TPUTs, transactions, EXCP count and device connect time. This is most often used by the performance analysts (turn it on for only one week and study the results).
Interval recording for types 30 and 32 records allows you to specify that records are to be written at specific intervals, such as every 30 minutes. A type 30.2 is written during every interval but the last, where a 30.3 is written. These records contain the same information as the step termination, type 30.4, but only contain totals for the interval. The primary purpose of interval recording is to provide statistics in case the system crashes during a step. For a batch job, this may not be too important, but for TSO it’s critical since a TSO step is the entire logon session. If you do not use interval recording for TSO and you have an “unscheduled IPL” in the middle of the day, you will have lost all SMF measurements for logged on TSO users during the period of time up to the crash. If you do charge back to TSO users, you will have lost revenue.
(Note to capacity planners—you’ll still get your performance group service units from RMF measurements, but no individual totals by TSO user or job.) In a chargeback environment or one where capacity planners are using SMF data, this could amount to a loss of a day’s worth of information or billing data. Interval recording is also used where you want to collect information by job by time of day. For example, my CICS address space job totals may show that I had 10M EXCPs to device 6174, but I may need to know whether those were spread throughout the day or during a single period of activity. Interval recording provides that information.
You can specify interval recording by subsystem (batch, TSO, STC and ASCH) in SMFPRMxx in SYS1.PARMLIB. Different interval times can be specified for each subsystem. An unfortunate side issue of intervals is the complexity of processing them. In general, they need to be sorted prior to processing. For example, we may only want to use interval records if we lost the step termination records. Since the 30.4 follows the 30.2 and 30.3 records, we can’t process them until we’ve passed them. And we can’t simply turn off step termination (30.4) since if a job executes in less than an interval of time, only a 30.4 will be created and no interval records are written. All in all, it’s a messy job to process them! Several installations have written exits to bypass interval records for everything but important workloads (onlines, production batch and TSO). Where do you do that? In the IEFU83/IEFU84/IEFU85 exits, of course!
Type 30, subtype 6 interval records are written for system address spaces (such as MASTER, TCPIP, XCFAS, TRACE, ZFS) if interval recording is specified for the STC subsystem. This is the only way to get information about these system address spaces. These interval records differ from the subtypes 2 and 3 because they contain cumulative information, not simply totals for the interval. If you do not collect and report on this data, you could underestimate your capacity load by up to 20 percent.
We hope this description of the life of an address space has helped you understand when SMF types 30, 6, 26 and 32 records are written, and when user exits are called. Two of the exits in Table 1, IEFU29 and IEFU29L, will be discussed in a later article when we discuss dumping SMF records to an external file.