Wouldn’t it be nice if there was a one-stop program or report you could use to completely solve any performance problem you’re troubleshooting? While, regrettably, that doesn’t exist, but you can use the FICON director activity report to drill down further into I/O performance problems. Only this report provides FICON frame pacing delay information, giving you a valuable way to narrow down the potential root cause(s) of a performance problem.
This article continues a series on System z I/O performance and considers the Resource Measurement Facility (RMF) 74-7 record—the RMF FICON director activity report. The article “Performance Troubleshooting Using the RMF Device Activity Report” (available at http://esmpubs.com/gm9fn) covered one of the key RMF reports used in mainframe I/O performance management/troubleshooting: SMF 74-1, the RMF direct access device activity report, which contains information on the various components of response time. You can use this report to further narrow down what may be the root cause of the problem and learn what other RMF reports you should check.
Figure 1 illustrates a sample environment and where the various I/O-related RMF reports fit. Let’s assume your review and analysis of the RMF device activity report showed that PEND and CONN are the abnormal response time components. You would want to determine if your Fibre Connection (FICON) Storage Area Network (SAN) could be the component of your infrastructure causing PEND and/or CONN to be abnormally high.
The introduction of the FICON I/O protocol to the mainframe I/O subsystem provided the ability to process data rapidly and efficiently. Two main changes that FICON made to the mainframe channel I/O infrastructure prompted the requirements for a new RMF record. The first change was that, unlike Enterprise Systems Connection (ESCON), FICON uses buffer credits to account for packet delivery and provide a data flow control mechanism. The second change was the introduction of FICON cascading, and the long distance capabilities it introduced, which were practical with ESCON.
Similar to the ESCON directors that preceded them, FICON directors and switches have a feature called Control Unit Port (CUP). Among the many functions of the CUP feature is its ability to provide host control functions such as blocking and unblocking ports, safe switching and in-band host communication functions such as port monitoring and error reporting. Enabling CUP on FICON directors, while also enabling RMF 74 subtype 7 (RMF 74-7) records for your z/OS system, yields a new RMF report called the FICON director activity report. Data is collected for each RMF interval if FCD is specified in your ERBRMFnn parmlib member, and in SYS1.Parmlib, the IECIOSnn says FICON STATS=YES. RMF will format one of these reports per interval for each FICON director that has CUP enabled and the parmlib specified.
If you’re using FICON virtual fabrics and have your FICON directors partitioned into multiple, smaller logical switches, you will have one RMF 74-7 report per switch address (per logical switch). The FICON director activity report captures information based on an interval that’s set for RMF and tells it when to create this report along with others. In essence, the report captures a snapshot of data and the counters based on a time interval (such as 20 minutes). Often, you need to run these reports more than once and change the interval periods for troubleshooting to determine if there’s a trend.
This RMF report is often overlooked, but contains meaningful data concerning FICON I/O performance—in particular, frame pacing delay. Frame pacing delay has been around since fiber channel SAN was first implemented in the late ’90s by our open systems friends. But until the increased use of cascaded FICON, its relevance in the mainframe space was completely overlooked.
Figure 2 shows an example of a FICON director activity report. This report is vendor-agnostic, meaning it will contain the same fields and information regardless of which FICON director vendor manufactured the director. It’s also the same for any model director/switch from a given vendor.