Aug 1 ’09

Mainframe Network Security: A Strategy for Evaluating Your Risks

by Stu Henderson in z/Journal

One reason IBM mainframes are considered more secure than other platforms is that their network security has a reliable architecture. Unfortunately, this security can be easily compromised if available tools aren’t effectively implemented. This is often the result of lack of understanding of how the security works and of what tools are available.

While many people understand IBM’s System Network Architecture (SNA) and many more understand TCP/IP, few understand both. Many mainframe network security exposures originate in the gray area between the two. If you haven’t conducted a systematic network security review, you’re likely to be unnecessarily exposed to some security risks in your mainframe network. These risks can be controlled with available tools, but only if you first identify the risks.

Whether you know SNA, TCP/IP, neither or both, this article will help you conduct your own systematic risk assessment of your entire mainframe network. We’ll show you a structured approach to systematically investigate your mainframe network, its associated security risks, and the tools (both available and implemented) you can use to manage the risks. To flesh out the details of this approach, you will likely need help from your VTAM systems programmer, your mainframe TCP/IP administrator, and your security software administrator.

The security risks to be addressed with either type of network are:

• Unauthorized reading of sensitive data (including possibly userids and passwords)

• Unauthorized connecting to a computer

• Falsely assuming the identity of a computer, program, or user when connecting to another computer.


On IBM mainframes, the VTAM software controls all telecommunications. When VTAM was first created, all communications were completely based on SNA, whether it was a terminal talking to a program such as CICS or Time Sharing Option (TSO), or two programs on different computers exchanging information.

Several years ago, IBM added TCP/IP support to VTAM. This was a smart move since UNIX, Windows, Novell, the Internet, and several other platforms all use TCP/IP to talk to each other. As IBM expanded its use of TCP/IP on the mainframe, many people believed incorrectly that SNA was on its way to extinction.

Most mainframes formerly used a hardware device called a Network Control Processor (NCP) to connect terminals to the mainframe. At one time, most terminals were hard-wired into the NCP and used the SNA protocol to communicate with it.

With time, hard-wired terminals were replaced with desktop computers running Windows, and connected in a LAN. This LAN used TCP/IP, not SNA. In a typical configuration, each department in a company might have its desktop computers connected in a departmental LAN, with all the LANs hard-wired to the NCP.

As TCP/IP usage started to expand, IBM replaced NCPs with a different type of hardware called an Open Systems Adapter (OSA). The OSA uses TCP/IP. LANs are now connected to the OSA using TCP/IP.

Some people assumed this meant SNA is no longer used. They fail to realize that the connections of terminals to the mainframes through OSAs still use SNA. The SNA is packaged inside the TCP/IP packets sent between the LANs and OSAs.

Another factor that caused people to assume the end of SNA was IBM’s introduction of new functionality called Enterprise Extender (EE), which uses Unformatted Datagram Protocol/ Internet Protocol (UDP/IP), which is somewhat similar to TCP/IP. UDP/IP actually passes SNA packets across the network by enveloping them in a UDP/IP wrapper. This means SNA is still being used; it’s just hidden in UDP/IP packets.

All the security mechanisms for TCP/IP and UDP/IP (such as firewalls and encryption) provide no protection against SNA attacks on your network. The SNA packets containing the attack messages aren’t affected by TCP/IP packet filtering. If the SNA message is encrypted on one end, then decrypted on the other, it’s just as dangerous as it originally was.

Both SNA and TCP/IP are likely to be part of your mainframe networks for years to come. Whether you’re familiar with security for SNA or TCP, the following descriptions will help you understand both sides.

How SNA Works

SNA is IBM’s original telecommunication architecture for mainframes and other platforms. While use of TCP/IP is growing and many people claim SNA is “going away,” it will be a major part of our networking for the foreseeable future.

SNA is a means to connect—to support communication between two endpoints. The end-points may be terminals, programs, or hardware devices such as an Automated Teller Machine (ATM). Each end-point is defined as a Logical Unit (LU). Each LU has an LU name. Each terminal and other hardware device is assigned an LU name. Each program that talks over the network (such as CICS or TSO) is assigned an APPLID name; that is, an LU name denoting a program. An LU is considered an entry point to a network, whether it’s a terminal or an APPLID.

All LU and APPLID names are defined to the VTAM software in a control file called SYS1.VTAMLST. When a terminal requests a logical connection in the same network to a program such as CICS, VTAM only allows the connection if both the terminal LU and APPLID are defined in SYS1.VTAMLST. This provides an important security control since no terminal or program could talk over the network without being defined to VTAM.

How TCP/IP Works

TCP/IP consists of two parts: the IP part provides routing of messages among computers; the TCP part provides application support, passing messages to the programs for the various applications. A port number identifies each application. For example:

• Ports 20 and 21 are often used for File Transport Protocol (FTP)

• Port 23 is for remote logons

• Port 25 is for email

• Port 80 is for the httpd server (the program that talks to Internet Explorer over the Internet).

On the mainframe, TCP/IP is a started task that depends on UNIX System Services, the standard z/OS UNIX. From VTAM’s perspective, TCP/IP executes as an APPLID.

How to Evaluate SNA Security

We’ll show you how to evaluate SNA security in three steps:

• On a single mainframe (MVS image)

• On several computers connected as part of one network

• On computers on a series of connected networks.

Our approach is to identify all the MVS images, all the networks, and then all the cross-network connections. Next, we’ll evaluate all related risks and the tools available to provide security.

On the mainframe, all SNA options and objects are defined to the VTAM software in a data set called SYS1. VTAMLST. Within a given MVS image, VTAM only permits connections between LUs that are defined in SYS1. VTAMLST. The VTAM systems programmer is usually the only one who updates this control file.

Step 1: On each MVS image, you can learn the name of SYS1.VTAMLST by examining the Job Control Language (JCL) for the VTAM started task, which is often named NET. In that JCL will be a Data Definition (DD) card with the DDNAME of VTAMLST. This will tell you the dsname used in your installation for SYS1.VTAMLST. This partitioned data set will have:

• Start-up members (named ATCSTRxx)

• Configuration members (named CONFIGyy)

• Other members identifying applids, terminals, adjacent networks, and other resources.

When the system is IPL’ed, MVS issues a START NET command to start VTAM. This command will tell VTAM the last two characters of the start-up member (ATCSTRxx). VTAM reads this member and from it determines the options to execute with. The start-up member also contains an operand named CONFIG=yy, which specifies the last two characters of the configuration member. VTAM reads the configuration member, which contains the names of other members that describe the APPLIDs and other resources VTAM is to use. VTAM then reads each of these members, saving the information in tables in memory. VTAM uses these tables to determine whether to accept requests for binds between a terminal and an APPLID.

Much of the security and control is derived from this rule: Within a given MVS image, no terminal or APPLID can communicate over the network unless it’s defined in the SYS1. VTAMLST data set.

Step 2: Let’s assume you have two MVS images, one for test and one for production. Further assume that a terminal defined to VTAM on the test image wants to connect to an APPLID (say a CICS region) defined to the production VTAM. This is possible if each VTAM is defined to the other VTAM, and if each VTAM has the other VTAM’s resources defined as “cross-domain resources.” The other VTAM and the other resources will be defined in members of SYS1.VTAMLST. The resulting connection is called a “cross-domain bind.” The term “domain” here corresponds to one MVS image.

Much of the security and control with cross-domain connections is derived from this second rule: With cross-domain connections, no terminal or APPLID can communicate over the network unless it’s defined in SYS1. VTAMLST on both domains.

Step 3: Our last step is to examine cross-network connections (such as if your SNA network is connected to a business partner’s SNA network). Unlike the previous two steps in which VTAM only allows connections between terminals and APPLIDs defined in its control file, cross-network connections can allow dynamic binding. A terminal in your network can connect to an APPLID defined in your business partner’s network without the APPLID being defined in your SYS1.VTAMLST.

This may be perfectly acceptable if you can trust your partner. However, you will want to know to what other networks your partner’s network is connected. A terminal in your network can connect to an APPLID in some other network as long as that other network is connected to your partner’s network, or to a network connected to your partner’s network, and so on. If this dynamic binding is permitted, then it’s possible for some other network to spoof (assume the identity of) your partner’s network. You will want to make sure you have controls in place to prevent this. 

To summarize, follow these steps to evaluate the SNA part of your network:

1. Map your network’s name, its MVS images, terminals, APPLIDs, crossdomain connections, and partner networks. You will get this information from the SYS1.VTAMLST data sets on each MVS image. You may need your VTAM systems programmer’s assistance in interpreting and verifying this information.

2. Ensure that each APPLID calls your security software to verify userids and passwords. Make sure no APPLID uses hard-coded lists of userids and passwords, especially vendor-supplied ones.

3. For each path in your network that carries sensitive data, ensure there’s effective encryption, either from the hardware or software.

4. For each possible connection you want to restrict, ensure that needed restrictions are in place. These will often be provided by your VTAM systems programmer; for example, by deactivating cross-domain definitions or by using USSTABs (a VTAM method that can control binds).

5. Use your security software to restrict which users can bind to any sensitive APPLIDs. RACF uses the APPL resource class; ACF2 uses SOURCE definitions, and TopSecret uses FACILITY definitions.

6. Use your security software to prevent any program from spoofing an APPLID. This could allow the program to put up a fake logon screen on someone’s terminal and use it to harvest userids and passwords. VTAM calls the security software in the VTAMAPPL resource class to prevent this, but it provides protection only if the security software has rules in place.

7. Identify the networks your network is connected to; for example, your partners’ networks. These will be called “adjacent networks” and are identified in SYS1.VTAMLST. Identify what networks they’re connected to. Determine what your contract with your partner says about who is liable in the event of security breaches. Ask your VTAM systems programmer what VTAM options and software tools can prevent spoofing of networks. For more information, see “An Often Overlooked Security Hole in SNA Networks” in the April/May 2008 issue of z/Journal.

8. Review protection on the Windows LANs used to sign onto the mainframe. Determine what would prevent someone from installing a sniffer program to learn other users’ mainframe userids and passwords. Encryption isn’t sufficient protection since someone executing a sniffer program could record and play back the logon message without needing to decrypt it. Implementing Kerberos in Windows Active Directory can provide protection over the LAN. Physical protection of the link between the LAN and the OSA would provide further protection.

9. Determine whether the OSA that connects LANs to the mainframe is shared between MVS images. This could permit a user on the test system to access the production system, bypassing any firewall you think is isolating test from production. You will need to review your Hardware Configuration Definition (HCD) and Input/Output Control Program (IOCP) hardware configuration data sets to determine this. (Thanks to David Hayes and Kevin Metcalfe of the U.S. Government Accountability Office for identifying and researching this possible exposure.)

10. Summarize your findings, including any uncontrolled security risks and recommended further protections. Review this with your VTAM systems programmer.

How to Evaluate TCP/IP Security

We’ll show you how to evaluate TCP/IP security first on a single mainframe (MVS image). Our approach is to:

• Identify all the MVS images

• For each MVS image, identify all the TCP/IP stacks

• For each stack, identify the IP addresses and ports that can be used

• Evaluate all related risks and tools available to provide security, including some tools unique to mainframe TCP/IP.

On z/OS, you may have several “stacks” or copies of TCP/IP on each MVS image. To develop a big picture of all the TCP/IP activity, you need to identify all the stacks, and then for each stack, identify the security and control options.

Once you’ve identified the stacks, for each stack you need to review the TCP/IP control file, often called PROFILE.TCPIP. This control file will list the IP addresses, port numbers, hardware devices called OSA that link LANs to VTAM, daemons (started tasks) that are automatically started when TCP/IP starts, and the names of APPLIDs to which Telnet users can request a bind.

In summary, here are the steps to evaluate the TCP/IP part of your network:

1. Map your MVS images, and for each, the IP addresses, port numbers, daemons, and physical connections it uses. You may need your TCP/IP administrator’s assistance in interpreting and verifying this information.

2. Determine what firewalls protect your mainframe TCP/IP network and what techniques they use, such as packet filtering, network address translation, and intrusion detection systems. Include both external firewalls and the policy agent software that provides firewall-like functionality and comes free with mainframe TCP/IP.

3. Summarize the IP addresses used on your mainframe and their names. For example, you might call the IP addresses inside your firewall by one name and the addresses outside your firewall by a different name. Determine whether you need to define these names to the security software to control who can use which IP addresses. Use the SERVAUTH resource class with the security software to accomplish this. The use of RACF, ACF2, or Top Secret to protect IP addresses, ports, and other resources is one advantage that mainframe TCP/IP has over other platforms.

4. Identify what ports are protected and what daemon (application program) is associated with each port. In the PROFILE.TCPIP control file, the RESTRICTLOWPORTS operand can prevent unauthorized programs from opening any of the ports numbered 1-1023. On the PORT and PORTRANGE statements, the RESERVED, DENY, and SAF keywords can prevent use of other ports, or restrict who can use them by means of the security software. Evaluate the risk if an unauthorized program executing on the mainframe were to open a port and start reading and writing over your TCP/IP network. Unless every port is restricted or protected, this may be a risk you want to minimize. You may decide to implement formal change control, only permitting a program to open a port to which the program has been approved.

5. Evaluate each daemon (a program dedicated to one application and usually to one port number). Each daemon is an MVS started task. Each is assigned a userid and may have privileges in the security software or superuser privilege in UNIX System Services. Each will have its own JCL and often its own control file specifying security options. Most daemons will be defined in the PROFILE.TCPIP control file, but a few, such as Telnet, will be defined in the /etc/inetd.conf control file for UNIX System Services. Ensure that daemons don’t have security software privileges nor root privilege in UNIX System Services unless there’s a clear reason (such as IBM documentation) to do so.

6. For each daemon, determine whether it needs encryption to protect sensitive data over the network. Most daemons can easily activate system Secure Sockets Layer (SSL) or Kerberos on MVS if directed by control options.

7. For each daemon, determine how it establishes users’ identities. Many will rely on userids and passwords, some on other means. Some may permit anonymous logins. Some (such as the httpd daemon) will assign userids on the basis of matching filenames to directives in their control file. A few, which should seldom be used, such as Trivial File Transport Protocol (TFTP), always permit anonymous logins.

8. For Telnet (remote login) connections, determine which APPLIDs Telnet users can bind to. This is a little-understood gray area between VTAM and TCP/IP. The ALLOWAPPL operand of the PROFILE.TCPIP control file specifies which APPLIDs a user can bind to by connecting through Telnet. The default value is *, which means any APPLID, which could make it possible for anyone on the Internet to bind to any APPLID on your system. This risk is greatly reduced if every APPLID calls the security software to verify userids and passwords. It’s also greatly reduced if firewalls or the security software can restrict who can connect to the Telnet port.

9. Note that the FTP daemon supports functions beyond uploading and downloading both UNIX System Services and MVS data sets. Depending on control file options, it can permit submission of batch jobs, operator commands, access to print files on the print queue, and access to DB2 data. Most of these functions can be secured with security software.

10. Ensure that the files and data sets containing programs, JCL, and control information are protected with security software and reasonable change control.

11. Summarize your findings, including any uncontrolled security risks and recommended further protections. Review this with your TCP/IP administrator and VTAM systems programmer.


Mainframe computers with z/OS will continue to use both SNA and TCP/IP for telecommunications. The tools are easily available to make them both secure. To keep them secure, managers will need to be sure that all security exposures are evaluated and that reasonable, practical protection is implemented as needed.