IT Management

With z/VM 4.3, an IFCONFIG command similar to what Linux staff are accustomed to was introduced, which greatly eases network configuration. Other z/VM TCP/IP enhancements further help dynamically configure the network. This is particularly important given the increasing rarity of non- TCP/IP-attached terminals: If a z/VM TCP/IP configuration error occurs, it is often necessary to gain access to the system console to restore a previous, working configuration.

In most installations, the networking group considers the mainframe an “endpoint” on the network, with a single IP address. When a block of addresses is requested for Linux guests to be created, the result is often an extended discussion, with iterations required before routing is correctly set up. It is often best to gather all affected staff, draw the proposed network on the board as if Linux guests were separate machines and, once it’s agreed on, only then draw a box around the outside, showing that the guests are all inside the mainframe. This usually allows everyone to understand the proposed layout. The same approach is effective when a Storage Area Network (SAN) will be involved with a mainframe Linux project.

Once configuration and routing issues are understood, however, networking with z/VM is fast and easy. With guest LANs — virtualized network connections within the physical machine, created and operated by the Control Program (CP) — there are no network cards to install, no wires to plug in. Throughput is at memory speeds — about 2.4GB per second on older machines, and even faster on z990s. And, with CP managing the connectivity, there are no collisions or dropped packets!

CULTURE WARS

The biggest challenge for any Linux on zSeries project is the inevitable “culture clash” between mainframe and other IT staff. For more than 20 years, these two groups have, in most shops, eyed each other warily across a gulf. Mainframers have snorted derisively at the “toy machines,” while the distributed staff laughed about gray-haired dinosaurs with their line-mode interfaces. Existing Unix and Windows personnel have often waged war in an effort to protect their turf, and the network group is often also separate with its own prejudices.

Suddenly, a zSeries Linux project requires these groups to work together for a common goal — one that provides a learning experience for all of them. Even something as seemingly simple as terminology becomes a significant issue; some terms are unfamiliar, and some even have different meanings (see Figure 2)!

Recognize that Linux on z/VM implementation requires collaboration and a sense of ownership across multiple teams. Migrating applications from Windows, for example, often involves all of the teams (see Figure 3).  

PAIN FOR LINUX PEOPLE . . .

Linux system administrators are familiar with poorly (or un-) documented code; however, they are used to having code as a reference. In this regard, z/VM itself, for which most source code is provided, is familiar (although few Linux users know S/390 Assembler). What Linux users are not used to is the quantity and quality of IBM documentation. They are used to acquiring documentation as a series of how-to documents from the Web, supplemented by reference books from various publishers.

When confronted with an IBM bookshelf of z/VM manuals alone, the typical reaction is one of mild to moderate shock. “How are we supposed to find anything?!” they cry. Of course, there is a good reason why IBM refers to its technical writing staff as “Information Developers”: IBM has been creating computer documentation longer than anyone, and produces the most usable manuals in the world. Once the organization of IBM documentation is understood, this hurdle is usually overcome.

Still, it is true that gaining VM expertise is not as easy as it once was. VMers in many installations have retired or moved on, and IBM’s VM curriculum has been pared down drastically over the years, although IBM and others are once again introducing new courses. IBM used to publish a “VM Primer” manual. Used copies are still valuable for beginning VMers, and are often available from used-book Websites. Prentice-Hall recently published Linux on the Mainframe (ISBN 01-3101415-3) by John Eilert, Maria Eisenhaendler, Ingolf Salm, and Dorothea Matthaeus, which may be helpful.

SHARE is also an invaluable resource. A week at SHARE offers a full VM curriculum, as well as the opportunity to talk to dozens of long-time VMers and Linux on zSeries experts — all for far less than most courses. In addition, local user groups, both VM- and Linux-focused, often offer useful sessions or at least contact with peers facing the same issues.

Other mainframe aspects provide more shock for distributed folks: 3270 terminals, 3480 tape cartridges, record formats, serial numbers ... not to mention EBCDIC instead of ASCII! (Being a true Linux, Linux on zSeries does, however, use ASCII.)

For Linux users, some utilities are available on z/VM using the OpenExtensions Shell and Utilities, but most familiar tools such as vi, emacs, diff, and grep are not otherwise usable. Equivalents exist, but still require a learning curve. The z/VM function used by most will likely be the system editor, XEDIT. While very different from traditional Linux editors, XEDIT is very powerful, and some long-time Linux users have even found that they prefer it!

. . . A ND PAIN FOR MAINFRAME PEOPLE

One of the biggest issues for mainframe programmers is the fact that the Linux command language is case-sensitive. In addition, people create files and commands in mixed case! Thus, “logout” is not the same as “Logout” or “LOGOUT,” although most would agree that creating commands with similar names is asking for trouble. This is surprisingly hard to learn: Programmers with years of experience find themselves entering a command that they are certain is correct, only to have it fail because of a case issue.

ASCII is as much an obstacle for mainframe folks as EBCDIC is for Linux folks. Many mainframe veterans have the entire character set memorized in hexadecimal — in EBCDIC. This is somewhat less than useful in an ASCII environment, alas.

Free and commercial tools are available to ease the transition to Linux for mainframe users. This includes REXX for Linux (Regina REXX, uni-REXX, and S/REXX), and there are XEDIT and ISPF implementations (THE [The Hessling Editor], S/EDIT, and uni-XEDIT/uni-SPF). Note that THE and Regina REXX are free. For more information, see the sidebar.

On an ongoing basis, Internet mailing lists VMESA-L and LINUX-390 are invaluable for both Linux and mainframe staff. Hosted at Marist College and the University of Arkansas, they offer friendly, helpful peer assistance with z/VM and Linux problems.

LONG-TERM HEADACHES

All of the aforementioned issues, however, are short-term pain, eventually cured through learning. More significant are long-term issues such as DASD, system, and user management.

Each Linux guest requires a new Linux install — typically about 2GB of data. This takes a while to load, and seems wasteful of both DASD (which is cheap these days, but hardly free) and time. There are ways to share significant amounts of Linux data read/only, but they are non-trivial without vendor products or significant Linux experience.

Linux performance management is still somewhat a black art, particularly on zSeries. Besides the fact that the few Linux tuning APIs are poorly documented, there is a larger issue: Linux was built for single-user machines, so it assumes it owns the entire physical system. This can make it a greedy guest.

For example, it uses all available memory to cache file buffers, which is probably not particularly wise since z/VM caches data in memory and so do most hardware controllers. Particularly when read/only file sharing is in effect, this means that a given byte of read/only data could exist in multiple Linux caches as well as z/VM minidisk cache and hardware cache!

The solution is to keep virtual storage sizes for Linux guests as small as possible.  This usually means reducing storage until Linux starts swapping, and then adding a small amount more (or letting Linux swap a bit). Reducing storage this way prevents Linux from using storage for file buffers, which usually isn’t appropriate on z/VM. Linux guest throughput has been drastically improved by reducing virtual storage size — in one case, from 2GB to 64MB!

When idle, Linux is not particularly frugal with CPU. After all, it’s idle, and “unused cycles are wasted cycles.” Of course, on a shared system, this is not something well-behaved guests practice. By default, Linux wakes up every few milliseconds and looks for pending work. The result on z/VM is that an unfettered Linux guest never becomes idle in z/VM terms. Since Linux guests tend to have fairly large working sets (the number of pages of virtual memory actually in use) — typically 100 percent of their virtual storage size — this severely limits the number of Linux guests that can run simultaneously, particularly if virtual storage sizes are not carefully tuned.

The result is common and predictable: Things run fine until a magic number of guests is reached, and then they just seem to stop. At that point, the z/VM Control Program (CP) notices that it is going to overcommit real storage, so it declines to run some of the guests, instead putting them on the “Eligible list.” This is correct behavior: The alternative, with well-behaved guests, is to spend most of the system paging in each guest in turn, running it briefly, then paging it back out so the next guest can run. However, Eligible list processing is based on the assumption that guests will voluntarily become idle periodically, or will at least perform I/O or some other operation that will allow other guests to run. Since an idle Linux guest never becomes idle and does no I/O, the result is that it is the idle guests that wind up using all the CPU!

There is also a “patch” for the Linux kernel known as the “notimer patch.” This changes the Linux kernel behavior to avoid the wakeup loop when the guest is idle, and increases the number of viable Linux guests on a system by several orders of magnitude.

If Linux must swap, it should use VDISK (virtual disk-in-storage): pseudodisk space implemented via z/VM real memory and paging space. This provides the fastest performance while not allocating fixed amounts of real DASD.

In any case, carefully watch paging: With several large Linux guests, insufficient paging space can result in z/VM

ABENDs fairly easily. Long-term solutions to these issues are in development, but in the meantime, Linux is still perfectly usable, provided these problems are understood and compensated for.

DON’T BE SWAYED BY FEAR, UNCERTAINTY, AND DOUBT!

With a weak economy, a recent war, and headlines about layoffs, many employees are nervous. When they hear someone discuss “server consolidation,” what they hear is “more layoffs.” Other issues, such as the recent SCO lawsuit against Linux and IBM for alleged misuse of Unix source code, as well as Microsoft and Sun publicity about why they believe Windows or Solaris are better, also can discourage consideration of Linux.

Linux on zSeries, however, is an opportunity for companies to save money and provide better service with the same staff (or to grow headcount more slowly), and for staff to learn new, marketable skills. While Linux may not be the right choice for every installation, it’s at least worth evaluating for most enterprises! Z

6 Pages