Apr 27 ’12

Integrating x86 Applications Into Linux on System z Environments

by Utz Bacher in z/Journal

With the advent of architectural plurality in zEnterprise, the term integration requires additional consideration. Introducing x86 and POWER-based blade servers in the zEnterprise BladeCenter Extension (zBX), zEnterprise shifts focus from the instruction set architecture question to workload needs.

A best-fit paradigm allows implementing business solutions on zEnterprise without debating what components will be part of the mainframe environment. The Uniform Resource Manager (zManager) provides common platform and virtualization management, and adds higher-level awareness such as policy-driven workload management and optimization. Its scope evolves from the Hardware Management Console (HMC) as a management console to the role of a management server for the entire multi-architectural environment of zEnterprise. An Application Programming Interface (API) opens this functionality to additional management software as well as roll-your-own automation.

From the platform, Operating System (OS) and workload perspective, System z always provided a holistic view beyond the platform's individual server instances. This is sufficient tradition to explore ways to bring the Linux worlds—System z and x86—closer together with zEnterprise. Extending converged operations from the zManager to the OS and application levels paints the picture of a Linux image with both System z and x86 resources. Applications of both natures can be operated in this environment, enabling a consolidated view on mixed architecture business solutions.

The application integration technology explained here centers this hybrid view around a Linux on System z environment. Linux on System z controls the OS and application lifecycle, virtually providing a co-processor notion of zBX resources to Linux on System z. x86 applications continue to run in their x86 virtual server images, but appear as local processes to the system administrator. The x86 OS image is tied to the Linux on System z instance to enable synergistic management.

Advantages of such a hybrid system notion (and its use at the OS and application operations levels) include but aren’t limited to the following:

• It’s a consumable implementation of the best-fit approach.

• The operational unification reduces the complexity to run hybrid solutions.

• Existing processes around Linux on System z can be extended to the hybrid scope.

While the handling of x86 resources is folded into System z management, the underlying hardware platforms still provide their unique characteristics. The discussion about architecture set-wise placement of applications can focus on system characteristics, including capacity, scalability, reliability, availability, and cost-efficiency. Note that hybrid isn’t about leveling architecture environments; it’s about combining the intrinsic strengths of architectures. It’s about having the best of all worlds, and eventually tapping the x86 application ecosystem for Linux on System z environments.

Extending processes to the hybrid scope applies to procedures such as user management and failover setups as long as all state relevant for failover remains on System z file systems; it also applies to file management (e.g., synchronized backup and lifecycle management of data) and operational patterns such as automation.

The Next Wave of Consolidation

The rise of Linux on System z is closely related to a wave of consolidation projects building on the mainframe's virtualization. Efficient management of tens and hundreds of virtual servers, the ability to highly utilize resources, and the foundation of a powerful, resilient platform are mainframe advantages that helped move many distributed servers into System z environments. Some systems couldn’t be easily consolidated for cost-effectiveness or availability reasons, at least without reworking components of the solutions.

With zBX, zEnterprise can change this picture and complete consolidation cases. Now these servers can be integrated into the mainframe environment, and Linux on System z's application integration technology can reduce the complexity of managing this heterogeneous application landscape, also reducing the number of application management endpoints.

The operational focus is essential to the discussion of application integration; this technology is an ideal match for scenarios where business solutions in Linux on System z are extended by x86 components, centering the hybrid setup around System z procedures and setups. In contrast, when a zEnterprise environment embraces management patterns, procedures, and possibly personnel of both the distributed and System z worlds, or x86 virtual servers are mainly managed on an image granularity, then extending System z operational patterns to x86 nodes won’t add value. Also, x86 installations not related to Linux on System z back-end software probably won’t see significant benefit from operational integration in Linux on System z.

Dream and Reality

Two major design goals have dominated the development of application integration:

• Don't change the x86 and System z OS environments so existing Independent Software Vendor (ISV) certification statements remain valid; don't modify the kernel or system packages.

• Don't fool the system administrator; don't add too much magic that can sometimes translate to complexity.

With these goals in mind, a genuine single system image will remain unrealistic. However, a pragmatic implementation can come close to this vision in many respects.

The first step in integrating Linux on x86 components into a Linux on System z environment is to attach an x86 virtual server. This virtual server's definition and instantiation are performed through the zManager, and IP connectivity through the Intra-Ensemble Data Network (IEDN) is required between the System z and x86 instances. Installation of a SUSE or Red Hat Linux environment is also a prerequisite, along with the application integration rpm packages on both sides. Then, the x86 virtual server can be attached to the Linux on System z environment. This attachment established various integration points, some of which are explained in the following paragraphs.

The x86 node's root file system will be exported to System z via Network File System (NFS). NFS was a pragmatic choice since it was included in all distributions (see first design goal), and its concepts are widely understood (see second design goal). The NFS mount allows full access from System z to the whole x86 environment. It doesn’t allow the x86 node to access the System z node's file system; however, application integration accommodates optional sharing of directories of the System z virtual server with x86 nodes.

A common use case for sharing is home and data directories. To ensure access permissions are consistent, users and groups are kept in sync between the System z master and an attached node. The use of standardized lock files as honored by common Linux libraries and tools will prevent simultaneous and potentially conflicting changes to the user or group databases on both sides. Application integration can change the networking setup of the nodes to employ IP masquerading and establish the System z node as a Network Address Translation (NAT) gateway to x86 nodes.

To invoke an x86 application, the operator can simply start the binary accessible through the file system export. Application integration will get initiative since the x86 binary can’t be executed in Linux on System z directly and the binfmt_misc handler is called, catching this exception. This handler will start the binary on the attached node in its unchanged x86 environment. To represent the application in the Linux on System z environment, the binfmt_misc handler uses code that acts as a shadow process.

Shadow processes serve as proxies on System z to the x86 application's processes. In particular, they don’t consume significant amounts of cycles and memory. Real processes and shadow processes have a 1:1 relationship (see Figure 1). Many process attributes, including effective and real user and group IDs, arguments, environment, and resource limits are used to execute on x86 and are kept in sync between real and shadow processes; however, process IDs will be different on both sides (i.e., pid namespaces are disjoint). The name of the shadow process will correspond (and be kept in sync) to the x86 process name, but will be prefixed to indicate the x86 process ID and hostname.

Figure 2 shows how shadow processes can appear using the “ps” command. The x86 process lifecycle will be mirrored on System z, including events such as forking and process exiting (e.g., if a process on x86 forks, the shadow process will also fork). Threads are shadowed, too. Open files, as used for standard I/O and pipes, and terminal capabilities, are forwarded between shadow and real processes. Signals sent to the shadow process will be passed on to the real x86 process, completing the representation of x86 applications in Linux on System z.

The accurate mapping of real processes to shadow processes allows for monitoring and automation from the Linux on System z console. Resource consumption is provided through a merged /proc file system that includes System z and x86 processes. Application integration-specific tooling (e.g., ai_top) builds on this infrastructure for a consolidated view over System z and x86 resources such as CPU and memory. Examining this x86 resource usage from System z is an explicit task that provides a clear picture of local and remote resource consumption (second design objective).

Programs can be installed through executing installation executables, transparently installing applications on the x86 environment. Alternatively, rpm is a common software packaging and distribution format. Application integration introduces the notion of so-called meta-rpms: x86 rpms (32- or 64-bit) are wrapped in s390x-type meta-rpms. These meta-rpms can be installed on the System z host, and during their installation, the inside x86 rpm will be deployed and installed on an attached x86 node. This allows for a consistent package management procedure for both native s390x packages and meta-rpms.

A local query to the System z rpm database reveals the packages installed on attached x86 nodes through the presence of corresponding meta-rpms. Dependencies of x86 rpms can be mapped in the corresponding meta-rpm namespace. This allows for dependency resolution through meta-rpms on the System z host already, and enables the use of repositories and package managers such as Zypper or Yum.

Consolidated system logging on the System z side and synchronized clocks through a System z-based time server complement the operational unification aspects of Linux on System z. All these facets of application integration provide a combined OS environment that ties x86 operations into Linux on System z management. For some tasks, specific tooling is provided since full transparency can’t be achieved without semantical issues or significant kernel changes—contradicting the basic design goals.

Application integration is freely available for download from the Linux on System z developerWorks Web pages (see www.ibm.com/developerworks/linux/linux390/applint.html), including a manual explaining concepts and use of the packages. It has the status of a technology study, presenting an early phase of application integration to seek early feedback on future development directions. Some limitations may apply, and one of them is still a 1:1 relationship between System z and x86 nodes: one System z image can only attach one x86 image at a time, and one x86 image can only be attached to one System z image. Also, support is provided through an email address (mail to:applint@de.ibm.com) on a best-can-do basis; everything beyond this level needs discussion in the individual case.


How has this improved the operational situation in mixed architecture solutions? The challenge of handling hybrid environments has been reduced toward the complexity to manage a series of System z images—which many customers have under control today. It will enable consolidation scenarios that previously weren’t possible:

• Applications not available for Linux on System z, such as x86-only components of application suites, virus scanners, software optimized on instruction set specifics, or merely Oracle's Java Environment
• Compute-intensive workloads not running economically on System z (e.g., commercial High-Performance Computing [HPC] and deep analytics software working with System z back-end data), or Extract, Transform and Load (ETL) scenarios with complex transformations
• Home-grown applications that can’t be ported easily; reasons can include endianness issues, the lack of knowledge about the application's internals, and fear of touching the application.

Application integration doesn’t preclude higher-level management products. Where such software works against the resource types synchronized by application integration, x86 applications and resources will enjoy management transparently (e.g., for automation across the hybrid complex). When management software requires explicit introspection in x86 nodes (e.g., using agents), OS-level operation of these agents can still occur via Linux on System z.


While the zManager provides common platform and virtualization management across the hybrid environment, application integration will provide unified operations on the OS and application level. It integrates Linux on x86 workload into Linux on System z environments and lets you focus on a best-fit selection of application components. A question remains to be answered in each individual case: Does zEnterprise emphasize the integration of distributed components into System z from a workload and operational perspective, or will it focus on providing a common home for the System z and distributed worlds? There will always be reason for both.