May 1 ’07

Migrating to Linux on System z: Lessons Learned From the Province of Quebec’s Award-Winning Project

by Editor in z/Journal

The Province of Quebec (DGTIC), an internal government organization providing IT services to more than 125 departments and agencies, last year was able to migrate close to 200 Oracle databases and other types of services from Unix environments (SUN-Solaris and IBM-AIX) to Linux on System z. This migration will result in 90 percent reduction in Oracle licensing cots.

This project also resulted in the selection of DGTIC as winner of the 2007 Excellence in Technology Award from SHARE.

"The statistics behind the work done at The Province of Quebec over the past year tell a compelling story," says Martin Timmerman, president of SHARE. "It is phenomenal to see the real successes the organization has achieved through its hard work. It is a well-deserved recipient of an award that on a yearly basis offers proof of how SHARE can solve business issues through IT technology."

In 2007, 100 more Oracle databases (migrations or new implementations) are planned as part of the continuation of this highly successful project.

As a member of SHARE, DGTIC has attended many meetings. Through peer interaction at SHARE, DGTIC personnel greatly benefited from the cross-pollination of ideas and technologies. The Province of Quebec found SHARE's VM and Linux education invaluable in helping configure this solution. The networking aspects are somewhat complex due to the many different agencies supported by these various Oracle databases. This article describes some of the challenges of the project, the critical success factors, results, and lessons learned.

Challenges

One challenge was the complexity of providing data processing services, including networking, data processing, and telecommunications service to a large and varied group of departments and agencies across the Province of Quebec. The DGTIC provides services on many hardware platforms, including the mainframe, midrange, and PC servers. On the mainframe side, there are 19 z/OS images providing services such as DB2 and CICS.

"We have mainframe services such as z/OS and z/VM," says Marc Plamondon, mainframe software manager for the DGTIC. "We have five z/890s and one z/800 and a z9 EC, representing just under 5,000 MIPs. We also have 450 Unix and Windows servers encompassing 700 images on AIX, Sun, and some HP and Data General servers."

Developing the architecture for the project also was challenging. The architecture was subject to approval by the DGTIC architecture board and developed over a one-year period. During this phase, there was no live box on the floor. At the conclusion of the architecture phase, a z9 EC machine with five Integrated Facilities for Linux (IFLs) was purchased with 48GB of central storage. We also installed z/VM 5.2.0 and several Novell SUSE SLES 8 and 9 Linux virtual machines.

Regular bi-monthly reviews were conducted to ensure the architecture was understood and that it conformed to DGTIC standards for security, naming conventions, software levels, and network topology. The challenges during this phase were mostly around acceptance of the idea that virtual servers could achieve similar results as physical servers, especially on the mainframe. Security zone isolation, again, needed to be proved. These challenges were addressed through presentations, discussions, and training.

Security is provided in z/VM by the control program and RACF. Essential z/VM resources such as login passwords, mdisk linkage by rules-based authorization, and vswitch and vlan membership are protected by RACF. Linux security is provided by hardening, authentication extensions with the Pluggable Authentication Modules, and by periodic ethical hacking attempts. An outside security firm we engaged has been unable to hack or crack into our Linux virtual machines. Success!

The architecture addressed a key concern of the architecture board: client isolation.  Client isolation was proved to all levels of the staff by training sessions emphasizing the inherent abilities of z/VM and the IBM mainframes architecture of storage, CPU, I/O, and network isolation. Through this training, the board was sold on the strengths of the platform. Safe, isolated, and secure rules the day. IBM, with z/VM and hardware facilities, has been doing virtualization and isolation for 35 years.

Success Factors

DGTIC was paying for Oracle licenses on many servers in the midrange platform. In addition, maintenance costs for upkeep of all these servers was quite high. A strategic alternative was needed. As an existing large z/OS shop, DGTIC was a logical candidate for using Linux on z/VM.

“The DGTIC was already a mainframe shop with mature staff and processes,” says Jocelyn Hamel, advisory project manager from IBM who serves as the project manager for the proof of concept and mentor for the subsequent phases of the project. “A proof of concept of Linux on the z/VM platform was a calculated risk. Down the road, we saw that the risk was low and the financial gain high.”

The proof of concept found a home on a z/800 mainframe that was available due to another project being cancelled. IBM and the business partner responsible for the DGTIC, Novipro, agreed to extend the floor life of the z/800 at no cost, along with providing a free license of the z/VM operating system for several months.

The proof of concept was conducted using z/VM 5.1 along with SUSE SLES 8 in 2004. The goal of the proof of concept was to determine usability and the stability of the platform.

The proof of concept was positive. The systems and applications tested successfully without the need for recompiling or reinstallation. Performance factors weren’t a determining success factor; the business case was more important. It focused primarily on reducing the number of Oracle licenses. In addition, there were other clients successfully running Oracle on the mainframe, and based on DGTIC’s experience in the proof of concept phase, unloading and reloading data was relatively easy. Oracle proved to be the suitable application.

“The business case identified the potential applications for the client,” says Hamel, the project manager. “Areas included Oracle, WebSphere Application Server (WAS), TAM/LDAP, firewalls, and the portal. The first phase of the project with the most gain and least risk was Oracle. There was an existing customer base.”

The business case included a cost structure showing a reduction of TCO (software, hardware, and man power costs) of 30 percent per year. The z/VM-Linux project is scheduled to break even within three years after migrating 80 WAS implementations, which will be the second payback project in addition to the Oracle database savings. The entire z9 EC mainframe complex will be repaid within two years during the Oracle database phase with the Oracle instances migrated or created on the Z9-EC.

A critical success factor was technical training. More than 200 person days of courseware, including lectures and labs, was delivered. These courses were delivered multiple times to DGTIC staff, including systems programmers, Unix and Linux administrators, network programmers, planners, security staff, architects, and analysts.

Management sponsorship was another critical success factor. At the DGTIC, our sponsor was the operations director for all platforms. Management and executives were given briefings on the high-level architecture and on what the IBM z9/EC and z/VM could achieve and provide. Having an early win and a big win with disaster recovery success was vindication for our sponsor, and management approvals and interest in the project increased.

Results

Upon approval of the business case and the architecture, a z9 EC was purchased. This box is in place today with five Logical Partitions (LPARs) running z/VM and Linux. More than 35 production Linux virtual machines are running in one LPAR, providing database services to the government portal and other types of applications to a number of clients. There are another 100 Linux virtual machines running Oracle and providing services to development, test, lab, and other users.

An important part of the architecture was the design of the replication tool for the Linux virtual machines. The cloner at the DGTIC was written in-house and provides a 3270 interface that can create a live Linux machine within 10 minutes. The cloner does much more than disk copying; the interface lets the issuer choose a version of SUSE SLES, an Oracle release, vswitch and vlan membership, and an IP address.

In today’s production environment, our planned best practices are in place. Resource sharing is key. Besides the standard sharing provided by the hardware and CP (memory, CPU, minidisks, virtual networking), the DGTIC environment now allows for sharing the Linux /usr file system in all clones. This occurs through training the Linux virtual machine to CP LINK to the minidisk with /usr read-only, and by instructing Linux in the /etc/fstab to mount /usr read-only. This achieves the following three important goals:

• Reduces the amount of disk storage used

• Lets Linux machines share the same executables as much as possible

• Ensures that important programs and data files can’t be tampered with.

The DGTIC is a heavy user of OSA ports and vswitch networking. The DGTIC is using 13 OSA devices providing 26 OSA ports. It uses 40 separate networks. Each security zone is given its own OSA port and vswitch membership. Different clients can’t see each other’s network transmissions, yet memory and CPU resource sharing is maximized. Clients in the production zone have two OSA ports to allow network failover and redundancy.

Prior to z/VM and Linux, the mainframe environment provided mostly legacy applications. Now the applications can use z/VM with Linux as a platform choice competing well with Microsoft .NET and AIX. Looking ahead, z/VM with Linux will be part of the e-government service architecture including portal and Web. Oracle, WAS, Domino, and a hybrid TAM/LDAP are all options being considered. Many agencies are interested in the consolidation capabilities DGTIC has demonstrated.

z/VM with Linux should be a carrier for front-end, Service-Oriented Architecture (SOA) services. DGTIC can provide the bridge for .NET to legacy services; that represents the most economical method of integrating all the platforms for SOA.

Lessons Learned

Many valuable lessons have been learned. When performing proof of concepts involving performance characteristics or comparisons with other platforms, it’s important to plan the mainframe activities. The mainframe invariably won’t compare favorably if you take one application and plunk it into the mainframe. The mainframe will compare favorably when you run multiple instances of Linux servers and applications or if you have I/O-intensive applications.

Acceptance of the Oracle Linux virtual machine servers on z/VM has grown faster than planned; more than 100 servers were implemented. Today there are close to 200 Linux Oracle database instances with nearly 50 in production.

This environment is supported by a comparatively small staff. Two z/VM systems programmers under my tutelage support all the LPARS, z/VM, and vendor software. Two Linux system administrators support more than 100 Linux servers, a ratio of 50:1 virtual servers to a person. Additional personnel are required from time to time for security tasks, VM networking, disk storage management, automation, and performance monitoring and capacity planning.

On the technical side, we learned that Linux virtual machines like to consume resources, especially memory. A key factor in providing good response time is to provide a large amount of core memory and to have enough DASD paging space. Try to manage the z/VM paging space to never exceed 40 percent occupancy. Core memory should be allocated as central and a percentage as expanded storage. DGTIC runs with a maximum of 2GB of expanded storage, which provides good results. Your organization should find the optimal sizes for your workloads.

When doing a proof of concept, be careful of comparisons with other platforms. The hardware and software on the mainframe are optimized toward multi-tasking, multi-programming and I/Os, so avoid running one instance of Linux in a proof of concept. Rather, running multiple Linux virtual machines with the chosen applications will showcase the mainframe in a favorable, yet realistic fashion. After all, if using z/VM, you’ll most likely be using it to run many instances, not one.

When benchmarking Linux virtual machines, it’s important to avoid a “drop in and go” method. Since the mainframe is an intensively shared environment, Linux storage sizes should be carefully calibrated so they’re doing a small amount or zero swapping under normal workload conditions. Using the same storage sizes from a non-shared Intel server as your Linux virtual machine size is invariably a mistake, so avoid doing that.

Conclusion

The DGTIC project was highly successful. Interest from the DGTIC clients is quite high and the organization is ready to meet the upcoming demand.

“The Province of Quebec without question established a standard of excellence over the past year,” says Michael Bliss, IBM’s System z Technical Sales Support. “IBM is proud to see an organization experience wins to this degree using our technology in combination with SHARE’s educational opportunities.”