Dec 10 ’08
State of Oklahoma Department of Human Services: Making a Difference for the Business & Consolidating Servers
The State of Oklahoma Department of Human Services (OKDHS) delivered an application that might sound too good to be true—but isn’t. It’s a difference-maker for the business that achieved green initiative goals, lowered Total Cost of Ownership (TCO), eased management of system assets and their retirement cycles, and was delivered seamlessly and transparently to users.
Serving more than 100,000 children per year, the OKDHS was looking for more effective ways to orient field staff to results-driven performance. A major part of the effort was giving field and central office staff ready access to the department’s data repository for child welfare (known as the KIDS system) with new and highly agile reporting tools.
Four years ago, it took OKDHS users in field offices and headquarters up to three weeks to produce a usable report. Today, these same users can define and produce reports in 24 hours of programming time, obtaining results that are both standard number- and data-oriented reports and pictorial representations of the data, such as pie and bar charts. The new reports let staff quickly see how they’re doing with their child caseloads—and target any critical areas needing immediate response.
Agile reporting was achieved by moving staff from traditional FOCUS (from Information Builders) on an HP UNIX platform to its WebFOCUS product residing on virtualized Linux on System z. This virtualized migration from HP-UX to Linux on System z occurred transparently to OKDHS staff. OKDHS IT coordinated renovation of the reporting function with retirement of aging UNIX assets, simultaneously gaining advantage from new system flexibility and capabilities on System z that simplified asset and workflow management.
The Need for Consolidation
The original migration and consolidation goal for OKDHS was to move the child welfare system from HP-UX to Linux on System z. “We run 15 different systems on the System z,” says Chris Little, OKDHS z/VM administrator. “One of the goals we had for this HP-UNIX to Linux on System z migration was to also migrate from legacy FOCUS on HP-UX to a newer reporting tool on Linux on System z that would make report requests and production easier for our end business users.”
There were several phases—and objectives—in the HP server consolidation project to System z.
“First, DHS had mission-critical systems that it wanted to better position for end business users so they could go about the work of serving the residents of the State of Oklahoma,” says Little. “This involved finding a more user-friendly reporting tool that happened to run in the Linux on System z environment and that our users could learn as they achieved immediate results. At the same time, OKDHS was reaching the end of its technology lifecycle for many of its servers, including the HP-UX machines.”
OKDHS IT elected to take a slow but sure route to the transformation of its IT architecture.
Facing Early Fears
The first task was discussing the new server consolidation direction with users to obtain their buy-in and support.
“Those of us on the business side had to be convinced that virtualization was going to benefit us,” says Bill Hindman, Programs Division IS administrator for the OKDHS Children and Family Services Division. “Previously, end business users had their own dedicated boxes for processing information and reports. Suddenly, we were being told these servers were being eliminated and that our applications were being moved to the mainframe. We also heard we would now be ‘sharing’ computer resources with other parts of the agency.”
The announcement of the consolidation strategy was initially met with understandable trepidation from users. “We wanted to be sure our needs would continue to be met with the new mainframe strategy, and that the resources would be there,” says Hindman.
OKDHS IT also had its share of challenges and fears.
IT wanted to replace an older reporting system and database residing on HP-UX with new reporting tools and data that resided in the Linux on System z environment.
“When we made the decision to port the KIDS system to the System z, we developed an architectural approach that took advantage of placing Web-facing report design tools on Windows servers and migrating all of the actual report processing to Linux on System z,” says Little. “We knew this new solution would produce greater turnaround on reporting for the division’s end users, and provide greater and less complicated access to data repositories that already were on the mainframe.”
As an early entrant into server consolidation, however, OKDHS IT had no way of absolutely knowing whether a complete toolset for the planned migration existed at every level of IT infrastructure.
“Not everything was in place for z/VM and Linux when we began our project three or four years ago,” says Little. “We worked closely with IBM since we knew that a number of elements of the environment weren’t yet fully mature. This was one reason why the migration project took as long as it did. … We carefully proceeded, beginning with a migration of smaller databases to the System z that weren’t mission-critical. A total of 10 people were assigned to this project, but none of them was a dedicated resource, as we were all splitting time between working on the migration and working on projects in our actual production environment.”
The migration project team held weekly conference meetings. They opted to use Tivoli Storage Manager as an integral part of their data backup solution, but they also recognized they were “leading edge,” since there were no other clients available to exchange ideas with when they first started the migration with this backup strategy.
“We encountered several obstacles,” says Little, “One of these involved the backup environment. We were using the RMAN interface for backup and recovery of our Oracle database and eventually we used Tivoli Storage Manager. Unfortunately, we found that some infrastructure tools were missing. Our distribution of SUSE Linux had supportive tools in place but we had a version of GCC [GNU Compiler Collection] that had to be built from scratch for Oracle.”
OKDHS IT actually found they needed two compilers. One was GCC, which linked the Oracle libraries and ran on SUSE Linux Enterprise Server (SLES). The other was a Micro Focus COBOL compiler required to support KIDS system interfaces in the Linux on System z environment.
It was GCC, which, at the time, did not ship with a service pack that fully supported the System z environment.
IT waited for the puzzle to fall into place, piece by piece, working methodically. The SUSE GNU compiler issue was resolved once the correct service pack arrived. This secured Oracle’s guarantee of support for the new environment on System z. Looking for every way to safely expedite the process, OKDHS IT was still able to “work ahead” by jump-starting problem resolution with a download of a copy of the GNU compiler so IT could build in the IBM patches that were needed for System z. Downloading and patching allowed IT to cautiously move forward, even as it was waiting for the correct service pack to arrive.
“We also encountered a major challenge with the data recovery and backup process,” says Little. “This process included parts of Oracle and Tivoli, and although the hurdles we faced were purely technical, they took a long time to overcome.”
Little and other DHS IT managers acknowledge that backup for the new virtual environment became a highly detailed process that involved the creation and documentation of multiple procedures and checkpoints.
“To facilitate backups, we took snapshots of every logout because, at first, we were getting a lot of corruption,” says Little. “We used flash copying with the IBM stack and took snapshots of the DASD. We still do this. The Tivoli Storage Manager hooks into the Oracle RMAN [Recovery Manager] utility to complete the backup mechanism.”
Little says that recovery and database backup became major challenges because there just wasn’t a good unified online solution at the time that would support RMAN, which is Oracle’s primary data backup utility.
As they overcame technical hurdles that were symptomatic of being an early adopter of a server consolidation strategy, OKDHS IT staffers also had to face their fears of the unknown. The level of fear varied, depending on the person with whom you talked.
“Some of the IT staff was very apprehensive,” says Aleta Seaman, database manager. “All of this was bleeding-edge technology when we started our consolidation. Still, both on the operating system and the hardware sides, people were excited about the possibilities.”
Little agrees. “A big part of the project was achieving a comfort level,” he says. “At the time we started our migration, Linux was still a dirty word in much of the IT industry. Linux was considered a ‘hobby’ or a ‘toy’ system then. We knew that the end result was to bring missioncritical applications to a new platform—and along the way, we had to feel comfortable by first performing the database migrations with databases that weren’t considered missioncritical. In this way, if an outage occurred and we had to move back to our original platform, we could do so without impacting a lot of people.”
OKDHS IT ran through its migration strategies many times to reassure IT staff and users. When appropriate, IT repeatedly ran its test suites, taking every precaution to ensure the right results came back from the tests. As a support “backbone,” both Oracle and IBM committed resources for the project.
“We were pleasantly surprised at how easy it was to move from the HP-UX platform to Linux on System z,” says Little. “We had anticipated this might involve significant retraining for staff, but it didn’t. One side benefit we hadn’t even hoped for was that so many of the scripts previously written for the HP-UNIX platform still ran without much modification on Linux on System z.”
Reaping Extra Benefits
When OKDHS IT started its server consolidation project four years ago, the department was still running a System z900 class machine. Today, OKDHS runs a System z9. From the purely IT side, OKDHS has seen benefits beyond the delivery of more facile applications to users.
“We were so close to the end of our lifecycle on HP-UX, and the hardware was getting very expensive to support,” says Sunni Majors, director of Enterprise Technical Services. “By moving Oracle to the Linux System z platform, we were able to restructure costs for CPUs and move away from licensing by user. This gave us a better return on our investment, since we experienced significant savings in our hardware and software licensing costs by consolidating resources onto a single platform.”
OKDHS IT also was pleasantly surprised at the ease of the server and application migration to Linux on System z, although it remains vigilant when it comes to asset management and TCO throughout its IT architecture.
“There is more to assessing the cost of doing a server consolidation and application migration than simply eliminating servers and licenses,” says Little. “You also have to assess where you will have to add incremental CPU memory to the System z, which is comparatively expensive when compared to similar costs in the Linux/ Intel environment. This is why we place a priority today on being extremely efficient with our System z resource utilization. The bottom line is [that] you have to make sure the way you scale your resources matches your ability to invest.”
Checking in With the Business
The OKDHS KIDS system is the bedrock of its child welfare program and is instrumental to that program’s administration.
“We receive 65,000 reports of child neglect and abuse each year, and track more than 100,000 children,” says Hindman. “When you extend this information to families, the amount of data is exponential.”
Hindman acknowledges that the end-to-end migration effort initiated by OKDHS IT was entirely transparent and seamless to users, who also have seen improvements in the level of computing service and in what they can do with their program reporting.
“We’re in the process now of planning a new enterprise application where our child welfare and family support systems will all be in one application on a single mainframe platform instead of siloed on different platforms,” says Hindman. OKDHS IT also is planning to move its data warehouse, now on 39 separate Windows servers, to the System z.
“We’re continuing to add functionality to our Linux on System z environment,” Little adds. “We’re now playing a little ‘catch up’ to handle all the changes in applications that we have undergone. The changes we’re adding are beginning to stress the load on the system to where we’re considering a possible CPU investment.”
OKDHS IT also has a proven methodology for reliability and backup, which was an early migration concern.
“On the HP-UX side, we just did the database backups via a SCSI [Small Computer System Interface] tape drive that was attached to the server,” says Little, “But on the System z, Tivoli Storage Manager was, and is, our primary enterprise-side backup solution.”
New agile reporting is making a difference in OKDHS’ ability to track results and outcomes, and they plan to continue to grow this reporting environment and the data warehouse access that accompanies it. Over the next few years, the department plans to add Key Performance Indicators (KPIs) as it continues to grow its data warehouse in the Oracle/Linux on System z environment.
“Linux on System z provides OKDHS with a flexible and scalable environment that’s also cost-effective,” says Sarjoo Shah, Data Services director. “OKDHS intends to leverage the Linux on System z environment to provide faster and agile solutions for its division through the use of OLAP [Online Analytical Processing]-based tools and the Web enablement of applications.”
Users also are pleased with future projects and directions.
“I spent time as an implementation manager with OKDHS in 1994, so I wasn’t just going to say ‘yes’ to mainframe virtualization,” says Hindman, “But I also have been with the agency for 30 years, and I had to look at the project from the standpoint of what was best for DHS. … I can say that moving to the mainframe was the right thing for us to do. We have the resources and the skillsets there, and the application consolidation has simplified our operations. The process of migrating off other platforms and onto the mainframe was totally seamless, and didn’t negatively impact end users. I would have to say we are very satisfied on the business side.”