Jul 1 ’03
As M&T Bank of Buffalo, NY, completes its acquisition of Allfirst Bank of Baltimore, MD, the newly combined institution is growing into one of the top-20 largest banks in the U.S. M&T Bank’s asset base is jumping from $33 billion to $49 billion, and the financial institution is adding 262 branches to its current network of 470. The acquisition also adds some 5,000 employees, increasing the bank’s workforce to about 14,000 tellers, customer service representatives, managers, and technical support personnel.
Meeting the Challenge
Combining two organizations of such size, of course, is challenging on many fronts. One issue M&T Bank doesn’t have to worry about, however, is additional backup and storage requirements that arise as it integrates Allfirst’s IT systems into its operations. That’s because M&T Bank already has a centralized enterprise storage management system in place, administered from its mainframe data center. The tools employed, FDR/UPSTREAM and UPSTREAM/SOS from Innovation Data Processing, are an intelligent storage management system featuring enterprisewide backup and recovery for open systems servers to z/OS- or OS/390- attached mainframe storage. “We expect to quickly integrate Allfirst’s systems into our backup and restore facility,” said Ronald Strozyk, manager of the Enterprise Storage Management Group at M&T Bank.
FDR/UPSTREAM backs up, manages, and restores data from workstations and servers from across the enterprise to all brands of mainframe-attached tape and disk storage subsystems. Data is transferred over SNA/APPC, TCP/IP, and local and wide area network connections. UPSTREAM/ SOS is designed to back up and restore open-systems data stored exclusively on EMC Symmetrix 5000-series and 8000-series storage subsystems. UPSTREAM/SOS transfers data between the Symmetrix and the MVS host across the mainframe’s native ESCON or FICON channel connection.
The bulk of M&T Bank’s core processing takes place on an IBM 9672-R66 mainframe, which is being upgraded to a z900 2064-1C4 system. Storage is managed within the EMC Symmetrix disk array, which supports more than 11TB of data and is connected to the enterprise via a Storage Area Network (SAN). Most of the bank’s data originates from more than 275 Novell and Windows 2000/NT servers, a large number of which are configured as clusters of four servers. The bank also maintains a number of Unix servers running Sun Solaris and AIX. M&T Bank is considered one of the country’s strongest and most highly regarded regional banks, with branches throughout New York, Pennsylvania, Maryland, and West Virginia, as well as a comprehensive set of channels via ATM, telephone, and Internet-based banking. In December 2002,
M&T Bank shareholders approved the acquisition of Allfirst Bank from Allied Irish Banks PLC. The acquisition, completed in the first quarter of 2003, will expand M&T Bank to approximately 700 offices throughout the mid-Atlantic region, placing M&T Bank among the 20 largest banks in the U.S.
Efforts are also under way to link Allfirst’s distributed systems — which consist of Windows 2000/NT, NetWare, Solaris, NCR, HP-UX, and Red Hat Linux servers — to M&T’s mainframe data center running FDR/UPSTREAM. “Allfirst had two data centers in the Baltimore region, and our plans are to keep one of those as a disaster recovery site, at a minimum,” Strozyk said. “This provides some new options as to how we architect and set up UPSTREAM.”
CENTRALIZING BACKUP STORAGE
Before M&T Bank was able to centralize its backup and storage operations to the mainframe, the financial institution faced many of the same issues encountered by companies with rapidly proliferating networks of distributed servers across departments. The bank relies on PC servers to support its Novell GroupWise enterprise e-mail and messaging infrastructure, as well as key customer service functions. “Our server farms got bigger and more spread out across New York state. Things just became quite unmanageable,” Strozyk recounted.
The institution’s network computing group, which oversees PC and Unix system deployments, had some backup and storage management solutions in place, but information from various servers was too scattered. “There were reliability issues. Backups — no matter where they are — often fail,” he explained. “We just didn’t have a good monitoring mechanism in place where someone could identify if a backup failed, and take corrective action.” It often took days of tracking and analysis to determine if and where glitches were occurring. “Administrators had to pour through hundreds of pages of backup return codes and messages to verify that the backups were successful,” Strozyk said. “That didn’t bode too well for being able to successfully restore or recover servers.”
In addition, disaster recovery procedures were put in the hands of individual departments. “If the user or department wanted the ability to recover data in a reasonable amount of time, they had to do their own backup,” Strozyk said. “ Recovering data from the distributed backup system could take days. The departments were responsible for pretty much their own recovery of individual files that were inadvertently deleted or may have been lost or corrupted.”
In an effort to get its arms around these glaring gaps in backup and storage management, M&T Bank delegated these responsibilities to its mainframe data center, which was experienced in automated, unattended backup procedures. “M&T Bank saw the need to get a sound backup philosophy in place; one that could be certified and verified for the open systems platforms,” Strozyk said.
Distributed storage solutions
After reviewing various approaches to distributed storage management, M&T Bank implemented FDR/UPSTREAM and UPSTREAM/SOS solutions, an EMC Symmetrix disk subsystem, and a SAN in the year 2000. The bank is now near completion of bringing management of distributed systems storage into the mainframe data center. “One of our requirements was to get a backup solution where we could leverage our mainframe tape subsystems,” Strozyk said. “We did not want to have to dedicate tape drives just to the backup process. The FDR/ UPSTREAM architecture allowed us to make use of resources we already had. It just didn’t make sense to purchase 10 or more mainframe-class tape drives.”
The net result of the deployment of UPSTREAM and related tools has been a consolidation of storage management from across the bank’s various departments. The benefits of the mainframe-based storage approach are very tangible, Strozyk noted. “We have absorbed the backup tasks into our existing enterprise storage department, and are now able to provide a reliable backup and a verifiable recovery process, which is something we didn’t have before.”
The largest contingent of files out on the bank’s network is the bank’s e-mail and messaging files, which are generated by Novell GroupWise. All of the bank’s employees access the GroupWise system, and are therefore protected by UPSTREAM. UPSTREAM is now seen throughout the organization as the bank’s backup tool of choice, since the tool protects everyone in the company as it backs up e-mail and messaging files. “Everyone’s quite thrilled with the results,” Strozyk said. “If you talk about backup and recovery for open systems, people automatically think of FDR/UPSTREAM.”
“The system currently backs up more than 1.2TB a week, typically with increments of up to 400GB,” said Mark Riggs, senior staff specialist in the Enterprise Storage Group of M&T Bank’s Systems Technology Department. UPSTREAM provides a full merge backup capability for applications, which only requires a first-time full backup to the mainframe-attached storage. All future backups are then incremental and full merge backups. The UPSTREAM Full Merge Backup facility drastically reduces the elapsed time for full backups by utilizing already existing file backups and only opening and transmitting files that have changed. In addition, full-merge backups at M&T are run weekly to reduce the amount of data transmitted. Each Novell file server contains about 1.2 million files, or about 200GB of data, he noted.
Using UPSTREAM, the data center also has the capability to perform an archival operation on all file systems across the bank. Based upon a set of user-defined rules, files that aren’t accessed within a given period of time are automatically archived to tape, and deleted from the server. The UPSTREAM archival tool used for space management can also provide auto-recall capabilities for Novell file servers by putting a stub file in place of the file.
Data is then vaulted, on a daily basis, to STK 9490E or 9840 tape drives supported within the mainframe tape silo. Typically, each tape holds about 20GB of GroupWise data, Riggs reported. Through a vaulting tape stacking process, UPSTREAM makes a secondary tape copy of the server data, which is shipped, daily, to an off-site facility for disaster recovery purposes.
M&T Bank also employs an agent that enables mainframe-based storage of data from its Oracle databases. UPSTREAM interfaces with Oracle Recovery Manager (RMAN) to back up Oracle data, but RMAN is still administered by the bank’s database administrators. These DBAs create the RMAN scripts that manage the Oracle backup process to the mainframe environment. M&T Bank’s DBAs appreciate the automatic interface between UPSTREAM and RMAN, Strozyk noted. “It gives them quite a bit of control.” Strozyk’s department is also testing an UPSTREAM plug-in that provides the same capabilities to the bank’s SQL Server database environments. In addition, the bank is testing UPSTREAM’s GroupWise agent, which performs backups while GroupWise is up and online. Since the agent supports incremental backups for changed data, it significantly reduces the amount of data backed up on a daily basis.
FDR/UPSTREAM provides comprehensive storage management facilities that include extensive disaster recovery features and protection. Fortunately, M&T has not experienced a major incident that has required a real-world restoration of multiple servers, Strozyk reported. However, daily file issues do occur and the bank’s e-mail, messaging, and other distributed server functions for file recovery can be brought back up in minutes, and if something major does happen, M&T is prepared. In extensive and realistic tests, Strozyk and Riggs have purposely disabled servers and successfully and rapidly restored the systems. “When we brought UPSTREAM in, it went through quite a few iterations of different types of scenarios for backup and recovery,” Strozyk related. “We’ve been able to put together some standard end-to-end procedures for recovering and rebuilding major file servers. Everything went very smoothly, and came up on a totally brand new server.”
Most of M&T Bank’s backup needs thus far have been individual user files that have required restoring. Such recoveries have been near instantaneous, managed from the UPSTREAM Director, a Java-based graphical user interface. “With UPSTREAM Director, restores have been fairly straightforward and centrally managed,” Strozyk said. “You have access to every single backup that’s been done from one point, for every server.”
Backup for each distributed server occurs automatically, on a daily basis, via a batch job initiated by CA-7, an automated scheduling system. UPSTREAM will provide notification of backup events to operators’ consoles. “Everything’s automated, and if a backup does fail for some reason, the mainframe operators are immediately notified,” Strozyk explained. “There is no longer any manual monitoring of this process.” In addition, new updates of the UPSTREAM software can be automatically installed across the enterprise via the UPSTREAM automatic upgrade feature.
The next challenge
The storage management team is currently the owner and administrator of the UPSTREAM tool. The next major challenge for Strozyk’s department is organizational — to accelerate the workflow related to restoration of files and systems — the only remaining bottleneck to rapid recovery. “Currently, by the time the end user notifies the help desk and a trouble ticket is issued, the process takes about a day,” Strozyk said. “We plan to train the help desk on the recovery processes. That way, when a user calls because he or she needs a file restored, the help desk would have the ability to do it right on the spot. Users will have their files back within a short period of time.”
For more information on FDR/UPSTREAM and UPSTREAM/SOS and the complete family of FDR storage management products, contact Innovation Data Processing at 973-890-7300, or visit www.innovationdp.fdr.com.