IT Management

Solution Showcase

2 Pages

In addition, disaster recovery procedures were put in the hands of individual departments. “If the user or department wanted the ability to recover data in a reasonable amount of time, they had to do their own backup,” Strozyk said. “ Recovering data from the distributed backup system could take days. The departments were responsible for pretty much their own recovery of individual files that were inadvertently deleted or may have been lost or corrupted.”

In an effort to get its arms around these glaring gaps in backup and storage management, M&T Bank delegated these responsibilities to its mainframe data center, which was experienced in automated, unattended backup procedures. “M&T Bank saw the need to get a sound backup philosophy in place; one that could be certified and verified for the open systems platforms,” Strozyk said.

Distributed storage solutions

After reviewing various approaches to distributed storage management, M&T Bank implemented FDR/UPSTREAM and UPSTREAM/SOS solutions, an EMC Symmetrix disk subsystem, and a SAN in the year 2000. The bank is now near completion of bringing management of distributed systems storage into the mainframe data center. “One of our requirements was to get a backup solution where we could leverage our mainframe tape subsystems,” Strozyk said. “We did not want to have to dedicate tape drives just to the backup process. The FDR/ UPSTREAM architecture allowed us to make use of resources we already had. It just didn’t make sense to purchase 10 or more mainframe-class tape drives.”

The net result of the deployment of UPSTREAM and related tools has been a consolidation of storage management from across the bank’s various departments. The benefits of the mainframe-based storage approach are very tangible, Strozyk noted. “We have absorbed the backup tasks into our existing enterprise storage department, and are now able to provide a reliable backup and a verifiable recovery process, which is something we didn’t have before.”

The largest contingent of files out on the bank’s network is the bank’s e-mail and messaging files, which are generated by Novell GroupWise. All of the bank’s employees access the GroupWise system, and are therefore protected by UPSTREAM. UPSTREAM is now seen throughout the organization as the bank’s backup tool of choice, since the tool protects everyone in the company as it backs up e-mail and messaging files. “Everyone’s quite thrilled with the results,” Strozyk said. “If you talk about backup and recovery for open systems, people automatically think of FDR/UPSTREAM.”

“The system currently backs up more than 1.2TB a week, typically with increments of up to 400GB,” said Mark Riggs, senior staff specialist in the Enterprise Storage Group of M&T Bank’s Systems Technology Department. UPSTREAM provides a full merge backup capability for applications, which only requires a first-time full backup to the mainframe-attached storage. All future backups are then incremental and full merge backups. The UPSTREAM Full Merge Backup facility drastically reduces the elapsed time for full backups by utilizing already existing file backups and only opening and transmitting files that have changed. In addition, full-merge backups at M&T are run weekly to reduce the amount of data transmitted. Each Novell file server contains about 1.2 million files, or about 200GB of data, he noted.

Using UPSTREAM, the data center also has the capability to perform an archival operation on all file systems across the bank. Based upon a set of user-defined rules, files that aren’t accessed within a given period of time are automatically archived to tape, and deleted from the server. The UPSTREAM archival tool used for space management can also provide auto-recall capabilities for Novell file servers by putting a stub file in place of the file.

Data is then vaulted, on a daily basis, to STK 9490E or 9840 tape drives supported within the mainframe tape silo. Typically, each tape holds about 20GB of GroupWise data, Riggs reported. Through a vaulting tape stacking process, UPSTREAM makes a secondary tape copy of the server data, which is shipped, daily, to an off-site facility for disaster recovery purposes.

M&T Bank also employs an agent that enables mainframe-based storage of data from its Oracle databases. UPSTREAM interfaces with Oracle Recovery Manager (RMAN) to back up Oracle data, but RMAN is still administered by the bank’s database administrators. These DBAs create the RMAN scripts that manage the Oracle backup process to the mainframe environment. M&T Bank’s DBAs appreciate the automatic interface between UPSTREAM and RMAN, Strozyk noted. “It gives them quite a bit of control.” Strozyk’s department is also testing an UPSTREAM plug-in that provides the same capabilities to the bank’s SQL Server database environments. In addition, the bank is testing UPSTREAM’s GroupWise agent, which performs backups while GroupWise is up and online. Since the agent supports incremental backups for changed data, it significantly reduces the amount of data backed up on a daily basis.

FDR/UPSTREAM provides comprehensive storage management facilities that include extensive disaster recovery features and protection. Fortunately, M&T has not experienced a major incident that has required a real-world restoration of multiple servers, Strozyk reported. However, daily file issues do occur and the bank’s e-mail, messaging, and other distributed server functions for file recovery can be brought back up in minutes, and if something major does happen, M&T is prepared. In extensive and realistic tests, Strozyk and Riggs have purposely disabled servers and successfully and rapidly restored the systems. “When we brought UPSTREAM in, it went through quite a few iterations of different types of scenarios for backup and recovery,” Strozyk related. “We’ve been able to put together some standard end-to-end procedures for recovering and rebuilding major file servers. Everything went very smoothly, and came up on a totally brand new server.”

Most of M&T Bank’s backup needs thus far have been individual user files that have required restoring. Such recoveries have been near instantaneous, managed from the UPSTREAM Director, a Java-based graphical user interface. “With UPSTREAM Director, restores have been fairly straightforward and centrally managed,” Strozyk said. “You have access to every single backup that’s been done from one point, for every server.”

Backup for each distributed server occurs automatically, on a daily basis, via a batch job initiated by CA-7, an automated scheduling system. UPSTREAM will provide notification of backup events to operators’ consoles. “Everything’s automated, and if a backup does fail for some reason, the mainframe operators are immediately notified,” Strozyk explained. “There is no longer any manual monitoring of this process.” In addition, new updates of the UPSTREAM software can be automatically installed across the enterprise via the UPSTREAM automatic upgrade feature.

The next challenge

The storage management team is currently the owner and administrator of the UPSTREAM tool. The next major challenge for Strozyk’s department is organizational — to accelerate the workflow related to restoration of files and systems — the only remaining bottleneck to rapid recovery. “Currently, by the time the end user notifies the help desk and a trouble ticket is issued, the process takes about a day,” Strozyk said. “We plan to train the help desk on the recovery processes. That way, when a user calls because he or she needs a file restored, the help desk would have the ability to do it right on the spot. Users will have their files back within a short period of time.”

For more information on FDR/UPSTREAM and UPSTREAM/SOS and the complete family of FDR storage management products, contact Innovation Data Processing at 973-890-7300, or visit

2 Pages