May 1 ’03
Integrating Legacy Applications as Web Services
Web services promise to lower the costs of integration and help legacy applications retain their value. This article introduces you to the basics of Web services, how they reduce costs and how you can use them to integrate mainframe applications with other enterprise applications.
Web services are platform-independent interfaces that allow communication with other applications using standards-based Internet technologies, such as HTTP and XML. They provide an opportunity for organizations to reduce the costs and complexities of application integration inside the firewall and create new possibilities for legacy applications to participate in e-business.
Proponents of Web services talk about a new era of interoperability. However, haven’t we heard all of this before? Applications using standardized interfaces such as Common Object Request Broker Architecture (CORBA), Java 2 Enterprise Edition (J2EE) or Microsoft’s Distributed COM (DCOM) were supposed to deliver the same benefits but fell short. Nonetheless, each of those interfaces has platform-specific and vendor-specific components that keep it from being technology-neutral. Therefore, companies that adopted CORBA interfaces were not immediately compatible with partner networks that used either DCOM or J2EE. However, Web services are standards-based, platform-independent, and the course Web services take is controlled by independent standards bodies, not individual corporations pushing their own technologies. Web services simplify the complexity of integration by reducing the number of Application Programming Interfaces (APIs) and the number of data formats to one.
Web services sound promising for new applications that have support for Internet technologies built-in, but what about legacy systems? How can they benefit from Web services integration and what does it take to integrate legacy applications as Web services? First, let’s discuss Web services in general, and then we will delve into their relevance to legacy applications.
Standards-Based and Platform-Independent
While applications have been able to communicate using Internet technologies for years, only recently have standards evolved to allow distributed applications to talk to each other. Web services depend upon three key open standards to allow them to communicate regardless of the hardware they run on, operating system they run under, or programming language used to produce them (see Figure 1).
Building on the foundation provided by Simple Object Access Protocol (SOAP), Web Service Definition Language (WSDL), and the Universal Description, Discovery and Integration (UDDI) standard, there are emerging standards for handling complex business process workflow, authentication, and message integrity. These transactions span multiple enterprises and long-lasting transactions to ensure that the outcomes of the transactions are reliable. Underlying all of this is the use of XML as the message format and a standard protocol such as HTTP as the transport method.
Standards-based components diminish the costs and skills required to integrate applications within an organization or between partners. Because there is widespread vendor support for Web services, interoperability among vendor solutions will improve. However, the greatest benefits of standards-based Web services for developers and administrators are reduced complexity and increased flexibility of integration architectures.
Reducing Complexity and Increasing e-Business Flexibility
The number of APIs and data formats that developers must learn and support affects the complexity and flexibility of integration. The more interfaces and formats, the greater the cost to implement and maintain an integration framework. SOAP provides a common interface to applications and XML provides a common data format — together, they simplify integration and lower integration costs.
Fewer Application Interfaces
Traditionally, developers create interfaces for each application in an integration project to allow inter-application communication. Because the APIs for each application are different, companies integrating large numbers of applications develop complex integration frameworks that require specialized development skills and lead to increased integration costs. Increased complexity also makes integration architectures less flexible, so that changes in the network and business processes take longer and become more difficult.
Reducing the number of APIs reduces complexity and increases flexibility. Web services based on Internet technologies are not specific to any platform and provide a single API for any application to use. They are loosely coupled and can be invoked directly as traditional APIs, or requests can be sent to a queuing system where transactions can occur at specified dates or times. (Integration using traditional APIs is tightly coupled in that each application accepts input and output using unique data formats. They are usually invoked synchronously and expect to receive an immediate response.)
Web services also provide greater flexibility when designing integration architectures. Because Web services use a common interface, changes to the network or business processes do not affect the ability of individual applications to communicate with each other.
Common Data Formats
Another source of complexity is the proliferation of data formats. In point-topoint integration, each application must transform its own data structures to those of every other application with which they share information. Reducing the number of data formats reduces the number of data transformations that take place as information passes through the integration framework.
Traditional Integration vs. Web Services Integration
Figures 2 and 3 show how reducing the number of APIs and using a common data format based on XML can reduce the complexity in an integration framework. In traditional integration, an application communicates with another application through the target application’s API. Data passing between applications undergoes a transformation from one format to another. Application programmers must be aware of each API and each data format in the framework, which creates a huge learning curve. As you add more applications to the framework, the complexity increases and the burden on developers grows.
Web services integration still requires that one application call another using an API, and data transformation still occurs. However, each application uses SOAP as a common API and XML as a common data format. Therefore, even as the number of applications in the integration framework increases, the burden on the developers remains the same.
Legacy Applications and Web Services Models
Mainframes are still the most reliable and scalable platforms for handling large amounts of data and large numbers applications on the mainframe is a good technical decision. In addition, integrating your applications is almost always less expensive than rewriting them. Now, by enabling your legacy applications as Web services, you can deliver integration projects faster and cheaper because your legacy application groups and Internet groups use their existing knowledge to integrate the applications.
There are two basic models for integrating legacy applications as Web services. The differences between these models depend upon where the Web Services exist, how they operate under-the-covers, and the types of applications you want to integrate. In this article, we will refer to these models as Adapters and Gateways.
- Adapters run on the mainframe and can use interfaces that permit tight integration with the target application.
- Gateways run off the mainframe on middle-tier servers and often use traditional methods, such as screen scraping.
Types of Legacy Applications
The choice between using adapters or gateways often depends on the types of applications you need to integrate. Figure 4 shows a high-level taxonomy of mainframe applications.
CICS represents the broadest category of OS/390 applications. CICS transactions fall into two sub-categories: “visual” and “non-visual.” A “visual” transaction is one that expresses a presentation interface to an end user at a terminal. You could also refer to a “visual” transaction as a “terminal-oriented” transaction. In contrast, “non-visual” transactions do not interact with an end user. Instead, another program invokes these transactions. (This type of transaction is also referred to as “COMMAREA transaction” because the input/output parameters are passed to/from the transaction using an area of storage referred to as the “communication area,” or COMMAREA.)
In the case of visual transactions, CICS application developers have always had a number of choices as to how to design their transaction to interact with an end user at a terminal. The majority of applications uses a component of CICS called Basic Mapping Support (BMS). BMS essentially handles the presentation logic of the transaction and relieves the application developer from having to encode and decode 3270 terminal datastreams. The remaining groups of applications that do not use BMS either include code to process 3270 datastreams, or rely upon a non-IBM solution to handle presentation logic.
IBM has traditionally provided facilities that allow you to use gateways to access CICS. However, with the release of CICS Transaction Server, IBM began providing facilities that allow the use of adapters to access legacy applications, so you can choose between adapter and gateways models based on your needs. The recent availability of adapters that support the full range of CICS applications allows remote applications to directly invoke almost any CICS application as a Web service.
There are solutions on the market that support COMMAREA, BMS, and non-BMS transactions all through a single adapter. Because most shops have a mix of application types, companies should look for this kind of adapter to avoid multiple software licenses and additional training on how to integrate the different application types within your organization.
Companies wanting to integrate IMS DC applications will most likely need to use gateway methods. While IMS applications usually separate application data and presentation logic, IMS DC does not provide generic facilities that allow an adapter to capture application data before it is sent and converted to XML/SOAP. Thus, the only other viable solution is to reengineer the application or use a gateway to access the application from a middle tier. A myriad of tools, including Novell’s eXtend (formerly SilverStream) or SEAGULL’s Transidiom, are able to make 3270 screens available to other applications as Web services.
In the adapter and gateway examples shown in Figures 5 and 6, we focus on using Web services with visual (terminal- oriented) applications because they are typically more difficult to integrate than non-visual applications.
Adapter Model for Web Services
Adapters allow you to transform your legacy applications into Web services without requiring the use of additional hardware, without changes to the legacy application and without falling back upon brittle techniques such as screen scraping as the enhanced access method. Compared to gateways, adapters yield better performance by running on the host and more reliable operation due to the elimination of the many layers data must pass through due to screen scraping. Adapters also provide access to application information such as status and error codes. This information is lost when data is sent to a terminal emulation technology such as a 3270-emulation client or Front-End Programming Interface (FEPI).
Figure 5 shows the basic model for accessing legacy applications as Web services directly through adapter technologies. This figure illustrates a Web services architecture: a Provider that supplies the Web service, a Requester that uses a Web service, and a Broker that finds Providers for Requesters. In this model, the legacy application is the Provider. The following steps represent how to find and use a Web service adapter (steps 1 through 3 are optional):
1. The Provider uploads a WSDL specification to publish its Web service with a Broker. 2. The Requester (usually a Java or .NET application) queries the Broker for a Web service by name or category. 3. The Broker selects a Provider and returns the Provider information to the Requester. 4. The Requester uses the information from the Broker to format and send a SOAP message to the Provider. 5. The Provider returns a SOAP/XML response to the Requester with the legacy application data enclosed.
Gateway Model for Web Services
Unlike adapters, gateways typically run on a physical or logical middle tier. Where gateways run is important because there are so few options for accessing the host from middle-tier servers, which means gateways usually involve some form of screen scraping. When gateways communicate with terminal-oriented legacy applications, they open a terminal session with the legacy application, send a request to the application, receive the terminal datastream, convert the contents to XML, and ship the XML document to the requester. (A variation of the gateway model is to use FEPI on the mainframe instead of a middle-tier terminal emulation client. This simply moves the middle tier onto the mainframe.)
Gateways allow you to get an application into the Web services mix, but screen scraping creates performance bottlenecks and multiple points of failure between the legacy application and the Web service. For this reason, gateways are probably best for short-term projects, either as a transition to using adapters or as a stopgap measure during application reengineering or platform migration.
Figure 6 shows the basic model for accessing legacy applications indirectly using a gateway technology such as a screen scraper.
Again, the diagram in Figure 6 illustrates a Web services architecture: a Provider that supplies the Web service, a Requester that uses a Web service, and a Broker that finds Providers for Requesters. The legacy application itself is not a Web service, but is accessed by an off-host Provider. The following steps represent how to find and use a Web service gateway (steps 1 through 3 are optional):
- The Provider uploads a WSDL specification to publish its Web service with a Broker.
- The Requester (usually a Java or .NET application) queries the Broker for a Web service by name or category.
- The Broker selects a Provider and returns the Provider information to the Requester.
- The Requester uses the information from the Broker to format and send a SOAP message to the Provider.
- The Provider starts an emulation session and conducts a series of transactions with the legacy application to collect the requested data.
- The Provider converts the legacy datastream into an appropriate format and returns a SOAP/XML response to the Requester with the legacy application data enclosed.
Dynamic Adapters vs. Code Generators
As we have seen, adapters provide several advantages over gateways when it comes to Web services integration. How those adapters operate can be just as important. Products that implement the adapter model for Web services usually fall into two camps: those that use code generation and those that are dynamic. Code generators have been around for some time and require developers to adhere to a fairly structured process. In the case of CICS applications, a scenario similar to the following occurs:
- Download program source code, copybook or screen maps to workstation running proprietary tool.
- Use a proprietary tool to generate Web services wrapper code.
- Upload generated code to mainframe.
- Compile, install and test generated code.
- Repeat process for each program and whenever the programs change.
While generated adapters can jumpstart the initial development process, they create several maintenance problems. First, the integration developer is generating code off the host and uploading the code to the mainframe. In most cases, this developer will know a lot about Web services and XML, but will know little about the mainframe. Thus, any changes to the mainframe application require coordination between the Web developer, the application developer, and the CICS administrator. Second, generated adapters create “net new code” that must be managed. Changes to either the legacy application or to the Web service will require repetition of the code generation process to keep the generated code in sync with the integrated application. Without proper management, you are likely to drown in a sea of generated code.
These issues led to the development of dynamic adapters. Dynamic adapters operate with little or no configuration, and changes to legacy applications are automatically incorporated into the SOAP/XML output. In many cases, there is no configuration required, while some cases may require a single-step process to specify Web service information for each application. As a result, there is no generated code and there is a clear division of labor: The CICS administrator installs the adapter and the Web developer simply invokes the adapter as a Web service.
As we have seen, one problem with traditional integration techniques is the proliferation of point-to-point communication and data conversions that must change as new applications are integrated or data formats change. The problem quickly becomes complicated when you add business partners, subsidiaries, mergers or acquisitions into the integration mix. Web services reduce the costs and complexities of integrating applications. By using standards-based technologies and widely available skillsets, Web services allow companies to develop flexible integration solutions that can evolve as their needs change.
The subsystem under which your legacy application runs determines your top-level integration choices. IMS provides limited facilities to create integration adapters, so IMS shops are often relegated to using screen scrapers and gateways. CICS Transaction Server includes facilities that allow third-party vendors to create adapters that can immediately enable legacy applications as Web services. These facilities also provide additional benefits over gateways, such as improved performance and increased stability vs. their screen scraping counterparts. By using the same industry standard technologies as Web services, some adapters make it possible for applications to transparently invoke CICS transactions within a Web services architecture and receive the resulting data as well-formed XML. For organizations that want to retain the value of their CICS applications, the combination of XML-enabling adapters and Web services offers a practical and powerful integration solution.
Web services are not a trend, but an industry-wide movement that can provide a long-term solution for companies that want to integrate legacy applications and data with new e-business processes. In the end, companies need to assess the value of the data contained in their legacy applications. Most companies have already determined that such data is highly valuable and they are looking for ways to preserve their investments. Given that recent surveys show the top strategic priorities of CIOs and CTOs is integrating systems and processes, the use of Web services will grow rapidly and you need to be ready for them.