Over the past decade, extension networks for storage have become commonplace and continue to grow in size and importance. Growth isn’t limited to new deployments but also involves the expansion of existing deployments. Requirements for data protection will never ease, as the economies of many countries depend on successful and continued business operations; thus, laws have been passed mandating data protection. Modern-day dependence on remote data replication (RDR) means there’s little tolerance for lapses that leave data vulnerable to loss. In IBM mainframe environments, reliable and resilient networks—to the point of no frame loss and in-order frame delivery—are necessary for error-free operation, high performance and operational ease. This improves availability, reduces risk and operating expenses and, most important of all, reduces risk of data loss.

A previous article, “Storage Networking Business Continuity Solutions,” Enterprise Tech Journal, October/November 2013 (available at, introduced the various topologies and protocols used for the networks associated with business continuity, disaster recovery and continuous availability (BC/DR/CA). This article focuses in-depth on one of those protocols—Fibre Channel over Internet Protocol (FCIP)—and how it’s used in a mainframe environment to provide long-distance extension networks between data centers. Because of the higher costs of long-distance dark fiber connectivity compared with other communications services, use of the more common and more affordable IP network services is an attractive option for FC extension between geographically separated data centers. FCIP is a technology for interconnecting FC-based storage networks over distance using IP (see Figure 1).


FCIP Basics
IP storage initiatives have evolved on a foundation of previously established standards for Ethernet and IP. For example, Ethernet is standardized in IEEE 802.3, Gigabit Ethernet (GbE) is standardized in IEEE 802.3z and 10 GbE is standardized in IEEE 802.3a. IP-related standards are established by the Internet Engineering Task Force (IETF) through a diverse set of request for comments (RFCs) that cover a wide range of protocol management issues. Transmission Control Protocol (TCP) was standardized in 1981 as RFC 793, User Datagram Protocol (UDP) was standardized in 1980 as RFC 768 and FCIP was standardized in 2004 as RFC 3821. 

Ethernet and IP are only half the equation for IP storage networks. IP storage technology must also accommodate previously established standards for Small Computer System Interface (SCSI), which is the purview of the InterNational Committee for Information Technology Standards (INCITS) T10. FCIP storage solutions must also follow the previously established standards for Fibre Channel Protocol (FCP) (INCITS T10) and Fibre Channel Transport (INCITS T11). 

These standards all provide guideposts for technology development with the ideal goal of product interoperability. Guideposts aren’t intended to be moved; in other words, new technology developments shouldn’t require changes to existing standards.

RFC 3821
The RFC 3821 specification is a 75-page document that describes mechanisms that allow the interconnection of islands of FC storage area networks (SANs) to form a unified SAN in a single fabric via connectivity between islands over IP. The chief motivation behind defining these interconnection mechanisms is a desire to connect physically remote FC sites, allowing remote disk/DASD access, tape backup and live/real-time mirroring of storage between remote sites. The FC standards have chosen nominal distances between switch elements that are less than the distances available in an IP network. Since FC and IP networking technologies are compatible, it’s logical to turn to IP networking for extending the allowable distances between FC switch elements. 

The fundamental assumption made in RFC 3821 that you should remember is that FC traffic is carried over the IP network in such a manner that the FC fabric and all the FC devices in the fabric are unaware of the presence of the IP network. This means the FC datagrams must be delivered in such time as to comply with the existing FC specifications. The FC traffic may span LANs, metropolitan area networks (MANs) and wide area networks (WANs) as long as this fundamental assumption is adhered to. 

FCIP Fundamental Concepts
FCIP is a technology for interconnecting FC-based storage networks over extended distances via IP networks. FCIP enables an end user to use his existing IP WAN infrastructure to connect FC SANs. FCIP is a means of encapsulating FC frames within TCP/IP and sending these IP packets over an IP-based network specifically for linking FC SANs over these WANs. FCIP implements tunneling techniques to carry the FC traffic over the IP network. The tunneling is Upper Layer Protocol (ULP) transparent; i.e., both FICON and FCP traffic can be sent via these FCIP tunnels over an IP-based network. 

FCIP supports applications such as RDR, centralized SAN backup and data migration over very long distances that are impractical or costly using native FC connections. FCIP tunnels, built on a physical connection between two extension switches or blades, allow FC I/O to pass through the IP WAN.

3 Pages