New Virtualization Techniques in Storage Networking: Fibre Channel Improves Utilization and Scalabil
A previous article (Consolidation Drives Virtualization in Storage Networks, z/Journal December 2006/January 2007) examined how N_Port_ID Virtualization (NPIV) and virtual fabrics were used to increase utilization and decrease the switch port count in a storage network. This follow-up article shows how port counts can be reduced even more with virtual fabric tagging and how Inter-Fabric Routing increases the reach and addressability of fabrics. These powerful techniques increase the utilization of storage networks and provide more flexibility to the management of the storage network.
Storage networks have scaled to proportions that require new architectures and best practices to provide solutions. Similar to how Local Area Networks (LANs) scaled with layer 3 routing to connect to other layer 2 LANs, Fibre Channel storage networks use Inter-Fabric Routing to communicate between fabrics. By keeping fabrics small and manageable, Inter-Fabric Routing enables devices to communicate with devices in other, possibly distant fabrics. Inter-Fabric Routing thus enables connectivity without significantly increasing the size of the fabrics. With network address translation, Inter-Fabric Routing is particularly useful in FICON environments where static addressing is required. Inter-Fabric Routing has several scalability benefits that will be highlighted in this article.
Consolidating Inter-Switch Links
Figure 1 is a recap of the consolidated storage network where two directors are connected via redundant Inter-Switch Links (ISLs) between Denver and Englewood, CO. In traditional storage networks with physical switches, redundant ISLs would be used to ensure a single failure didn’t cut the fabric in half and cause the fabric to rebuild every time the ISL goes down. With three separate fabrics, the redundancy leads to low usage on each ISL while connectivity is the main requirement. The virtual fabric architecture overcomes this limitation by enabling multiple virtual fabrics to use the same ISL. By increasing utilization and maintaining connectivity, virtual fabric tagging ISLs can reduce the number of long-distance links in this scenario. This is especially important when leased, long-distance links incur high recurring costs that add to the total cost of ownership of the storage network.
The virtual fabric tagging architecture adds a header after the start of the Fibre Channel frame so multiple virtual fabrics can use the same physical link. Figure 2 shows how a physical port is associated to the virtual switch through the tagging logic. An administrator assigns physical ports to one or more virtual switches. If more than one virtual switch is assigned to a physical port, the port will need to use virtual fabric tagging to route the frame to the desired virtual switch. Since vendors don’t currently support virtual fabric tagging on N_Ports, each N_Port is associated with a single virtual switch and no virtual fabric tagging header is added by the tagging logic. When both ports on a link have been configured to connect multiple virtual switches, the header is added by the tagging logic as it leaves the virtual switch and deleted by the tagging logic before it reaches the destination virtual switch. Virtual fabric tagging is associated at the physical port level and the configuration of the two ports on the link determines if the link uses virtual fabric tagging.
When a physical port comes online, a bit is set that notifies the other port if it supports virtual fabrics. If both ports on the link support virtual fabrics, then the ports exchange virtual fabric parameters that include the virtual fabrics or virtual switches that the port is associated with. The logical AND of the configured virtual fabrics of the two ports becomes the active configuration. If one port is associated to virtual fabrics 1, 2 and 3 while the other port is associated with only virtual fabrics 2 and 3, then only the association to virtual fabrics 2 and 3 come online. If one of the ports was associated to only virtual fabric 2, then the ISL wouldn’t need to do virtual fabric tagging, since only the connection to the virtual switch associated to virtual fabric 2 would come online. If the logical AND of the virtual fabrics yields no matching virtual fabrics, then the link isolates. When the administrator reconfigures virtual fabric parameters on a port, the ports will renegotiate the virtual fabric parameters and the matching virtual fabrics will come online.
Backtracking to Figure 2, the virtual fabric tagging ISLs have increased the addressing space by supporting three virtual switches in Denver and Englewood. By consolidating these long-distance ISLs, they were able to significantly reduce recurring costs proportional to the amount of consolidation. Adding another virtual fabric tagging ISL can aid all the fabrics instead of just one fabric. With the typically bursty nature of Fibre Channel traffic, each ISL could support bursts from the various endpoints and increase utilization. While consolidation can drive virtualization, virtualization also can drive consolidation in a virtuous cycle.
Acquisition, Consolidation, and Inter-Fabric Routing