Infiniband addresses QoS through the concept of VLs. Infiniband’s VLs enable different QoS guarantees across the fabric (e.g., priority, latency guarantees, bandwidth guarantees, etc.) by logically dividing a physical link into multiple virtual links. Each VL has its own independent resource (i.e., send and receive buffers) dedicated to traffic with specific service levels.
Infiniband’s VLs are based on independent datastreams for each VL level. Each port can support up to 16 VLs numbered 0 to 15. VL15 is reserved exclusively for subnet management and is the management VL. The others (VL0- VL14) are data VLs. Each port must support the management VL and at least one data VL starting with VL0. Flow control is on a per VL basis. One VL not having an input buffer available doesn’t prevent data from following on the other VLs.
Infiniband VLs enable the fabric to support different QoS over the same physical links, depending on how the subnet manager takes advantage of them. Not all ports have to support the same number of VLs for management to take advantage of it. The subnet manager assigns service levels to end nodes and configures each port with its own service level-to-VL mapping. For instance, the subnet manager can assign service levels based on priority, bandwidth negotiation, etc. and the end node uses that value. As the packet traverses the fabric, each port determines which VL the packet uses based on the service level in the packet and the port’s service level-to-VL mapping table.
Another possible use for VLs is for separation of traffic and fairness when multiple systems share the same subnet. In this case, the subnet manager uses a different set of SLs for each system and each set of SLs maps to different VLs at each port. So heavy traffic on one VL doesn’t impact the other systems.
FICON director vendors have been actively pursuing QoS for both the open systems and FICON environment. Due to the complexity, and lack of progress made by standards bodies on Class 4 CoS, these vendors haven’t implemented Class 4 service on their switches and directors. The director vendors have implemented other QoS type features, including virtual channels, ingress rate limiting, software prioritization schema and SID/DID prioritization. Most of these don’t provide true, end-to-end, QoS for FICON environments. Virtual channel technology is closely based on Infiniband VLs and deserves a brief look.
Virtual channel technology represented an important breakthrough in the design of large storage networks. The technology is similar to the Infiniband VL concept. To ensure reliable ISL communications, virtual channel technology logically partitions bandwidth in each ISL into many different virtual channels and prioritizes traffic to optimize performance and prevent Head of Line Blocking (HoLB). The FICON director’s operating system automatically manages virtual channel configurations, eliminating the need to manually fine-tune for maximum performance. This technology also works in conjunction with ISL trunking to:
• Improve the efficiency of switch-to-switch communications
• Simplify FICON storage network design
• Reduce the Total Cost of Ownership (TCO).
With virtual channels, all class F traffic for the entire fabric automatically receives its own queue and the highest priority. This ensures the important control frames, such as name server updates, zoning distribution, Registered State Change Notifications (RSCNs), etc. are never waiting behind “normal” payload traffic (also referred to as Class 2 or Class 3 traffic). For Class 2 or 3 traffic (host and storage devices), individual Security ID (SID) and Destination ID (DID) pairs are automatically assigned in a round-robin fashion based on DID across the four data lanes. This prevents HoLB throughout the fabric and since each virtual channel has its own credit mechanism and flow control, slower devices won’t “starve” faster.
While these current technologies are attractive for managing QoS in a small segment of a FICON configuration (between cascaded FICON directors, for example), none offer what customers are really looking for: a mechanism to manage QoS in their FICON environment from host to storage control unit. But there is a way. Sometimes, one must look to the past for inspiration.