This application note describes the use of 100GBASE-CWDM4 links between leaft EOR/TOR switches and a larger data center Spine switches. Optical interconnects between data center racks is moving increasingly to 100G. One significant trade-off on these connections is whether to go single mode or multimode. The least expensive optics modules are VCSEL-based multimode transceivers. At 100G these multimode modules are called 100GBASE-SR4. Currently the least expensive 100G singlemode transceiver type is the 100GBASE-PSM4. Once fiber cost and flexibility are considered, the 100GBASE-CWDM4 option becomes the popular choice among many data center builders.

The optical connector of the multimode 100GBASE-SR4 parts are MPO-12, of which 8 fibers are utilized for Tx/Rx pairs for 4 lanes each running at 25Gbps. The single mode 100GBASE-PSM4 also utilizes an MPO-12 connector. The single mode 100GBASE-CWDM4 module use a duplex-LC optical connector, therefore very inexpensive and flexible duplex single mode fiber jumpers are used for interconnection. The maximum reach of the 100GBASE-SR4 is 100 meters on OM4 multimode fiber. The current average US retail price of a 100 meter OM4 MPO-12 cable is approximately $420 while the same length duplex-LC jumper averages $60. The table below summarizes the full link costs including fiber and transceivers. The PSM4 type transceiver is clearly the most expensive. Including only the limited costs factors in this analysis, the single mode CWDM4 is only about a 10% premium over the multimode SR4. The multimode MPO-12 has additional costs such as, more expensive and lower density patch panels, additional bulk and weight of cables, etc.

For the purposes of this application note, two of Cisco’s latest Nexus 3000 series switches will be used. The Nexus 3464C will be used as the ‘Spine’ switch and the Nexus 34180YC will be used as an example ‘Leaf’ switch. The data center architecture will depict a group of leaf switches fully mesh-interconnected at 100Gbps with spine switches. This robust and low latency arrangement has become the norm in hyperscale data centers. 

CORE SWITCH: Cisco Nexus 3464C

The Nexus® 3464C platform is part of the latest generation high-speed, fixed, high-density Nexus® 3000 Series Switches. While the Nexus 9K line is focused on commercial data centers, the Nexus 3K is targeted at enterprise applications including: service provider hosting, financial networks, and secured (read ‘private’) cloud computing. The Nexus 3K offers flexible combinations of 10G, 25G, 40G and 100G-Ethernet interfaces and the ability to optimize switch capacity and packet-forwarding behavior to a customers’ particular preferences. The key features and benefits:

• Line rate L2/2 switching on all ports
• User programmable NXOS
• Pipeline forwarding programmability

The Cisco Nexus 3464C is a 2RU switch that supports 12.8 Terabits per second (Tbps) total throughput and up to 5.2 billion packets per second (Gpps). The switch includes 64 QSFP+ /QSFP28 slots supporting speeds onf 40Gbps, 4x10Gbps, 100Gbps and 4x25Gbps on a per slot basis. The switch also includes 2 SFP+ slots running at 10Gbps.

The N3464C may be equipped with a wide range of optical transceiver interface types across the many rates mentioned above. The tables below include the compatible transceiver types for the two categories of ports in the switch.

Table 1 SFP+/SFP28 Optical Transceivers

Table 2 QSFP+ 40G Optical Transceivers

Table 3 QSFP+ 100G Optical Transceivers

ACCESS SWITCH:  Cisco Nexus34180YC

A great companion leaf switch to the N3464C spine switch is the N34180YC. The switch includes 48 SFP+/SFP28 ports and 6 QSFP+/QSFP28 ports. The switch has a total throughput of 3.6Tbps and up to 1.4Gbpps. The N34180C has the same features and benefits as the N3464C but is designed for edge aggregation versus core switching.

The table below shows the range of optical transceivers that may be equipped in the 34180’s SFP+/SFP28 slots (the QSFP+/QSFP28 options are the same as shown in Tables 2&3).

Table 4 SFP+/SFP28 Optical Transceivers

100GBASE-CWDM4 OPTICAL TRANSCEIVERS: FluxLight QSFP-100G-SM-SR-FL

FluxLight’s QSFP-100G-SM-SR-FL (click link for datasheet) is a Cisco® compatible 100GBASE-CWDM4 QSFP28 Optical Transceiver and is factory pre-programmed with all the necessary configuration data for fully compatible operation with all Cisco equipment offering QSFP28 slots. The FluxLight QSFP-100G-SM-SR-FL includes a dual-LC connector and supports the follow 10km transmission over standard singlemode fiber.

FluxLight’s device performs identically to Cisco® original transceivers, are 100% compatible and support full hot swappable operation. The QSFP-100G-SM-SR-FL is 100% MSA (Multi-Source Agreement) compliant, see: QSFP MSA.

OS2 SINGLE MODE FIBER JUMPER CABLE: FluxLight PN: FL-LCLC-DXSM-2M>

The GLC-LH-SMD modules equipped in the Cisco Nexus 9396PX and Catalyst C9200 switches offer a duplex LC type connector and utilize single mode fiber (SMF) for transmission. Since this application involves transmission over outside plant fiber across a campus, the optical transceiver modules are typically connected to a nearby optical patch panel using a short single mode jumper cable. An example jumper, shown in the figure above, has dual-LC connectors at each end and is 2 meters long.  If the fiber patch panel is further away or has a connector type other that dual-LC, of course an alternative cables are available, see: FluxLight OS2 fiber jumpers.

Putting it all together

The figure below depicts the 2-tier Spine-Leaf data center network. To create a highly robust network each compute/storage server is dual-homed to two N34180YC leaf switches. Maintaining this robust architecture and ensuring the lowest possible latency, each leaf switch is connected to all spine switches. This allows for all East-West traffic to be a most two switch hops away.

The figure below depicts the physical connectivity to realize this design. Each N34180YC is equipped with one QSFP-100G-SM-SR-FL module for each N3464C spine switch. Since the 34180’s have 6 QSFP28 uplink slots, that allows for up to six 3464 spine switches. Each spine switch supports throughput of 12.8Tbps, this leads to a total center throughput of 76.8Tbps! Since each N3464C includes 64 QSFP28 slots, a total of 64 leaf switches are supported each with 48 SFP+/SFP28 ports for a total of 3,072 ports. If each server is dual-homed, that is 1,536 servers.

Cisco switches were used to demonstrate the application. FluxLight carries 100GBASE-CWDM4 modules compatible with a wide range of OEM switches. Some popular models are listed below.

Table 5 SFP Optical Transceivers for Popular Switch OEMs