Cisco Nexus 34180YC to Cisco UCS C460

There are many ways of connecting a Top of Rack (ToR) leaf switch to subtending computer and storage servers in the same or adjacent equipment racks. Other application notes in this series have depicted,

• The use of a 10GBASE-SR SFP+ installed in the TOR switch and a similar SFP+ installed in an appropriate Host Bus Adaptor (HBA) equipped server (see: SFP-10G-SR Application).
• A 40GBASE-SR4 QSFP+ in the TOR connected by an MPO-to-4xDuplex-LC hydra cable to four 10GBASE-SR SFP+ modules equipped in servers HBAs (see: QSFP-40G-SR4 Application).

The above methods are both high-performance, low-latency and relatively low power methods of ToR-Server interconnection. Many rack-mounted computer and storage servers come equipped with dual 10GBASE-T, 10GbE RJ45 ports. Although both power and latency are considerably higher using these ports versus using optical Ethernet, as shown in the tables below the cost savings of going the copper route are considerable. However, some extremely latency sensitive applications such as High Frequency Trading (HFT) will suffer some additional delay which may not be desirable or acceptable.

Table 1 10GbE Optical Interconnect Cost

Table 2 10GbE Copper Interconnect Cost

To depict this application with real-world products, a ToR switch from Cisco’s Nexus 3000 series and a rack-mounted service from Cisco UCS line have been selected. The only other pieces required are FluxLight SFP-10G-T-FL transceivers and a couple of CAT7 Ethernet patch cables.

TOR LEAF SWITCH: Cisco Nexus 34180YC

The Nexus 3400 platform is part of the latest generation of the fixed Nexus®3000 Series Switches. The 3000 series is squarely targeted for ToR applications. All members are compact (single rack unit) fixed configuration products. Across this family of products essentially all available rates of optical ethernet are offer, from 1G to 400G.

The specific switch cited in this example is the Nexus 34180YC. This switch offers great flexibility in ports speeds, covering 10G, 25G, 40G and 100G rates. The 34180YC is programmable allowing the user to customers to tailor packet forwarding behavior to the needs of their applications. For example, high speed financial trading applications can be optimized for the lowest possible latency. This switch is stocked with 48 SFP+/SFP28 ports (10G/25G) and 6 QSFP+/QSFP28 (40G/100G) ports. The switch supports full line rate Layer 2/3 switching on all these ports, totaling 3.6 Terabits/sec and 1.4 Gigapackets/sec.

The 34180YC may be equipped with a wide range of optical transceiver interface types across the many rates mentioned above. The tables below include the compatible transceiver types for the two categories of ports in the switch.

Table 3 SFP+/SFP28 Optical Transceivers

Table 4 QSFP+/QSFP28 Optical Transceivers

Rack-mounted Server: Cisco UCS C460 M4 Rack Server

Cisco’s C460 M4 server is a member of the Unified Computing Servers family. The basic unit is 4 rack units in height (4RU, approximately 7”) and is based on the Intel Xeon E7-4800 andE7-8800 family of processors. Each server may be equipped with up to 12 SFF drives (SSD, SAS or SATA) and up to 6 Terabytes of DDR3 or DDR4 memory in its 96 DIMM slots. It includes two 10GbE copper ports without the need for a Host Bus Adaptor. These servers offer ten PCIe Gen 3 slots and are designed for high availability with redundant power suppliers.

10GBASE-T OPTICAL TRANSCEIVERS: FluxLight SFP-10G-T-FL

FluxLight’s SFP-10G-T-FL (click link for datasheet) is a Cisco® compatible 10GBase-T SFP+ Optical Transceiver and is factory pre-programmed with all the necessary configuration data for fully compatible operation with all Cisco equipment offering SFP+ slots. The FluxLight SFP-10G-T-FL includes an RJ45 8-wire connector and supports the follow link lengths at various speeds and classes of Ethernet cable.

• 10GBase-T links up to 30m using Cat 7 Cable
• 5GBase-T links up to 50m on Cat5e or better
• 5GBase-T up to 75m on Cat5e or better
• 10/100/1000Base-T on Cat5e or better

Although Cisco does not offer a 10GBASE-T SFP+ transceiver, this devices performs identically to Cisco® original transceivers, are 100% compatible and support full hot swappable operation. The SFP-10G-T-FL is 100% MSA (Multi-Source Agreement) compliant, see: SFP+ MSA,

ETHERNET CAT7 PATCH CORD: FluxLight PN: FL-CAT7-10FT

To connect the SFP-10G-T-FLs in the Cisco Nexus 34180YC to the built-in 10GBase-T ports in the UCS C460 M4 servers, it is highly recommended to use a CAT7 Ethernet cable to reliably achieve full-duplex transmission at 10Gbps. 

Putting it all together

The Cisco Nexus 34180YC switch occupies 1RU and is typically placed at the top of the rack. Each UCS C460 M4 occupies 4RU. In this example two 10Gbps links are used per server, therefore, two FluxLight 10GBASE-T modules must be installed in the 34180YC for each server. So, in the depiction below, a total of 4 FluxLight SFP-10G-T-FL modules are used.

The SFP-10G-T-FL modules are inserted into 4 of the 48 the SFP+/SFP28 slots in the 34180YC. CAT7 Ethernet jumpers are used to connect each of these SFP modules to the built-in 10GbE RJ45 ports in the servers. Typically, the two cables between each server and the ToR switch are diversely routed (opposite sides of the rack) so that if one is accidentally removed it does not completely isolation that server.