Juniper Networks QFX3500 Switch Assessment February 2011

Juniper QFX3500 Switch Assessment Executive Summary Juniper Networks commissioned Network Test to assess the performance of the QFX3500, a 48x10G, 4x40G Ethernet top-of-rack switch optimized for data center deployment. Performance tests included throughput and latency measurements of unicast traffic in layer-2 and layer3 modes (including both cut-through and store-and-forward modes for layer-2 tests), as well as throughput and latency of layer-2 multicast traffic. Network Test also assessed the ability of the QFX3500 to handle storage and data traffic on a converged network. These tests validated the switch’s ability to act as a Fibre Channel gateway while concurrently handling Ethernet data traffic. Tests also validated support for IEEE 802.1Qbb priority-based flow control (PFC) and determined whether concurrent storage and multicast traffic would affect switch latency. Key results from QFX3500 testing include the following:          

Sub-microsecond average latency in store-and-forward mode regardless of frame length Line-rate throughput for all frame sizes in all test cases, both for unicast and multicast traffic Layer-2 and Layer-3 unicast throughput figures are virtually identical Unicast and multicast average latency are virtually identical Support for both cut-through and store-and-forward modes of operation IGMP snooping support for at least 1,400 multicast groups Interoperability with the Cisco MDS 9148 Fibre Channel switch Support for Fibre Channel over Ethernet (FCoE) and Fibre Channel gateway functionality Support for PFC prioritization of storage traffic No impact on latency of multicast traffic when concurrently forwarding multicast and FC/FCoE storage traffic  Junos operating system, including all supported layer-2 and layer-3 switching and routing protocols

Page

2

The remainder of this document discusses the test results in more detail. Besides presenting the test results, each section describes the test objective and procedure, as well as its meaning for network architects and network managers.

Juniper QFX3500 Switch Assessment Unicast Throughput Throughput describes the highest rate at which a switch forwards traffic with zero frame loss. It’s a critical metric, given that even a single dropped frame can have adverse effects on application performance. This test measures throughput for unicast traffic, as defined in RFCs 1242, 2285, 2544, and 2889. Tests involved a fully meshed pattern of traffic between 48 switch ports for a duration of 60 seconds per iteration, using Spirent TestCenter as the traffic generator. Network Test measured throughput using three switch configurations: In layer-2 mode, as a store-andforward device; in layer-2 mode, as a cut-through device; and in layer-3 mode, as a store-and-forward device1. Tests were run with 64, 128, 256, 512, 1,024, 1,280, 1,518, and 9,216-byte jumbo frames. The QFX3500 exhibited line-rate throughput for all frame sizes. In every single test case, the Juniper switch never dropped a frame. Figure 1 summarizes throughput results for the QFX3500.

Figure 1: QFX3500 Throughput

Page

As discussed in RFC 1242, a store-and-forward device caches an entire frame before making a forwarding decision, while a cut-through device begins forwarding a frame before it has been fully received. For layer-2 switching of unicast traffic, a device can be configured in either mode. For layer-3 forwarding of unicast traffic and for all multicast traffic, switches always use store-and-forward mode.

3

1

Juniper QFX3500 Switch Assessment Unicast Latency As specified in RFC 2544, latency describes the delay introduced by the switch at the throughput rate. Latency is arguably even more important than throughput in predicting application performance, since it affects every frame passing through a switch, regardless of load. Some applications, such as those involving voice and video and those used in high-frequency trading, are particularly sensitive to latency. Network Test measured the QFX3500’s minimum, average, and maximum latency in the same three scenarios as in the throughput tests: As a layer-2 store-and-forward device; as a layer-2 cut-through device; and as a layer-3 store-and-forward device (with one subnet and VLAN per switch port). When configured in layer-2 store-and-forward mode, the Juniper QFX3500 exhibited minimum unicast latency as low as 610 nanoseconds; average unicast latency as low as 850 nanoseconds; and maximum unicast latency not exceeding 1.1 microseconds. Moreover, store-and-forward latency did not rise significantly as frame lengths increased. Figure 2 summarizes minimum, average, and maximum layer-2 unicast store-and-forward latency for the QFX3500.

Page

4

Figure 2: Layer-2 Unicast Store-and-Forward Latency

When configured as a cut-through device, average latency for the QFX3500 ranged from 990 nanoseconds to 1.50 microseconds, and maximum latency did not exceed 1.64 microseconds.

Juniper QFX3500 Switch Assessment Sharp-eyed readers will note that cut-through latency figures increase with frame size, which is more often a characteristic of store-and-forward devices. This is not a measurement error; as required by RFC 1242, Network Test used last-in, first-out (LIFO) measurements for store-and-forward mode and first-in, first-out (FIFO) measurements for cut-through mode. Rather, the latency profile is explained by the switch ASIC (application-specific integrated circuit), which caches shorter frames (in this test, those of 512 bytes and less) before making forwarding decisions, even in cut-through mode. Figure 3 summarizes layer-2 unicast cut-through latency for the QFX3500.

Figure 3: Layer-2 Unicast Cut-Through Latency

Network Test also measured latency of the QFX3500 when configured as a layer-3 device, with different IPv4 subnets (and layer-2 VLANs) set up on each port. To determine layer-3 scalability, Network Test conducted the layer-3 tests twice: once with a single host per port, and again with the Spirent TestCenter test instrument emulating 43 hosts on each port, for more than 2,000 hosts total.

Page

Figure 4 summarizes results from layer-3 unicast latency tests of the Juniper QFX3500.

5

When configured in layer-3 mode, the QFX3500 delivered virtually identical latency when routing unicast traffic between one host per port or routing among more than 2,000 hosts.

Juniper QFX3500 Switch Assessment

Figure 4: Layer-3 Unicast Latency

Layer-2 Multicast Performance A separate set of tests focused on layer-2 multicast performance, using the aggregated multicast throughput and latency tests described in RFC 3918. Here, the focus was on multicast performance when the QFX3500 is configured in IGMP snooping mode. Tests used IGMPv2. One Spirent TestCenter port acted as a multicast transmitter and the remaining 47 ports acted as multicast receivers, each subscribed to the same 1,400 IP multicast groups. As in the unicast tests, the Juniper QFX3500 delivered line-rate throughput for all multicast traffic, regardless of frame length. Tests also validated the ability of the QFX3500 to forward traffic to 1,400 IP multicast groups, with all receiver ports subscribed to all groups.

Page

6

Moreover, multicast average and maximum latency was virtually identical to unicast latency. Figure 5 compares layer-2 unicast and multicast latency.

Juniper QFX3500 Switch Assessment

Figure 5: Unicast and Multicast Latency Compared

Converged Network Testing: Mixing Storage and Data Traffic Network Test examined the QFX3500’s ability to handle storage and data traffic on a converged network. Increasingly, network architects are looking to combine legacy Fibre Channel storage traffic with Ethernet data traffic onto converged data center networks. This convergence in turn requires data center switches to provide Fibre Channel and FCoE functionality; to interoperate with existing Fibre Channel equipment; to ensure lossless delivery of storage traffic; to prioritize storage traffic during periods of congestion; and to offer low latency when combining storage and data traffic.

Page

7

To assess the QFX3500’s ability to meet these requirements, Juniper and Network Test engineers constructed a test bed connecting the QFX3500 switch over a single 8-Gbit/s Fibre Channel link with a Cisco Systems MDS 9148 Fibre Channel switch. This interconnection was possible because up to 12 QFX3500 ports can be used for Fibre Channel connections. Engineers then configured the Spirent TestCenter traffic generator to offer FCoE storage traffic to two QFX3500 ports, and to offer data traffic to 40 remaining 10G Ethernet ports. In some tests, engineers deliberately attempted to overload the FCoE ports to verify correct PFC operation. Figure 6 shows the test bed layout for converged testing.

Juniper QFX3500 Switch Assessment

Figure 6: Converged Storage/Data Test Bed

The convergence tests, which were mostly functional in nature, validated the following attributes about the QFX3500: The QFX3500 handles both Fibre Channel and FCoE traffic, and acts as a gateway to other Fibre Channel devices Using 8-Gbit/s Fibre Channel interfaces, the QFX3500 interoperates with the MDS 9148 Fibre Channel switch from Cisco Systems The QFX3500 supports lossless delivery of storage traffic, even during periods of congestion The QFX3500 supports PFC, the IEEE 802.1Qbb specification Network Test also conducted one performance test in this area to determine whether latency would be affected when the QFX3500 concurrently handled storage and multicast data traffic. Engineers first measured baseline latency for multicast traffic alone, using one transmitter and 39 subscriber ports, again with 1,400 multicast groups as in previous tests. Engineers then reran the same multicast test, but this time also offered storage traffic between two FCoE ports on the QFX3500 switch.

Page

8

As shown in Figure 7, multicast latency is virtually identical in tests with and without storage traffic. These results, along with the fact that the switch never dropped storage frames, suggest there is no performance penalty involved when the QFX3500 handles converged data/storage traffic.

Juniper QFX3500 Switch Assessment

Figure 7: The Effect of FCoE on Multicast Latency

The QFX3500 also supports other convergence features such as Fibre Channel Initialization Protocol (FIP) snooping when the device acts as a transit switch; Enhanced Transmission Selection (ETS); and the data center bridging (DCBX) extensions to LLDP. However, Network Test did not evaluate these features in this round of testing.

Conclusion A key goal of these tests was to validate the performance characteristics of the Juniper QFX3500 when used as a data center switch, especially for those data centers looking to combine storage and data traffic on a single network. The QFX3500 delivered line-rate throughput in all tests while ensuring low latency for both unicast and multicast traffic. Throughput was virtually identical between layer-2 and layer-3 throughput, and between unicast and multicast latency.

Page

Finally, like other Juniper Ethernet switches, the QFX3500 supports the full range of protocols available through Junos, the operating system for Juniper devices in the data center and beyond.

9

The QFX3500 also proved capable in functional tests of data/storage network convergence. The switch ensured lossless forwarding of storage traffic, with prioritization of Fibre Channel and FCoE frames during periods of congestion. It also functioned as a gateway to other Fibre Channel equipment on the test bed.

Juniper QFX3500 Switch Assessment Appendix A: About Network Test Network Test is an independent third-party test lab and engineering services consultancy. Our core competencies are performance, security, and conformance assessment of networking equipment and live networks. Our clients include equipment manufacturers, large enterprises, service providers, industry consortia, and trade publications.

Appendix B: Hardware and Software Releases Tested This appendix describes the software versions used on the test bed. All tests were conducted in January 2011 at Juniper’s headquarters facility in Sunnyvale, CA, USA. Component Juniper QFX3500 Cisco MDS 9148 Spirent TestCenter

Version Junos 11.1B1.4 NX-OS 5.0(1a) 3.60.7966

Appendix C: Disclaimer Network Test Inc. has made every attempt to ensure that all test procedures were conducted with the utmost precision and accuracy, but acknowledges that errors do occur. Network Test Inc. shall not be held liable for damages which may result for the use of information contained in this document. All trademarks mentioned in this document are property of their respective owners.

Page

10

Version 2011021400. Copyright 2011 Network Test Inc. All rights reserved. Network Test Inc. 31324 Via Colinas, Suite 113 Westlake Village, CA 91362-6761 USA +1-818-889-0011 http://networktest.com [email protected]