Technical white paper

Networking best practices for Apache Hadoop on HP ProLiant servers Table of contents Executive summary ...................................................................................................................................................................... 2 HP Networking and IRF features ................................................................................................................................................ 2 Link Aggregation Control Protocol (LACP) ........................................................................................................................... 2 Intelligent Resilient Framework (IRF) ................................................................................................................................... 3 Hadoop network configuration on HP ProLiant servers ....................................................................................................... 3 Dual switch configuration using Intelligent Resilient Framework (IRF) ........................................................................ 4 HP ProLiant for Hadoop single rack design ............................................................................................................................. 7 Spanning Tree Protocol (STP) ................................................................................................................................................ 7 Server NIC bonding ................................................................................................................................................................... 8 Server with 802.1q VLAN tagging ......................................................................................................................................... 9 Enable tagged VLANs on HP 5900 ...................................................................................................................................... 10 BFD MAD enabled IRF ............................................................................................................................................................. 11 HP ProLiant for Hadoop data center connectivity ................................................................................................................ 12 Configuring HP 5900 network switch ................................................................................................................................. 12 Configure Cisco switch for HP 5900 .................................................................................................................................... 14 Appendix A: HP 5900 switch configuration ........................................................................................................................... 16 Appendix B: Glossary of terms ................................................................................................................................................. 18 For more information ................................................................................................................................................................. 19

Technical white paper | Networking best practices for HP ProLiant for Hadoop

Executive summary This white paper provides information on how to utilize best practices to configure networking for HP ProLiant servers for Hadoop. HP Networking solutions enable the following: • Breakthrough cost reductions by converging and consolidating server, storage, and network connectivity onto a common

fabric with a flatter topology and fewer switches than the competition. • Predictable performance and low latency for bandwidth-intensive server-to-server communications. • Improved business agility, faster time to service, and higher resource utilization by dynamically scaling capacity and

provisioning connections to meet virtualized application demands by leveraging technologies such as HP Intelligent Resilient Framework (IRF)1. • Removal of costly, time-consuming, and error-prone change management processes by

– Utilizing IRF to allow multiple devices to be managed using a single configuration file from a single, easy-to-manage virtual switch operating across network layers – Utilizing HP’s Intelligent Management Center (IMC) to manage, monitor, and control access to either a few or thousands of switches in multiple locations from a single pane of glass • Modular, scalable, industry standards-based platforms and multi-site, multi-vendor management tools to connect and

manage thousands of physical resources This document also includes explanations of different types of link aggregation protocols that HP leverages in its networking products to help meet the network resiliency needs of your network and business applications. The document also provides some example configurations when the customer is interacting with Cisco switches. For further details or instructions follow the steps described in those specific switch manuals. Target audience: Network and/or system administrators with basic to advanced knowledge in configuring the network of their rack mounted servers using HP ProLiant servers and HP Networking switches. This white paper describes testing performed in June 2014. DISCLAIMER OF WARRANTY This document may contain the following HP or other software: XML, CLI statements, scripts, parameter files. These are provided as a courtesy, free of charge, “AS-IS” by Hewlett-Packard Company (“HP”). HP shall have no obligation to maintain or support this software. HP MAKES NO EXPRESS OR IMPLIED WARRANTY OF ANY KIND REGARDING THIS SOFTWARE INCLUDING ANY WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE OR NONINFRINGEMENT. HP SHALL NOT BE LIABLE FOR ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES, WHETHER BASED ON CONTRACT, TORT OR ANY OTHER LEGAL THEORY, IN CONNECTION WITH OR ARISING OUT OF THE FURNISHING, PERFORMANCE OR USE OF THIS SOFTWARE.

HP Networking and IRF features The following are some of the features that are provided by HP 5900 switches that are utilized in Apache Hadoop configurations.

Link Aggregation Control Protocol (LACP) Link Aggregation Control Protocol (LACP) is an open industry IEEE standard (IEEE 802.3ad or IEEE 802.1ax) that provides a method to control the bundling of several physical ports together to form a single logical channel. An LACP-enabled port will send Link Aggregation Control Protocol Data Unit (LACPDU) frames across its link in order to detect a device on the other end of the link that also has LACP enabled. Once the other end receives the packet, it will also send frames along the same links enabling the two units to detect multiple links between themselves and then combine them into a single logical link. LACP can be configured in one of two modes: active or passive. In active mode it will always send frames along the configured links. In passive mode however, it acts as “speak when spoken to”, and therefore can be used as a way of controlling accidental loops (as long as the other device is in active mode). LACP is most commonly used to connect a user device (a server, workstation) or network switches with multiple links to a switch in order to form a single logical channel. To form a single logical channel on the server requires the configuration of NIC teaming, also known as bonding. For a Microsoft® Windows® Server 2012 host, NIC teaming is provided through the NIC

1

2

Reducing network complexity, boosting performance with HP IRF technology , http://h17007.www1.hp.com/docs/reports/irf.pdf

Technical white paper | Networking best practices for HP ProLiant for Hadoop

Teaming management interface or through the rich Windows PowerShell interface. Linux has the ability to natively configure NIC teaming.

Intelligent Resilient Framework (IRF) IRF technology extends network control over multiple active switches. Management of a group of IRF enabled switches is consolidated around a single management IP address, which vastly simplifies network configuration and operations. One IRF member operates as the primary system switch, maintaining the control plane and updating forwarding and routing tables for the other devices. If the primary switch fails, IRF instantly selects a new primary, preventing service interruption and helping to deliver network, application, and business continuity for business-critical applications. Within the IRF domain, network control protocols operate as a cohesive whole to streamline processing, improve performance, and simplify network operation. Routing protocols calculate routes based on the single logical domain rather than the multiple switches it represents. Moreover, edge or aggregation switches that are dual homed to IRF-enabled core or data center switches “see” the associated switches as a single entity, thus enabling true active/active architecture, eliminating the need for slow convergence and active/passive technologies such as spanning tree protocol (STP). Operators have fewer layers to worry about, as well as fewer devices, interfaces, links, and protocols to configure and manage. Intelligent Resilient Framework (IRF) overcomes the limitations of legacy spanning tree designs by providing rapid failover for delay-sensitive, mission-critical applications and dramatically improving network utilization and performance in the network core. IRF is an innovative HP switch platform virtualization technology that allows customers to dramatically simplify the design and operations of their data center and campus Ethernet networks. • Virtualizes switching platforms to dramatically simplify the design and operations of the network fabric. • Flatten networks by eliminating the need for a dedicated aggregation layer. • Provide more direct, higher capacity connections between users and network resources.

For more information on IRF, you can view two white papers at http://h17007.www1.hp.com/docs/reports/irf.pdf and http://bizsupport1.austin.hp.com/bc/docs/support/SupportManual/c02648772/c02648772.pdf

Hadoop network configuration on HP ProLiant servers Figure 1 shows the rack configuration of HP ProLiant servers. The HP ProLiant DL360p and DL380p Gen8 servers are configured with 2 10GbE network ports with FlexibleLOM support (561FLR-T 2p 10Gb Ethernet adapter) which allows HP Integrated Lights-Out 4 (iLO 4) management traffic to be shared on NIC1 of the server. Two HP 5900AF-48XGT-4QSFP+ switches are configured with IRF. NIC1 from each server is connected to switch 1 (SW1 – 5900AF-48XGT Ethernet switch) and NIC2 from each server is connected to switch 2 (SW2 – 5900AF-48XGT Ethernet switch) as shown in Figure 2. The two 10GbE NICs are configured as a bonded pair and trunked to the HP switches with two VLANs. VLAN X allows communication within the Hadoop rack and VLAN Y is used to communicate with the public or Customer Enterprise Network (CEN).

3

Technical white paper | Networking best practices for HP ProLiant for Hadoop

Figure 1. HP ProLiant for Hadoop rack configuration

Rack 1 HP 642 1075mm Shock Intelligent Rack Ethernet Switch - R01SW02 Ethernet Switch - R01SW01 ManagementNode - R01MGT HeadNode - R01HN02 HeadNode - R01HN01 WorkerNode - R01WN018 WorkerNode - R01WN017 WorkerNode - R01WN016 WorkerNode - R01WN015 WorkerNode - R01WN014 WorkerNode - R01WN013 WorkerNode - R01WN012 WorkerNode - R01WN011 WorkerNode - R01WN010 WorkerNode - R01WN009 WorkerNode - R01WN008 WorkerNode - R01WN007 WorkerNode - R01WN006 WorkerNode - R01WN005 WorkerNode - R01WN004 WorkerNode - R01WN003 WorkerNode - R01WN002 WorkerNode - R01WN001

PDU u42 u41 u40 u39 u38 u37 u36 u35 u34 u33 u32 u31 u30 u29 u28 u27 u26 u25 u24 u23 u22 u21 u20 u19 u18 u17 u16 u15 u14 u13 u12 u11 u10 u9 u8 u7 u6 u5 u4 u3 u2 u1

R001PD004

u42 u41 u40 u39 u38 u37 u36 u35 u34 u33 u32 u31 u30 u29 u28 u27 u26 u25 u24 u23 u22 u21 u20 u19 u18 u17 u16 u15 u14 u13 u12 u11 u10 u9 u8 u7 u6 u5 u4 u3 u2 u1

AF520A PDU 5900AF-48XGT switch 5900AF-48XGT switch DL360p 8SFF DL360p 8SFF DL360p 8SFF

DL380p 12LFF DL380p 12LFF DL380p 12LFF DL380p 12LFF DL380p 12LFF DL380p 12LFF DL380p 12LFF DL380p 12LFF DL380p 12LFF DL380p 12LFF DL380p 12LFF

R001PD002

R001PD001

R001PD003

PDU

DL380p 12LFF DL380p 12LFF DL380p 12LFF DL380p 12LFF DL380p 12LFF DL380p 12LFF DL380p 12LFF

Dual switch configuration using Intelligent Resilient Framework (IRF) The recommended configuration for performance and resiliency is to leverage HP’s Intelligent Resilient Framework (IRF) in the data center switches (Figure 2). When IRF is used, the HP ProLiant servers for Hadoop can take advantage of link aggregation to boost performance in addition to providing resiliency even during a complete switch failure. When IRF is used, you can create a link aggregation across two separate switches since IRF enables the two switches to act as one. Ports 51 and 52, 40GbE QSFP ports, are configured as IRF ports. Port 51 on SW1 is connected to Port 52 on SW2 and Port 52 on SW1 is connected to Port 51 on SW2 using HP X240 40G QSFP+ to QSFP+ 1m Direct Attach Copper Cable (JG326A) cables. Figure 2 shows the connection on the HP 5900 switch. The multi-active detection (MAD) feature detects identical active IRF virtual devices and handles multi-active collisions on a network. To detect and handle multi-active collisions, the MAD feature identifies each IRF virtual device with an active ID, which is the member ID of the master switch. BFD MAD is implemented by using the Bidirectional Forwarding Detection (BFD) protocol, which helps fast detect link failures and loss of IP connectivity. Section “BFD MAD enabled IRF” details how to configure the switch to enable BFD MAD protocol. 4

Technical white paper | Networking best practices for HP ProLiant for Hadoop

Figure 2. HP ProLiant servers for Hadoop Intelligent Resilient Fabric (IRF) configuration

R01SW02 HP 5900AF-48XGT-4QSFP+

BFD MAD

IRF

R01SW01 HP 5900AF-48XGT-4QSFP+

bond

TOP

Figure 2 above shows how server network NICs are connected to the HP 5900 switches. Figure 3 shows the ports assigned to different networks on the HP 5900 switch. Figures 3 and 4 show connections from ports 1-18 connecting to Hadoop data nodes. Ports 45 and 46 are connected to Hadoop head nodes. Port 47 is connected to the management node, port 48 on SW1 is connected to iLO of the r01mgt management node. Ports 49 and 50, 40GbE QSFP ports, can be used for customer uplinks. If the customer’s upstream switch does not support 40GbE connections, the 40GbE ports can be split into four 10GbE ports to connect to the customer network using either a DA splitter cable or an SR fiber optics splitter cable. Ports 19 to 32 are available for customer usage. How these ports can be configured to connect to a customer network is described in section “HP ProLiant for Hadoop data center connectivity”. Reserved ports can be used for management and future use, these ports can be used to plug into the iPDU (if used or other management elements). Figure 3. HP ProLiant for Hadoop switch configuration

HP 5900AF-48XGT-4QSFP 48 RJ-45 1/10GbE ports

Hadoop Network

Customer Network

4 QSFP+ 40-GbE ports

Reserved

Hadoop Network

Customer Network

IRF

5

Technical white paper | Networking best practices for HP ProLiant for Hadoop

Figure 4 shows a detailed network wiring diagram of the connections from the servers and switches on HP ProLiant for Hadoop. Figure 4. HP ProLiant for Hadoop network wiring with ports information

Color key 40 GbE Uplink Hadoop Customer IRF Reserved

Rack1 Ethernet Switch R01SW02 Ethernet ports 1

3-16

17

19-32

33-42

R01WN01

R01WN03

R01WN17

Customer

Reserved

Port2

Port2

Port2

43

45

47

49

51

R01HN01

R01MGT

Customer

R01SW02

Port2

Port2 48

P51

2

3-16

18

19-32

33-42

44

46

R01WN02

R01WN04

R01WN18

Customer

Reserved

BFD MAD

R01HN02

Port2

Port2

Port2

Port2

Port2

43

45

47

49

51

R01HN01

R01MGT

Customer

R01SW01

Port1

Port1

50

52

Customer

R01SW02 P52

Ethernet Switch R01SW01 Ethernet ports 1

3-16

17

19-32

33-42

R01WN01

R01WN03

R01WN17

Customer

Reserved

Port1

Port1

Port1

P51

2

3-16

18

19-32

33-42

44

46

48

50

52

R01WN02

R01WN04

R01WN18

Customer

Reserved

BFD MAD

R01HN02

R01MGT

Customer

R01SW01

Port1

Port1

Port1

Port1

Port1

iLO

P52

Connect a laptop to port 48 of the top Ethernet switch (r01sw02) and configure an IP on the private network so that you have connectivity to the network for further configuration of the cluster. The switch configuration is shown in Appendix A. In this document VLAN 1120 is a private VLAN and VLAN 120 is a public VLAN. Ifcfg-eth0 is configured to be slave of ifcfg-bond0 Ethernet device. The section below “Server NIC bonding” describes the configuration of the private network with 172.24.1.x IP addresses. The section “Server with 802.1q VLAN tagging” provides information on how the public VLAN network is configured. If the customer VLAN is different than 120, they will have to replace the VLAN number 120 to represent their customer VLAN. Each interface port on the switch has been configured to support this network configuration. In the example below, port 1 of switch SW1 is configured to permit untagged 1120 traffic and allow tagged 120 VLAN traffic. The details on switch configuration are provided in the “Enable tagged VLANs on HP 5900” section. The ports that are assigned to the Hadoop nodes are configured as hybrid ports as they allow untagged network traffic (VLAN 1120) within the ProLiant for Hadoop rack and allow tagged network traffic (VLAN 120) for CEN traffic.

6

Technical white paper | Networking best practices for HP ProLiant for Hadoop

Note: HP 5900 switch port interface configuration interface Ten-GigabitEthernet1/0/1 port link-type hybrid undo port hybrid vlan 1 port hybrid vlan 120 tagged port hybrid vlan 1120 untagged port hybrid pvid vlan 1120

For the network traffic to flow from the ProLiant for Hadoop rack to the customer / public network, one can configure multiple ports on the 5900AF-48XGT switches as shown in the example below to permit VLAN 120. The section “Configuring HP 5900 network switch” shows a sample of how the switch is configured.

Note: Configuring customer uplink interface Ten-GigabitEthernet1/0/43 port link-type trunk port trunk permit vlan 120 port link-aggregation group 1

If the customer has a Cisco switch that connects to the HP 5900 switch the section “Configure Cisco switch for HP 5900” will provide information on how to configure the customer’s Cisco switch so that it communicates with the HP 5900 switch and passes appropriate traffic through their networks. The section “Spanning Tree Protocol (STP)” provides information on STP configuration.

HP ProLiant for Hadoop single rack design Following sections in the document focus on best practices for single rack configuration, focusing on: • Spanning Tree Protocol (STP) • Server NIC bonding • Server with 802.1q VLAN tagging • Enable tagged VLANs on HP 5900 • BFD MAD enabled IRF

Spanning Tree Protocol (STP) Spanning Tree Protocol (STP) is a Layer 2 protocol that runs on bridges and switches. The specification for STP is IEEE 802.1D. The main purpose of STP is to ensure that you do not create loops when you have redundant paths in your network. While STP/RSTP (Rapid Spanning Tree Protocol)/MSTP (Multiple Spanning Tree Protocol)/PVST (Per VLAN Spanning Tree) are fairly effective in preventing unwanted network loops, convergence can still take several seconds, affecting applications that cannot handle that length of delay. In addition, the performance of STP is poor because it blocks all parallel paths except the one it has selected as active. When you enable the spanning tree feature globally on an HP 5900 switch, the device operates in STP, RSTP, PVST or MSTP mode, depending on the spanning tree mode setting. When the spanning tree feature is enabled, the device dynamically maintains the spanning tree status of VLANs based on received configuration Bridge Protocol Data Units (BPDUs). When the spanning tree feature is disabled, the device stops maintaining the spanning tree status. To enable the spanning tree feature globally on HP 5900 switch, type: [HP]stp global enable

7

Technical white paper | Networking best practices for HP ProLiant for Hadoop

Set the spanning tree mode on HP 5900 Switch. A spanning tree device operates in MSTP mode by default. To set the spanning tree mode use the command: “stp mode { mstp |pvst | rstp | stp }”. Specify the mode for the customer network: [HP]stp mstp pvst rstp stp

mode ? Multiple spanning tree protocol mode Per-Vlan spanning tree mode Rapid spanning tree protocol mode Spanning tree protocol mode

If a port directly connects to a server rather than to another device or a shared LAN segment, this port is regarded as an edge port. You can enable the port to transit to the forwarding state rapidly by configuring it as an edge port. HP recommends that you configure the ports directly connected to servers as edge ports. Following is the instruction to change a port to an edge port: [HP] interface Ten-GigabitEthernet 1/0/8 [HP-Ten-GigabitEthernet1/0/8] stp edge-port Use “display stp” to display the spanning tree status and statistics information. Based on the information, you can analyze and maintain the network topology or determine whether the spanning tree is working correctly. [HP]disp stp brief Additional information regarding STP configuration can be found at http://networktest.com/hpiop/hpiopcookbook.pdf HP/Cisco Switching and Routing Interoperability Cookbook.

Server NIC bonding The following are the instructions on configuring servers with bonding. On the servers, two 10GbE adapter NICs are configured as bonded. The following is the configuration showing bond0 being the master and eth0 as slave. Red Hat® Enterprise Linux allows administrators to bind multiple network interfaces together into a single channel using the bonding kernel module and a special network interface called a channel bonding interface. Channel bonding enables two or more network interfaces to act as one, simultaneously increasing the bandwidth and providing redundancy. To create a channel bonding interface, create a file in the /etc/sysconfig/network-scripts/ directory called ifcfg-bondN, replacing N with the number for the interface, such as 0. The contents of the file can be identical to whatever type of interface is getting bonded, such as an Ethernet interface. The only difference is that the DEVICE directive is bondN, replacing N with the number for the interface. The NM_CONTROLLED directive can be added to prevent NetworkManager from configuring this device. The following is a sample channel bonding configuration file: The interface bond0 is configured with the IP address and bonding options; additional information on the values is below. # cat /etc/sysconfig/network-scripts/ifcfg-bond0 DEVICE=bond0 IPADDR=172.24.1.1 NETMASK=255.255.255.0 BOOTPROTO=none ONBOOT=yes NM_CONTROLLED=no BONDING_OPTS="mode=balance-rr primary=eth0 miimon=500 updelay=1000" TYPE=Ethernet IPV6INIT=no USERCTL=no After the channel bonding interface is created, the network interfaces to be bound together must be configured by adding the MASTER and SLAVE directives to their configuration files. The configuration files for each of the channel-bonded interfaces can be nearly identical. For example, if two Ethernet interfaces are being channel bonded, both eth0 and eth1 may look like the following example: # cat /etc/sysconfig/network-scripts/ifcfg-eth0 DEVICE=eth0 HWADDR="2c:76:8a:54:77:a0" BOOTPROTO=none 8

Technical white paper | Networking best practices for HP ProLiant for Hadoop

ONBOOT=yes NM_CONTROLLED=no MASTER=bond0 SLAVE=yes USERCTL=no IPV6INIT=no

Server with 802.1q VLAN tagging In the Hadoop configuration each server has two NICs that are configured with two VLANs. The HP 5900 switch only allows VLAN 120 traffic on uplinks; all other traffic is blocked within the rack. The private VLAN 1120 that is used within the HP 5900 switch is not transmitted outside the HP ProLiant for Hadoop rack. The server is dual homed with an internal private IP and an external CEN IP. To permit CEN traffic from the server, the bonded NIC has to be configured with VLAN tagging. The following are the instructions on how to configure VLAN tagging on the servers. Ensure that the module 802.1q is loaded by entering the following command: lsmod | grep 8021q If the module is not loaded, load it with the following command: modprobe 8021q To configure the VLAN interface configuration, create a file in the /etc/sysconfig/network-scripts/ directory. The configuration filename should be the physical interface, plus a . (period) character, plus the VLAN ID number. For example, if the VLAN ID is 120, and the physical interface is eth0, then the configuration filename should be ifcfgeth0.120: # cat /etc/sysconfig/network-scripts/ifcfg-bond0.120 DEVICE=bond0.120 IPADDR=10.120.45.200 NETMASK=255.255.252.0 GATEWAY=10.120.45.1 BOOTPROTO=none ONBOOT=yes NM_CONTROLLED=no BONDING_OPTS="mode=balance-rr primary=eth0 miimon=500 updelay=1000" PEERDNS=no USERCTL=no VLAN=yes The following bonding policies are set on the servers. mode= Where allows you to specify the bonding policy and can be one of the following: • balance-rr or 0

Sets a round-robin policy for fault tolerance and load balancing. Transmissions are received and sent out sequentially on each bonded slave interface beginning with the first one available. • miimon=

Specifies (in milliseconds) how often MII link monitoring occurs. This is useful if high availability is required because MII is used to verify that the NIC is active. To verify that the driver for a particular NIC supports the MII tool, type the following command as root: # ethtool | grep "Link detected:" In this command, replace with the name of the device interface, such as eth0, not the bond interface. If MII is supported, the command returns: Link detected: yes If using a bonded interface for high availability, the module for each NIC must support MII. Setting the value to 0 (the default), turns this feature off.

9

Technical white paper | Networking best practices for HP ProLiant for Hadoop

• updelay=

Specifies (in milliseconds) how long to wait before enabling a link. The value must be a multiple of the value specified in the miimon parameter. The value is set to 0 by default, which disables it.

Enable tagged VLANs on HP 5900 The following instructions are for configuring the HP 5900 switch to support tagged VLANs configured on the servers described previously. 1.

Log into the network device via the console and enter system-view. system-view

2.

Use vlan vlan-id to create a VLAN. [HP]vlan 120 [HP-VLAN120]quit

3.

The following instructions will configure the range of ports on the switch, ports 1-18 and ports 45-48 on both the HP 5900 switches. Use the following command to enter interface range view to bulk configure multiple interfaces with the same feature instead of configuring them one by one. Use interface range name name interface interface-list to create an interface range, configure a name for the interface range, and enter the interface range view. Use interface range name name without the interface keyword to enter the view of an interface range with the specified name. interface range name Hadoop-net interface Ten-GigabitEthernet1/0/1 to TenGigabitEthernet1/0/18 Ten-GigabitEthernet2/0/1 to Ten-GigabitEthernet2/0/18 Ten-GigabitEthernet1/0/45 to Ten-GigabitEthernet1/0/48 TenGigabitEthernet2/0/45 to Ten-GigabitEthernet2/0/48 Commands in steps 4, 5 and 6 are configured on the ports specified in the command.

4.

Use port link-type to configure the link type of a port. Parameters: access: Configures the link type of a port as access. hybrid: Configures the link type of a port as hybrid. trunk: Configures the link type of a port as trunk. Usage guidelines: To change the link type of a port from trunk to hybrid or vice versa, first set the link type to access. A. The configuration made in Ethernet interface view applies only to the port. B. The configuration made in aggregate interface view applies to the aggregate interface and its aggregation member ports. C. If the system fails to apply the configuration to the aggregate interface, it stops applying the configuration to aggregation member ports. D. If the system fails to apply the configuration to an aggregation member port, it skips the port and moves to the next member port. E. The configuration made in S-channel interface view applies only to the interface. [HP-if-range-hadoop-net] port link-type hybrid

5.

10

Use undo port hybrid vlan to remove the hybrid ports from the specified VLANs. Use port hybrid vlan to assign the hybrid ports to the specified VLANs. Parameters: vlan-list: Specifies a list of existing VLANs in the format of: vlan-list = { vlan-id1 [ to vlan-id2 ] }& where vlan-id1 and vlan-id2 each are in the range of 1 to 4094, vlan-id2 cannot be smaller than vlan-id1, and & indicates that you can specify up to 10 vlan-id1 [ to vlan-id2 ] parameters. tagged: Configures the ports to send the packets of the specified VLANs without removing VLAN tags. untagged: Configures the ports to send the packets of the specified VLANs after removing VLAN tags.

Technical white paper | Networking best practices for HP ProLiant for Hadoop

Usage guidelines: A hybrid port can carry multiple VLANs. If you execute the port hybrid vlan command multiple times, the VLANs that the hybrid port allows are the VLANs that are specified by vlan-list in each execution. [HP-if-range-hadoop-net] undo port hybrid vlan 1 [HP-if-range-hadoop-net] port hybrid vlan 120 tagged [HP-if-range-hadoop-net] port hybrid vlan 1120 untagged 6.

Use the port hybrid pvid command to configure the PVID (Port VLAN ID) of the hybrid port. [HP-if-range-hadoop-net] port hybrid pvid vlan 1120 [HP-if-range-hadoop-net] quit

7.

Save the configuration. Once completed, exit the switch, and you are done.

BFD MAD enabled IRF The multi-active detection (MAD) feature detects identical active IRF virtual devices and handles multi-active collisions on a network. To detect and handle multi-active collisions, the MAD feature identifies each IRF virtual device with an active ID, which is the member ID of the master switch. If multiple identical active IRF virtual devices are detected, only the one that has the lowest active ID can operate in the active state and forward traffic. MAD sets all other IRF virtual devices in the recovery state, and shuts down all their physical ports but the console and IRF ports. BFD MAD is implemented by using the Bidirectional Forwarding Detection (BFD) protocol, which helps fast detect link failures and loss of IP connectivity. Figure 2 shows a link that is used between R01SW01 and R01SW02 HP5900 switches that is used for BFD MAD. Figure 4 shows the port that is used for BFD MAD configuration. BFD MAD is implemented by using the Bidirectional Forwarding Detection (BFD) protocol, which helps fast detect link failures and loss of IP connectivity. Set up dedicated BFD MAD links between each pair of IRF member switches. Do not use the BFD MAD links for data transmission. Assign port 44 of the BFD MAD links to a dedicated VLAN 3; create a VLAN interface for VLAN 3; and, assign a MAD IP address for each member switch. The MAD IP addresses are used for setting up BFD sessions between member switches, and they must be in the same network segment. Create VLAN 3, and add Ten-GigabitEthernet 1/0/44, Ten-GigabitEthernet 2/0/44 to VLAN 3. [HP] vlan 3 [HP-vlan3] port ten-gigabitethernet 1/0/44 ten-gigabitethernet 2/0/44 [HP-vlan3] quit Create VLAN-interface 3, and configure a MAD IP address for each member device on the VLAN interface. [HP] interface vlan-interface 3 [HP-Vlan-interface3] mad bfd enable [HP-Vlan-interface3] mad ip address 192.168.2.1 24 member 1 [HP-Vlan-interface3] mad ip address 192.168.2.2 24 member 2 [HP-Vlan-interface3] quit Disable the spanning tree feature on Ten-GigabitEthernet 1/0/44, Ten-GigabitEthernet 2/0/44. [HP] interface ten-gigabitethernet [HP-Ten-GigabitEthernet1/0/1] undo [HP-Ten-GigabitEthernet1/0/1] quit [HP] interface ten-gigabitethernet [HP-Ten-GigabitEthernet2/0/1] undo [HP-Ten-GigabitEthernet2/0/1] quit

1/0/44 stp enable 2/0/44 stp enable

Save the configuration of the switch. To view the configuration of BFD MAD on the switch: disp mad verbose Current MAD status: Detect Excluded ports(configurable): Excluded ports(can not be configured): FortyGigE1/0/51 FortyGigE1/0/52 FortyGigE2/0/51 FortyGigE2/0/52

11

Technical white paper | Networking best practices for HP ProLiant for Hadoop

MAD ARP disabled. MAD ND disabled. MAD LACP disabled. MAD BFD enabled interface: Vlan-interface3 mad ip address 192.168.2.1 255.255.255.0 member 1 mad ip address 192.168.2.2 255.255.255.0 member 2

HP ProLiant for Hadoop data center connectivity Following sections in the document focus on configuring HP 5900 switch to connect to customer datacenter network. • Configuring HP 5900 network switch • Configuring CISCO switch for HP 5900

Configuring HP 5900 network switch This section provides instructions for configuring a network switch for customer uplink(s). Configure customer uplinks using ports 19 through 32 10GbE ports or 49 and 50 40GbE ports on the HP 5900 switch that are labeled as customer in figures 3 and 4. The example below shows how to configure one port. If multiple ports need to be configured, repeat these steps for each port. If the customer wants to use 40GbE uplinks, “HP X240 40G QSFP+ to QSFP+ 5m Direct Attach Copper Cable (JG328A)” cables need to be ordered to connect to the customer’s spine switches with 40GbE connectivity. Additional 40GbE connectivity options are available to customers, for additional details refer to the QuickSpecs for HP 5900 switch series: • HP X240 40G QSFP+ to 4x10G SFP+ 1m Direct Attach Copper Splitter Cable (JG329A) • HP X140 40G QSFP+ MPO SR4 Transceiver (JG325B)

The 40GbE Ports 49 and 50 on the HP 5900 can be configured using the instructions below.

Note Port numbers will be formatted differently depending on the model and how the switch is configured. For example, a switch configured to use Intelligent Resilient Framework (IRF) will also include a chassis number as part of the port number.

In this example, we will be making a two-port link aggregation group in an IRF configuration. You can see that this is a two switch IRF configuration by observing the port number scheme of chassis/slot/port. A scheme of 1/0/43 means chassis 1, or switch 1, slot 0, port 43. A scheme of 2/0/43 means chassis 2, or switch 2, slot 0, port 43. In the following example, port 43 from each switch, r001sw001 and r001sw002, is configured. You can use any other ports to configure the bridge aggregation that provides an uplink from the switch. If you use 40GbE QSFP port 49 or 50, then you will have to replace “interface Ten-GigabitEthernet” to “interface FortyGigE”; other instruction are the same.

12

1.

Log into the network device via the console and enter system-view. system-view

2.

Create the Bridge Aggregation interface to contain the uplinks from your server. In this example we will be creating the interface of Bridge Aggregation 1. Your numbering may vary depending on the current configuration on the switch you are using. [HP] interface Bridge-Aggregation 1

3.

Give your new interface a description to ease identification: [HP-Bridge-Aggregation1] description bridge-aggregation-to-customer-network

4.

By default, an aggregation group operates in static aggregation mode. So you need to configure the aggregation group to operate in dynamic aggregation mode. [HP-Bridge-Aggregation1] link-aggregation mode dynamic

5.

Type quit to exit the bridge aggregation: [HP-Bridge-Aggregation1] quit

Technical white paper | Networking best practices for HP ProLiant for Hadoop

6.

Enter the first interface that you will be aggregating. The example below adds port 43 in the bottom switch to link aggregation group 1: [HP] interface Ten-GigabitEthernet 1/0/43

7.

Put the port in the link aggregation group: [HP-Ten-GigabitEthernet1/0/43] port link-aggregation group 1

8.

Enable the interface. If it is already enabled, it will tell you that the interface is not shut down. [HP-Ten-GigabitEthernet1/0/43] undo shutdown

9.

Type “quit” and repeat steps 6-7 for all interfaces that will be in the link aggregation group. The example below adds port 43 in the second switch to link aggregation group 1. [HP] interface Ten-GigabitEthernet 2/0/43 [HP-Ten-GigabitEthernet2/0/43] port link-aggregation group 1 [HP-Ten-GigabitEthernet2/0/43] undo shutdown

10. Return to the Bridge Aggregation for the final configuration: [HP] interface Bridge-Aggregation 1

Note If you get an error similar to “Error: Failed to configure on interface…” during any of the following steps, you will need to run the following command on the interface that has the error and then re-run step 6-7. [HP] interface Ten-GigabitEthernet 1/0/43 [HP-Ten-GigabitEthernet1/0/43] default This command will restore the default settings. Continue? [Y/N]: Y If the default command is not available: [HP-Ten-GigabitEthernet1/0/43] port link-type access

11. Change the port type to a trunk: [HP-Bridge-Aggregation1] port link-type trunk 12. Enable the interface: [HP-Bridge-Aggregation1] undo shutdown 13. Set the Port Default VLAN ID (PVID) of the connection. The PVID is the VLAN ID the switch will assign to all untagged frames (packets) received on each port. Another term for this would be your untagged or native VLAN. By default, it is set to 1, but you will want to change it if your network is using another VLAN ID for your untagged traffic. [HP-Bridge-Aggregation1] port trunk pvid vlan 120 14. If you configured your vSwitch to pass multiple VLAN tags, you can configure your bridge aggregation link at this time by running the following command. Repeat for all the VLANs you need to pass through that connection. [HP-Bridge-Aggregation1] port trunk permit vlan 120 Please wait... Done. Configuring Ten-GigabitEthernet1/0/43... Done. Configuring Ten-GigabitEthernet2/0/43... Done. 15. If you set your PVID to something other than the default 1, you will want to remove that VLAN 1 and repeat step 13 for your PVID VLAN. [HP-Bridge-Aggregation1] undo port trunk permit vlan 1 Please wait... Done. Configuring Ten-GigabitEthernet1/0/43... Done.

13

Technical white paper | Networking best practices for HP ProLiant for Hadoop

Configuring Ten-GigabitEthernet2/0/43... Done. 16. Now display your new Bridge Aggregation interface to ensure things are set up correctly. You will want to make sure your PVID is correct, and that you are both passing and permitting the VLAN you defined. In this example, we are not passing the untagged traffic (PVID 1) and only packets tagged with VLAN ID 120. You will also want to make sure your interfaces are up, and you are running at the correct speed, two 10Gbps links would give you 20Gbps of aggregated performance. [HP] display interface Bridge-Aggregation 1 Bridge-Aggregation1 current state: UP IP Packet Frame Type: PKTFMT_ETHNT_2, Hardware Address: 000f-e207-f2e0 Description: bridge-aggregation-to-customer-network 20Gbps-speed mode, full-duplex mode Link speed type is autonegotiation, link duplex type is autonegotiation PVID: 120 Port link-type: trunk VLAN passing : 120 VLAN permitted: 120 Trunk port encapsulation: IEEE 802.1q … Output truncated… 17. Now check to make sure the trunk was formed correctly. If both connections have something other than “S” for the status, here are a few troubleshooting steps. If none of these work, then delete and recreate the bridge aggregation and reset all the ports back to default. Ensure that: A. You configured the interfaces correctly. B. You enabled (undo shutdown) the port on the switch. C. The VLANs being passed/permitted match that of the group. D. The port is connected to the switch on the interface you specified and is connected and enabled on the server: [HP] display link-aggregation verbose Bridge-Aggregation 1 Loadsharing Type: Shar -- Loadsharing, NonS -- Non-Loadsharing Port Status: S -- Selected, U -- Unselected Flags: A -- LACP_Activity, B -- LACP_Timeout, C -- Aggregation, D -- Synchronization, E -- Collecting, F -- Distributing, G -- Defaulted, H -- Expired Aggregation Interface: Bridge-Aggregation1 Aggregation Mode: Static Loadsharing Type: Shar Port Status Oper-Key --------------------------------------------------------------------------XGE1/0/43 S 2 XGE2/0/43 S 2 18. Save the configuration. Once completed, exit the switch, and you are done.

Configure Cisco switch for HP 5900 The following steps are for configuring Cisco switches to communicate with HP 5900 networking switches on HP ProLiant for Hadoop. In the following example two interface ports are configured as uplinks from the 5900 network switch to the customer Cisco switch. The ports only allow VLAN 120 traffic; all other traffic is blocked from the customer enterprise network. The private VLAN 1120 that is used within the HP 5900 switch is not transmitted outside the HP ProLiant for Hadoop rack.

14

1.

Specify the port-channel interface to configure, and enter the interface configuration mode. The range is from 1 to 4096. [Cisco]interface Port-channel 1

2.

Configure the interface for Layer 2 switching. Enter the switchport command once without any keywords to configure the interface as a Layer 2 port before you can enter additional switchport commands with keywords.

Technical white paper | Networking best practices for HP ProLiant for Hadoop

switchport 3.

Specify the encapsulation command with the dot1q keyword to support the IEEE standard. switchport trunk encapsulation dot1q

4.

For 802.1Q trunks, specify the native VLAN. Note: If you do not set the native VLAN, the default is used (VLAN 1). switchport trunk native vlan 120

5.

Configure the list of VLANs allowed on the trunk. All VLANs are allowed by default. switchport trunk allowed vlan 120

6.

Remove VLAN 1 from the trunk. switchport trunk allowed vlan remove 1

7.

Configure the interface as a Layer 2 trunk. (Required only if the interface is a Layer 2 access port or to specify the trunking mode.) switchport mode trunk

8.

Specify the interface to configure. Specify the port mode for the link in a port channel. After LACP is enabled, configure each link or the entire channel as active or passive. When you run port channels with no associated protocol, the port-channel mode is always on. The default port-channel mode is on. interface GigabitEthernet1/1 switchport switchport trunk encapsulation dot1q switchport trunk native vlan 120 switchport trunk allowed vlan 120 switchport trunk allowed vlan remove 1 switchport mode trunk channel-group 1 mode active

9.

Specify other interfaces that connect to the HP networking switches. interface GigabitEthernet1/2 switchport switchport trunk encapsulation dot1q switchport trunk native vlan 120 switchport trunk allowed vlan 120 switchport trunk allowed vlan remove 1 switchport mode trunk channel-group 1 mode active

10. Verify port-channel 1 is correctly formed with the 5900 switch. show interfaces port-channel 1 etherchannel

15

Technical white paper | Networking best practices for HP ProLiant for Hadoop

Appendix A: HP 5900 switch configuration #

irf-port 2/1

version 7.1.045, Release 2307

port group interface FortyGigE2/0/51

#

#

sysname HP

irf-port 2/2

#

port group interface FortyGigE2/0/52

irf domain 1001

#

irf mac-address persistent timer

interface Bridge-Aggregation1

irf auto-update enable

description bridge to TOR to allow VLAN 120

irf link-delay

port link-type trunk

irf member 1 priority 9

undo port trunk permit vlan 1

irf member 2 priority 1

port trunk permit vlan 120

#

#

lldp global enable

interface Vlan-interface120

#

ip address 10.120.46.60 255.255.252.0

interface range name Hadoop-net interface TenGigabitEthernet1/0/1 to Ten-GigabitEthernet1/0/18 TenGigabitEthernet2/0/1 to Ten-GigabitEthernet2/0/18 TenGigabitEthernet1/0/45 to Ten-GigabitEthernet1/0/47 TenGigabitEthernet2/0/45 to Ten-GigabitEthernet2/0/47

#

# system-working-mode standard password-recovery enable # vlan 1 # vlan 120 description Public VLAN # vlan 1120 description Private VLAN # irf-port 1/1 port group interface FortyGigE1/0/51 # irf-port 1/2 port group interface FortyGigE1/0/52

interface Vlan-interface1120 ip address 172.24.1.210 255.255.255.0 # interface FortyGigE1/0/49 shutdown # interface FortyGigE1/0/50 shutdown # interface FortyGigE2/0/49 # interface FortyGigE2/0/50 # interface FortyGigE1/0/51 # interface FortyGigE1/0/52 # interface FortyGigE2/0/51 # interface FortyGigE2/0/52 #

(Continued on next page)

16

(Continued on next page)

Technical white paper | Networking best practices for HP ProLiant for Hadoop

interface M-GigabitEthernet0/0/0 ip address 172..24.1.210 255.255.255.0

#

#

interface Ten-GigabitEthernet1/0/45

interface Ten-GigabitEthernet1/0/1

port link-type hybrid

port link-type hybrid

undo port hybrid vlan 1

undo port hybrid vlan 1

port hybrid vlan 120 tagged

port hybrid vlan 120 tagged

port hybrid vlan 1120 untagged

port hybrid vlan 1120 untagged

port hybrid pvid vlan 1120

port hybrid pvid vlan 1120

#

#

interface Ten-GigabitEthernet1/0/46

.

port link-type hybrid

.

undo port hybrid vlan 1

.

port hybrid vlan 120 tagged

.

port hybrid vlan 1120 untagged

#

port hybrid pvid vlan 1120

interface Ten-GigabitEthernet1/0/16

#

port link-type hybrid

interface Ten-GigabitEthernet1/0/47

undo port hybrid vlan 1

port link-type hybrid

port hybrid vlan 120 tagged

undo port hybrid vlan 1

port hybrid vlan 1120 untagged

port hybrid vlan 120 tagged

port hybrid pvid vlan 1120

port hybrid vlan 1120 untagged

#

port hybrid pvid vlan 1120

interface Ten-GigabitEthernet1/0/17

#

#

interface Ten-GigabitEthernet1/0/48

interface Ten-GigabitEthernet1/0/18

#

# . . . interface Ten-GigabitEthernet1/0/43 # interface Ten-GigabitEthernet1/0/43 port link-type trunk port trunk permit vlan 120 undo port trunk permit vlan 1 port link-aggregation group 1 #

(Continued top of previous page)

17

Technical white paper | Networking best practices for HP ProLiant for Hadoop

Appendix B: Glossary of terms Table 1. Glossary

18

Term

Definition

Bridge Aggregation

Comware OS terminology for Link Aggregation.

Distributed Trunking (DT)

A link aggregation technique, where two or more links across two switches are aggregated together to form a trunk.

IEEE 802.3ad

An industry standard protocol that allows multiple links/ports to run in parallel, providing a virtual single link/port. The protocol provides greater bandwidth, load balancing, and redundancy.

Intelligent Resilient Framework (IRF)

Technology in certain HP Networking switches that enables the ability to connect similar devices together to create a virtualized distributed device. This virtualization technology realizes the cooperation, unified management, and non-stop maintenance of multiple devices.

LACP

Link Aggregation Control Protocol (see IEEE 802.3ad or 802.1ax)

Port Aggregation

Combining ports to provide one or more of the following benefits: greater bandwidth, load balancing, and redundancy.

Port Bonding

A term typically used in the UNIX®/Linux world that is synonymous to NIC teaming in the Windows world.

Spanning Tree Protocol (STP)

Spanning Tree Protocol (STP) is standardized as IEEE 802.1D and ensures a loop-free topology for any bridged Ethernet local area network by preventing bridge loops and the broadcast traffic that results from them.

Trunking

Combining ports to provide one or more of the following benefits: greater bandwidth, load balancing, and redundancy.

Technical white paper | Networking best practices for HP ProLiant for Hadoop

For more information For more information on HP Networking products, visit hp.com/go/networking FlexNetwork Architecture Delivers Higher Speed, Lower Downtime With HP IRF Technology, http://www3.networktest.com/hpirf/hpirf1.pdf HP ConvergedSystem for Big Data, hp.com/go/convergedsystem/bigdata HP Solutions for Apache Hadoop, hp.com/go/hadoop ProLiant servers, hp.com/go/proliant

To help us improve our documents, please provide feedback at hp.com/solutions/feedback.

Sign up for updates hp.com/go/getupdated © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein. Microsoft and Windows are U.S. registered trademarks of Microsoft Corporation. Red Hat is a registered trademark of Red Hat, Inc. in the United States and other countries. UNIX is a registered trademark of The Open Group. 4AA5-3279ENW, May 2014