Protocols for Data Center Network Virtualization and Cloud Computing
Washington University in Saint Louis Saint Louis, MO 63130
[email protected] Tutorial at 2014 IEEE 15th International Conference on High Performance Switching and Routing, Vancouver, Canada July 1, 2014 Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
1
©2014 Raj Jain
Overview 1. Part I: Network Virtualization 2. Part II: Data Center Bridging 3. Part III: Carrier Ethernet for Data Centers Break 4. Part IV: Virtual Bridging 5. Part V: LAN Extension and Partitioning Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
2
©2014 Raj Jain
Part I: Network Virtualization
1. Virtualization 2. Why Virtualize? 3. Network Virtualization 4. Names, IDs, Locators 5. Interconnection Devices Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
3
©2014 Raj Jain
Part II: Data Center Bridging
1. 2. 3. 4. 5.
Residential vs. Data Center Ethernet Review of Ethernet devices and algorithms Enhancements to Spanning Tree Protocol Virtual LANs Data Center Bridging Extensions
Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
4
©2014 Raj Jain
Part III: Carrier Ethernet for Data Centers
Provider Bridges (PB) or Q-in-Q 2. Provider Backbone Bridges (PBB) or MAC-in-MAC 3. Provider Backbone Bridges with Traffic Engineering (PBBTE) Note: Although these technologies were originally developed for carriers, they are now used inside multi-tenant data centers (clouds) 1.
Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
5
©2014 Raj Jain
Part IV: Virtual Bridging
1. 2. 3. 4. 5.
Virtual Bridges to connect virtual machines IEEE Virtual Edge Bridging Standard Single Root I/O Virtualization (SR-IOV) Aggregating Bridges and Links: VSS and vPC Bridges with massive number of ports: VBE
Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
6
©2014 Raj Jain
Part V: LAN Extension and Partitioning
1. 2. 3. 4.
Transparent Interconnection of Lots of Links (TRILL) Network Virtualization using GRE (NVGRE) Virtual eXtensible LANs (VXLAN) Stateless Transport Tunneling Protocol (STT)
Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
7
©2014 Raj Jain
Part I: Network Virtualization
1. Virtualization 2. Why Virtualize? 3. Network Virtualization 4. Names, IDs, Locators 5. Interconnection Devices Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
8
©2014 Raj Jain
Virtualization “Virtualization means that Applications can use a resource without any concern for where it resides, what the technical interface is, how it has been implemented, which platform it uses, and how much of it is available.” -Rick F. Van der Lans in Data Virtualization for Business Intelligence Systems
Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
9
©2014 Raj Jain
5 Reasons to Virtualize 1. 2. 3. 4.
5.
Sharing: Break up a large resource Large Capacity or high-speed E.g., Servers Isolation: Protection from other tenants E.g., Virtual Private Network Aggregating: Combine many resources in to one, e.g., storage Dynamics: Fast allocation, Change/Mobility, Follow the sun (active users) or follow the moon (cheap power) Ease of Management Easy distribution, deployment, testing
Washington University in St. Louis
10Gb
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
10
Switch Switch Switch Switch
©2014 Raj Jain
Virtualization in Computing
Storage: Virtual Memory L1, L2, L3, ... Recursive Virtual CDs, Virtual Disks (RAID), Cloud storage Computing: Virtual Desktop Virtual Server Virtual Datacenter Thin Client VMs Cloud Networking: Plumbing of computing Virtual Channels, Virtual LANs, Virtual Private Networks
Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
11
©2014 Raj Jain
Network Virtualization 1.
2.
3. 4.
Network virtualization allows tenants to form an overlay network in a multi-tenant network such that tenant can control: 1. Connectivity layer: Tenant network can be L2 while the provider is L3 and vice versa 2. Addresses: MAC addresses and IP addresses 3. Network Partitions: VLANs and Subnets 4. Node Location: Move nodes freely Network virtualization allows providers to serve a large number of tenants without worrying about: 1. Internal addresses used in client networks 2. Number of client nodes 3. Location of individual client nodes 4. Number and values of client partitions (VLANs and Subnets) Network could be a single physical interface, a single physical machine, a data center, a metro, … or the global Internet. Provider could be a system owner, an enterprise, a cloud provider, or a carrier.
Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
12
©2014 Raj Jain
Network Virtualization Techniques Entity NIC Switch L2 Link L2 Network using L2
Partitioning SR-IOV VEB, VEPA VLANs VLAN
L2 Network using L3 NVO3, VXLAN, NVGRE, STT Router VDCs, VRF L3 Network using L1 L3 Network using MPLS, GRE, L3* PW, IPSec Application ADCs
Aggregation/Extension/Interconnection** MR-IOV VSS, VBE, DVS, FEX LACP, Virtual PortChannels PB (Q-in-Q), PBB (MAC-in-MAC), PBB-TE, Access-EPL, EVPL, EVP-Tree, EVPLAN MPLS, VPLS, A-VPLS, H-VPLS, PWoMPLS, PWoGRE, OTV, TRILL, LISP, L2TPv3, EVPN, PBB-EVPN VRRP, HSRP GMPLS, SONET MPLS, T-MPLS, MPLS-TP, GRE, PW, IPSec Load Balancers
*All L2/L3 technologies for L2 Network partitioning and aggregation can also be used for L3 network partitioning and aggregation, respectively, by simply putting L3 packets in L2 payloads. **The aggregation technologies can also be seen as partitioning technologies from the provider point of view. Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
13
©2014 Raj Jain
Names, IDs, Locators Name: John Smith ID: 012-34-5678 Locator: 1234 Main Street Big City, MO 12345 USA
Locator changes as you move, ID and Names remain the same. Examples: Names: Company names, DNS names (Microsoft.com) IDs: Cell phone numbers, 800-numbers, Ethernet addresses, Skype ID, VOIP Phone number Locators: Wired phone numbers, IP addresses
Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
14
©2014 Raj Jain
Interconnection Devices LAN= Collision Domain Application Transport Network Datalink Physical Washington University in St. Louis
H H
B
H H
Gateway Router Bridge/Switch Repeater/Hub
Extended LAN =Broadcast domain Router Application Transport Network Datalink Physical
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
15
©2014 Raj Jain
Interconnection Devices (Cont)
Repeater: PHY device that restores data and collision signals Hub: Multiport repeater + fault detection and recovery Bridge: Datalink layer device connecting two or more collision domains. MAC multicasts are propagated throughout “extended LAN.” Router: Network layer device. IP, IPX, AppleTalk. Does not propagate MAC multicasts. Switch: Multiport bridge with parallel paths These are functions. Packaging varies. No CSMA/CD in 10G and up No CSMA/CD in practice now even at home or at 10 Mbps
Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
16
©2014 Raj Jain
Fallacies Taught in Networking Classes 1. 2. 3. 4. 5. 6. 7. 8.
Ethernet is a local area network (Local < 2km) Token ring, Token Bus, and CSMA/CD are the three most common LAN access methods. Ethernet uses CSMA/CD. Ethernet bridges use spanning tree for packet forwarding. Ethernet frames are limited to 1518 bytes. Ethernet does not provide any delay guarantees. Ethernet has no congestion control. Ethernet has strict priorities. Ethernet has changed. All of these are now false or are becoming false.
Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
17
©2014 Raj Jain
Summary of Part I
1. 2. 3.
4.
Virtualization allows applications to use resources without worrying about its location, size, format etc. Ethernet’s use of IDs as addresses makes it very easy to move systems in the data center Keep traffic on the same Ethernet Cloud computing requires Ethernet to be extended globally and partitioned for sharing by a very large number of customers who have complete control over their address assignment and connectivity Many of the previous limitations of Ethernet have been overcome in the last few years.
Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
18
©2014 Raj Jain
Levels of Network Virtualization
NIC
Bridge L2 Network
Router L3 Network
Data Center
Networks consist of: Network Interface Card (NIC) – L2 Links - L2 Bridges - L2 Networks - L3 Links - L3 Routers - L3 Networks – Data Centers – Global Internet. Each of these needs to be virtualized
Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
19
©2014 Raj Jain
Part II: Data Center Bridging
1. 2. 3. 4. 5.
Residential vs. Data Center Ethernet Review of Ethernet devices and algorithms Enhancements to Spanning Tree Protocol Virtual LANs Data Center Bridging Extensions
Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
20
©2014 Raj Jain
Residential vs. Data Center Ethernet Residential Distance: up to 200m Scale: Few MAC addresses 4096 VLANs Protection: Spanning tree
Data Center/Cloud No limit
Millions of MAC Addresses Millions of VLANs Q-in-Q Rapid spanning tree, … (Gives 1s, need 50ms) Traffic engineered path
Path determined by spanning tree Simple service
Priority Aggregate QoS No performance/Error monitoring (OAM)
Service Level Agreement. Rate Control. Need per-flow/per-class QoS
Need performance/BER
Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
21
©2014 Raj Jain
Spanning Tree and its Enhancements
Helps form a tree out of a mesh topology A topology change can result in 1 minute of traffic loss with STP All TCP connections break Rapid Spanning Tree Protocol (RSTP) IEEE 802.1w-2001 incorporated in IEEE 802.1D2004 One tree for all VLANs Common spanning tree Many trees Multiple spanning tree (MST) protocol IEEE 802.1s-2002 incorporated in IEEE 802.1Q2005 One or more VLANs per tree.
Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
22
©2014 Raj Jain
IS-IS Protocol
Intermediate System to Intermediate System (IS-IS) is a protocol to build routing tables. Link-State routing protocol Each nodes sends its connectivity (link state) information to all nodes in the network Dijkstra’s algorithm is then used by each node to build its routing table. Similar to OSPF (Open Shortest Path First). OSPF is designed for IPv4 and then extended for IPv6. IS-IS is general enough to be used with any type of addresses OSPF is designed to run on the top of IP IS-IS is general enough to be used on any transport Adopted by Ethernet
Ref: http://en.wikipedia.org/wiki/IS-IS Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
23
©2014 Raj Jain
Shortest Path Bridging
IEEE 802.1aq-2012 Allows all links to be used Better CapEx IS-IS link state protocol Switch Switch Aggregation (similar to OSPF) is used to build shortest path trees for each node to Switch Switch Switch Switch Access every other node within the SPB domain Server1 Server2 Server3 Server4 Equal-cost multi-path (ECMP) used to distribute load
Ref: http://en.wikipedia.org/wiki/Shortest_Path_Bridging http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm Washington University in St. Louis
24
©2014 Raj Jain
What is a LAN? Server Client 1 Bridge Client n Router LAN 1
LAN 2 Router
LAN = Single broadcast domain = Subnet No routing between members of a LAN Routing required between LANs
Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
25
©2014 Raj Jain
Virtual LAN R S
S
S R
Virtual LAN = Broadcasts and multicast goes only to the nodes in the virtual LAN LAN membership defined by the network manager Virtual
Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
26
©2014 Raj Jain
IEEE 802.1Q-2011 Tag
Tag Protocol Identifier (TPI) Priority Code Point (PCP): 3 bits = 8 priorities 0..7 (High) Canonical Format Indicator (CFI): 0 Standard Ethernet, 1 IBM Token Ring format (non-canonical or non-standard) CFI now replaced by Drop Eligibility Indicator (DEI) VLAN Identifier (12 bits 4095 VLANs) Switches forward based on MAC address + VLAN ID Unknown addresses are flooded.
Untagged Frame
DA SA T/L Data CRC 32b IEEE 802.1Q-2011 Header
Tagged Frame
DA SA TPI Priority CFI/DEI VLAN ID T/L Data CRC 48b 48b 16b
3b
1b
12b
16b
32b
Ref: Canonical vs. MSB Addresses, http://support.lexmark.com/index?page=content&id=HO1299&locale=en&userlocale=EN_US Ref: G. Santana, “Data Center Virtualization Fundamentals,” Cisco Press, 2014, ISBN:1587143240 http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm Washington University in St. Louis ©2014 Raj Jain
27
Data Center Bridging (DCB)
Goal: To enable storage traffic over Ethernet Four Standards: Priority-based Flow Control (IEEE 802.1Qbb-2011) Enhanced Transmission Selection (IEEE 802.1Qaz-2011) Congestion Control (IEEE 802.1Qau-2010) Data Center Bridging Exchange (IEEE 802.1Qaz-2011)
Ref: M. Hagen, “Data Center Bridging Tutorial,” http://www.iol.unh.edu/services/testing/dcb/training/DCB-Tutorial.pdf Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
28
©2014 Raj Jain
Ethernet Flow Control: Pause Frame Switch 1
Switch 2 Pause
Defined in IEEE 802.3x-1997. A form of on-off flow control. A receiving switch can stop the adjoining sending switch by sending a “Pause” frame. Stops the sender from sending any further information for a time specified in the pause frame. The frame is addressed to a standard (well-known) multicast address. This address is acted upon but not forwarded. Stops all traffic. Causes congestion backup.
Ref: http://en.wikipedia.org/wiki/Ethernet_flow_control http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm Washington University in St. Louis
29
©2014 Raj Jain
Priority-based Flow Control (PFC) Priority 0 Pause
…
…
Priority 1
Priority 7
IEEE 802.1Qbb-2011 IEEE 802.1Qbb-2011 allows any single priority to be stopped. Others keep sending
Ref: J. L. White, “Technical Overview of Data Center Networks,” SNIA, 2013, http://www.snia.org/sites/default/education/ tutorials/2012/fall/networking/JosephWhite_Technical%20Overview%20of%20Data%20Center%20Networks.pdf http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm Washington University in St. Louis ©2014 Raj Jain
30
Enhanced Transmission Selection
IEEE 802.1Qaz-2011 Goal: Guarantee bandwidth for applications sharing a link Traffic is divided in to 8 classes (not priorities) The classes are grouped. Standard requires min 3 groups: 1 with PFC (Storage with low loss), 1 W/O PFC (LAN), 1 Strict Priority (Inter-process communication and VOIP with low latency) LAN Group 1
Groups Classes
0
1
Group 2 2
3
4
Best Effort
Group 3 5
6
7
Storage Low Loss Inter-Process Comm+VOIP Low Delay
Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
31
©2014 Raj Jain
ETS (Cont) Transmit Queue 0
Class Group 3
50 50
Transmit Queue 1
10
Transmit Queue 2
Class Group 2
Transmit Queue 4
20 30
20
50
20 30
50
Transmit Queue 6
20 20 30
20 20 30
t=1 t=2 t=3
t=1 t=2 t=3
Transmit Queue 7
10
Transmit Queue 3
Transmit Queue 5
Class Group 1
50
Bandwidth allocated per class group in 1% increment but 10% precision (±10% error). Max 75% allocated Min 25% best effort Fairness within a group All unused bandwidth is available to all classes wanting more bandwidth. Allocation algorithm not defined. Example: Group 1=20%, Group 2=30% http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
Washington University in St. Louis
©2014 Raj Jain
32
Quantized Congestion Notification (QCN) Source
Switch
Switch
Destination
Switch
Destination
IEEE 802.1Qau-2010 Dynamic Congestion Notification A source quench message is sent by the congested switch direct to the source. The source reduces its rate for that flow. Sources need to keep per-flow states and control mechanisms Easy for switch manufacturers but complex for hosts. Implemented in switches but not in hosts Not effective. The source may be a router in a subnet and not the real source Router will drop the traffic. QCN does not help in this case.
Ref: I. Pepelnjak, “DCB Congestion Notification (802.1Qau),” http://blog.ipspace.net/2010/11/data-center-bridging-dcb-congestion.html http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm Washington University in St. Louis ©2014 Raj Jain
33
DCBX
Data Center Bridging eXchange, IEEE 802.1Qaz-2011 Uses LLDP (Link Level Discovery Protocol) to negotiate quality metrics and capabilities for Priority-based Flow Control, Enhanced Transmission Selection, and Quantized Congestion Notification New TLV’s Priority group definition Group bandwidth allocation PFC enablement per priority QCN enablement DCB protocol profiles FCoE and iSCSI profiles
Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
34
©2014 Raj Jain
Summary of Part II
1. 2. 3. 4.
Ethernet’s use of IDs as addresses makes it very easy to move systems in the data center Keep traffic on the same Ethernet Spanning tree is wasteful of resources and slow. Ethernet now uses shortest path bridging (similar to OSPF) VLANs allow different non-trusting entities to share an Ethernet network Data center bridging extensions reduce the packet loss by enhanced transmission selection and Priority-based flow control
Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
35
©2014 Raj Jain
Part III: Carrier Ethernet for Data Centers
Provider Bridges (PB) or Q-in-Q 2. Provider Backbone Bridges (PBB) or MAC-in-MAC 3. Provider Backbone Bridges with Traffic Engineering (PBBTE) Note: Although these technologies were originally developed for carriers, they are now used inside multi-tenant data centers (clouds) 1.
Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
36
©2014 Raj Jain
Ethernet Provider Bridge (PB) Customer A VLANs 1-100
CE
PE
Customer B VLANs 1-100
CE
PE
Service Provider S-VLAN 1 S-VLAN 2
PE
CE
Customer A VLANs 1-100
PE
CE
Customer B VLANs 1-100
IEEE 802.1ad-2005 incorporated in IEEE 802.1Q-2011 Problem: Multiple customers may have the same VLAN ID. How to keep them separate? Solutions: 1. VLAN translation: Change customer VLANs to provider VLANs and back 2. VLAN Encapsulation: Encapsulate customer frames
Ref: D. Bonafede, “Metro Ethernet Network,” http://www.cicomra.org.ar/cicomra2/asp/TUTORIAL-%20Bonafede.pdf
Ref: P. Thaler, et al., “IEEE 802.1Q,” IETF tutorial, March 10 2013, http://www.ietf.org/meeting/86/tutorials/86-IEEE-8021-Thaler.pdf Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
37
©2014 Raj Jain
Provider Bridge (Cont)
Q-in-Q Encapsulation: Provider inserts a service VLAN tag VLAN translation Changes VLANs using a table Allows 4K customers to be serviced. Total 16M VLANs 8 Traffic Classes using Differentiated Services Code Points (DSCP) for Assured Forwarding S-Tag C-DA C-SA Type S-VID Type C-VID Type Payload 88A8 8100 48b 48b 16b 16b 16b 16b 16b
Washington University in St. Louis
Priority
CFI
S-VLAN ID
3b
1b
12b
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
38
©2014 Raj Jain
Provider Backbone Network (PBB) Subscriber Provider Subscriber Subscriber
Provider Backbone
Provider
Provider
Subscriber
Subscriber
Problem: Number of MAC addresses passing through backbone bridges is too large for all core bridge to remember Broadcast and flooded (unknown address) frames give unwanted traffic and security issues Solution: IEEE 802.1ah-2008 now in 802.1Q-2011 Add new source/destination MAC addresses pointing to ingress backbone bridge and egress backbone bridge Core bridges onlyhttp://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm know edge bridge addresses
Washington University in St. Louis
©2014 Raj Jain
39
MAC-in-MAC Frame Format
Provider backbone edge bridges (PBEB) forward to other PBEB’s and learn customer MAC addresses PB core bridges do not learn customer MACs B-DA = Destination backbone bridge address Edge Determined by Customer Destination Address Edge Backbone VLANs delimit the broadcast domains in the backbone
Core Core
Edge Edge
Backbone
B-DA B-SA Type B-VID Type I-SID C-DA C-SA Type S-VID Type C-VID Type Payload 88A8 88E7 88A8 8100 48b
48b
16b
16b
16b
32b
48b
48b
16b
16b
16b
16b
16b
I-Tag
PBB Core switches forward based on Backbone Destination Bridge Address and Backbone-VLAN ID (60 bits) Similar to 802.1ad Q-in-Q. Therefore, same EtherType.
Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
40
©2014 Raj Jain
PBB Service Instance
Service instance ID (I-SID) indicates a specific flow All frames on a specific port, or All frames on a specific port with a specific service VLAN, or All frames on a specific port with a specific service VLAN and a specific customer VLAN SID 1 20 33 401 502
Definition Port 1 Port 2, S-VLAN=10 Port 2, S-VLAN=20 Port 2, S-VLAN=30, C-VLAN=100 Port 3, S-VLAN=40, C-VLAN=200
Washington University in St. Louis
B-VLAN 1 3 6 4 4
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
41
Port 1 Port 2
B-VLAN=1 B-VLAN=3 B-VLAN=6
Port 3
B-VLAN=4
Service Instance Mapping ©2014 Raj Jain
Connection Oriented Ethernet
Connectionless: Path determined at forwarding Varying QoS Connection Oriented: Path determined at provisioning Path provisioned by management Deterministic QoS No spanning tree, No MAC address learning, Frames forwarded based on VLAN Ids and Backbone bridges addresses Path not determined by customer MAC addresses and other customer fields More Secure Reserved bandwidth per EVC Pre-provisioned Protection path Better availability Working Path CE
PE
PE
CE
Protection Path Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
42
©2014 Raj Jain
PBB-TE
Provider Backbone Bridges with Traffic Engineering (PBB-TE) IEEE 802.1Qay-2009 now in 802.1Q-2011 Provides connection oriented P2P (E-Line) Ethernet service For PBB-TE traffic VLANs: Turn off MAC learning Discard frames with unknown address and broadcasts. No flooding Disable Spanning Tree Protocol. Add protection path switching for each direction of the trunk Switch forwarding tables are administratively populated using management Same frame format as with MAC-in-MAC. No change.
Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
43
©2014 Raj Jain
PBB-TE QoS
Classification, Policing, Marking
Washington University in St. Louis
Scheduling, Remarking
Subscriber
Shaping
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
44
UNI-C
Service Provider
UNI-N
Subscriber
Wholesale Service Provider
ENNI
ENNI
UNI-N
Guarantees QoS No need for MPLS or SONET/SDH UNI traffic is classified by Port, Service VLAN ID, Customer VLAN ID, priority, Unicast/Multicast UNI ports are policed Excess traffic is dropped No policing at NNI ports. Only remarking, if necessary. Traffic may be marked and remarked at both UNI and NNI UNI-C
©2014 Raj Jain
Ethernet Tagged Frame Format Evolution
Original Ethernet C-DA C-SA Type Payload
IEEE 802.1Q VLAN C-DA C-SA Type C-VID Type Payload 8100
IEEE 802.1ad PB C-DA C-SA Type S-VID Type C-VID Type Payload 88A8 8100
IEEE 802.1ah PBB or 802.1Qay PBB-TE
B-DA B-SA Type B-VID Type I-SID C-DA C-SA Type S-VID Type C-VID Type Payload 88A8 88E7 88A8 8100
Tag Type Customer VLAN Service VLAN or Backbone VLAN Backbone Service Instance Washington University in St. Louis
Value 8100 88A8 88E7
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
45
©2014 Raj Jain
Comparison of Technologies Basic MPLS Ethernet No Protection Fast Reroute No Circuit Based Yes Inefficient Priority Diffserve
PB
PBB-TE
Legacy Services Traffic Engineering Scalability
No
Protection Fast Reroute VLAN Circuit Based Yes No. P2P only Diffserve+ Diffserve+ Guaranteed Guaranteed Yes (PWE3) No No
No
Yes
No
Yes
Limited
Complex
Q-in-Q
Cost OAM
Low No
High Some
Medium Yes
Q-in-Q+ Mac-in-MAC Medium Yes
Resilience Security Multicast QoS
Ref: Bonafede
Washington University in St. Louis
SPB/LAG
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
46
©2014 Raj Jain
Summary of Part III 1. 2.
3. 4.
PB Q-in-Q extension allows Internet/Cloud service providers to allow customers to have their own VLAN IDs PBB MAC-in-MAC extension allows customers/tenants to have their own MAC addresses and allows service providers to not have to worry about them in the core switches PBB allows very large Ethernet networks spanning over several backbone carriers PBB-TE extension allows connection oriented Ethernet with QoS guarantees and protection
Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
47
©2014 Raj Jain
Part IV: Virtual Bridging
1. 2. 3. 4. 5.
Virtual Bridges to connect virtual machines IEEE Virtual Edge Bridging Standard Single Root I/O Virtualization (SR-IOV) Aggregating Bridges and Links: VSS and vPC Bridges with massive number of ports: VBE
Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
48
©2014 Raj Jain
vSwitch
Problem: Multiple VMs on a server need to use one physical network interface card (pNIC) Solution: Hypervisor creates multiple vNICs connected via a virtual switch (vSwitch) pNIC is controlled by hypervisor and not by any individual VM Notation: From now on prefixes p and v refer to physical and virtual, respectively. For VMs only, we use upper case V. VM vNIC
Hypervisor
VM vNIC …
vSwitch
VM vNIC
pNIC
pSwitch
pM
Ref: G. Santana, “Datacenter Virtualization Fundamentals,” Cisco Press, 2014, ISBN: 1587143240 http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm Washington University in St. Louis
49
©2014 Raj Jain
Virtual Bridging vM1
vM2
vNIC1 vNIC2 VEB pM pNIC
vM1
vM2
vNIC1 vNIC2 pM
VEPA pNIC
vM1
vM2
pM Hypervisor vNIC1 vNIC2 pNIC
pSwitch
Where should most of the tenant isolation take place? 1. VM vendors: S/W NICs in Hypervisor w Virtual Edge Bridge (VEB)(overhead, not ext manageable, not all features) 2. Switch Vendors: Switch provides virtual channels for interVM Communications using virtual Ethernet port aggregator (VEPA): 802.1Qbg (s/w upgrade) 3. NIC Vendors: NIC provides virtual ports using Single-Route I/O virtualization (SR-IOV) on PCI bus Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
50
©2014 Raj Jain
Virtual Edge Bridge
IEEE 802.1Qbg-2012 standard for vSwitch Two modes for vSwitches to handle local VM-to-VM traffic: Virtual Edge Bridge (VEB): Switch internally. Virtual Ethernet Port Aggregator (VEPA): Switch externally VEB could be in a hypervisor or network interface card may learn or may be configured with the MAC addresses VEB may participate in spanning tree or may be configured\ Advantage: No need for the external switch in some cases VM VM
VM VEB vSwitch
VM
pSwitch
VM Washington University in St. Louis
VEPA vSwitch
pSwitch
VM http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
51
©2014 Raj Jain
Virtual Ethernet Port Aggregator (VEPA)
VEPA simply relays all traffic to an external bridge External bridge forwards the traffic. Called “Hairpin Mode.” Returns local VM traffic back to VEPA Note: Legacy bridges do not allow traffic to be sent back to the incoming port within the same VLAN VEPA Advantages: Visibility: External bridge can see VM to VM traffic. Policy Enforcement: Better. E.g., firewall Performance: Simpler vSwitch Less load on CPU Management: Easier Both VEB and VEPA can be implemented on the same NIC in the same server and can be cascaded.
Ref: HP, “Facts about the IEEE 802.1Qbg proposal,” Feb 2011, 6pp., http://h20000.www2.hp.com/bc/docs/support/SupportManual/c02877995/c02877995.pdf http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm Washington University in St. Louis
52
©2014 Raj Jain
Combining Bridges
Problem: Number of VMs is growing very fast Need switches with very large number of ports Easy to manage one bridge than 100 10-port bridges How to make very large switches ~1000 ports? Solutions: Multiple pSwitches to form a single switch 1. Fabric Extension (FEX) 2. Virtual Bridge Port Extension (VBE)
Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
53
©2014 Raj Jain
Fabric Extenders
Fabric extenders (FEX) consists of ports that are managed by a remote parent switch 12 Fabric extenders, each with 48 host ports, connected to a parent switch via 4-16 10 Gbps interfaces to a parent switch provide a virtual switch with 576 host ports Chassis Virtualization All software updates/management, forwarding/control plane is managed centrally by the parent switch. A FEX can have an active and a standby parent. vSwitch
Parent Switch vSwitch …
Fabric Extender
Fabric Extender
Fabric Extender
Ref: P. Beck, et al., “IBM and Cisco: Together for a World Class Data Center,” IBM Red Book, 2013, 654 pp., ISBN: 0-7384-3842-1, http://www.redbooks.ibm.com/redbooks/pdfs/sg248105.pdf http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm Washington University in St. Louis ©2014 Raj Jain
54
Virtual Bridge Port Extension (VBE)
IEEE 802.1BR-2012 standard for fabric extender functions Specifies how to form an extended bridge consisting of a controlling bridge and Bridge Port Extenders Extenders can be cascaded. Some extenders may be in a vSwitch in a server hypervisor. All traffic is relayed by the controlling bridge Extended bridge is a bridge. Extended Bridge
Controlling Bridge Bridge Port Extender
Bridge Port Extender Bridge Port Extender
Server
Server
Washington University in St. Louis
Server
Server
VM
VM Server
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
55
©2014 Raj Jain
Summary of Part IV 1. 2.
3.
Network virtualization includes virtualization of NICs, Bridges, Routers, and L2 networks. Virtual Edge Bridge (VEB) vSwitches switch internally while Virtual Ethernet Port Aggregator (VEPA) vSwitches switch externally. Fabric Extension and Virtual Bridge Extension (VBE) allows creating switches with a large number of ports using port extenders (which may be vSwitches)
Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
56
©2014 Raj Jain
Part V: LAN Extension and Partitioning LAN
1. 2. 3. 4.
Transparent Interconnection of Lots of Links (TRILL) Network Virtualization using GRE (NVGRE) Virtual eXtensible LANs (VXLAN) Stateless Transport Tunneling Protocol (STT)
Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
57
©2014 Raj Jain
Challenges of LAN Extension
Broadcast storms: Unknown and broadcast frames may create excessive flood Loops: Easy to form loops in a large network. STP Issues: High spanning tree diameter: More than 7. Root can become bottleneck and a single point of failure Multiple paths remain unused Tromboning: Dual attached servers Core and switches generate excessive Aggregation cross traffic Access Security: Data on LAN extension Server must be encrypted
Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
58
LAN
LAN Extension
Core Aggregation Access Server ©2014 Raj Jain
TRILL
Transparent Interconnection of Lots of Links Allows a large campus to be a single extended LAN LANs allow free mobility inside the LAN but: Inefficient paths using Spanning tree Inefficient link utilization since many links are disabled Inefficient link utilization since multipath is not allowed. Unstable: small changes in network large changes in spanning tree IP subnets are not good for mobility because IP addresses change as nodes move and break transport connections, but: IP routing is efficient, optimal, and stable Solution: Take the best of both worlds Use MAC addresses and IP routing
Ref: RFCs 5556, 6325, 6326, 6327, 6361, 6439 http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm Washington University in St. Louis
59
©2014 Raj Jain
TRILL Architecture
Routing Bridges (RBridges) encapsulate L2 frames and route them to destination RBridges which decapsulate and forward Header contains a hop-limit to avoid looping RBridges run IS-IS to compute pair-wise optimal paths for unicast and distribution trees for multicast RBridge learn MAC addresses by source learning and by exchanging their MAC tables with other RBridges Each VLAN on the link has one (and only one) designated RBridge using IS-IS election protocol RB2 H1
RB1
RB4
RB3
H2
Ref: R. Perlman, "RBridges: Transparent Routing," Infocom 2004 Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
60
©2014 Raj Jain
TRILL Encapsulation Format Outer Header TRILL header Original 802.1Q packet Version Res. 2b
2b
MultiOptions Hops Egress Ingress Options Destination Length to Live RBridge RBridge 1b 5b 6b 16b 16b
For outer headers both PPP and Ethernet headers are allowed. PPP for long haul. Outer Ethernet header can have a VLAN ID corresponding to the VLAN used for TRILL. Priority bits in outer headers are copied from inner VLAN
Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
61
©2014 Raj Jain
TRILL Features
Transparent: No change to capabilities. Broadcast, Unknown, Multicast (BUM) support. Autolearning. Zero Configuration: RBridges discover their connectivity and learn MAC addresses automatically Hosts can be multi-homed VLANs are supported Optimized route No loops Legacy bridges with spanning tree in the same extended LAN
Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
62
©2014 Raj Jain
TRILL: Summary
TRILL allows a large campus to be a single Extended LAN Packets are encapsulated and routed using IS-IS routing
Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
63
©2014 Raj Jain
GRE
Generic Routing Encaptulation (RFC 1701/1702) Generic X over Y for any X or Y Over IPv4, GRE packets use a protocol type of 47 Optional Checksum, Loose/strict Source Routing, Key Key is used to authenticate the source Recursion Control: # of additional encapsulations allowed. 0 Restricted to a single provider network end-to-end Offset: Points to the next source route field to be used IP or IPSec are commonly used as delivery headers Delivery Header GRE Header
Payload
Check- Routing Key Seq. Strict Recursion Flags Ver. Prot. Offset Check Key Seq. Source sum Present Present # Source Control # Type sum # Routing Present Present Route List 1b
1b
1b
Washington University in St. Louis
1b
1b
3b
5b
3b
16b
16b
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
64
16b 32b
32b Variable ©2014 Raj Jain
NVGRE
Network Virtualization using GRE Ethernet over GRE over IP (point-to-point) A unique 24-bit Virtual Subnet Identifier (VSID) is used as the lower 24-bits of GRE key field 224 tenants can share Unique IP multicast address is used for BUM (Broadcast, Unknown, Multicast) traffic on each VSID Equal Cost Multipath (ECMP) allowed on point-to-point tunnels Provider Edge Router
Customer Edge Switch Ethernet
Provider Core Router(s)
Provider Edge Router
Tunnel Provider Network IP
GRE
Ethernet
=
IP
GRE
Ethernet
Ref: M. Sridharan, “MVGRE: Network Virtualization using GRE,” Aug 2013, http://tools.ietf.org/html/draft-sridharan-virtualization-nvgre-03 http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm Washington University in St. Louis
65
Customer Edge Switch Ethernet
©2014 Raj Jain
NVGRE (Cont)
In a cloud, a pSwitch or a vSwitch can serve as tunnel endpoint VMs need to be in the same VSID to communicate VMs in different VSIDs can have the same MAC address Inner IEEE 802.1Q tag, if present, is removed. VM
VM
VM
VM
VM
VM
10.20.2.1
10.20.2.2
10.20.2.3
10.20.2.4
10.20.2.5
10.20.2.6
VM
VM
VM
VM
VM
VM
10.20.1.1
10.20.1.2
10.20.1.3
10.20.1.4
10.20.1.5
10.20.1.6
Subnet 192.168.1.X
Subnet 192.168.2.X
Virtual Subnet 10.20.2.X Virtual Subnet 10.20.1.X
Subnet 192.168.3.X
Internet Ref: Emulex, “NVGRE Overlay Networks: Enabling Network Scalability,” Aug 2012, 11pp., http://www.emulex.com/artifacts/074d492d-9dfa-42bd-9583-69ca9e264bd3/elx_wp_all_nvgre.pdf http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm Washington University in St. Louis
66
©2014 Raj Jain
VXLAN
Virtual eXtensible Local Area Networks (VXLAN) L3 solution to isolate multiple tenants in a data center (L2 solution is Q-in-Q and MAC-in-MAC) Developed by VMware. Supported by many companies in IETF NVO3 working group Problem: 4096 VLANs are not sufficient in a multi-tenant data center Tenants need to control their MAC, VLAN, and IP address assignments Overlapping MAC, VLAN, and IP addresses Spanning tree is inefficient with large number of switches Too many links are disabled Better throughput with IP equal cost multipath (ECMP)
Ref: M. Mahalingam, “VXLAN: A Framework for Overlaying Virtualized Layer 2 Networks over Layer 3 Networks,” draft-mahalingam-dutt-dcops-vxlan-04, May, 8, 2013, http://tools.ietf.org/html/draft-mahalingam-dutt-dcops-vxlan-04 Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
67
©2014 Raj Jain
VXLAN Architecture
Create a virtual L2 overlay (called VXLAN) over L3 networks 224 VXLAN Network Identifiers (VNIs) Only VMs in the same VXLAN can communicate vSwitches serve as VTEP (VXLAN Tunnel End Point). Encapsulate L2 frames in UDP over IP and send to the destination VTEP(s). Segments may have overlapping MAC addresses and VLANs but L2 traffic never crosses a VNI Tenant 3 Virtual L2 Network Tenant 1 Virtual L2 Network
Tenant 2 Virtual L2 Network
L3 Network Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
68
©2014 Raj Jain
VXLAN Deployment Example Example: Three tenants. 3 VNIs. 4 Tunnels for unicast. + 3 tunnels for multicast (not shown)
VM1-1 VNI 34
VM2-1 VNI 22
Hypervisor VTEP IP1
VM2-2 VNI 22
VM1-2 VNI 34
VM3-1 VNI 74
Hypervisor VTEP IP2
VM3-2 VNI 74
VM1-3 VNI 34
Hypervisor VTEP IP3
L3 Network Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
69
©2014 Raj Jain
VXLAN Encapsulation Format
Outer VLAN tag is optional. Used to isolate VXLAN traffic on the LAN Source VM ARPs to find Destination VM’s MAC address. All L2 multicasts/unknown are sent via IP multicast. Destination VM sends a standard IP unicast ARP response. Destination VTEP learns inner-Src-MAC-to-outer-src-IP mapping Avoids unknown destination flooding for returning responses Dest. Source VTEP VTEP MAC MAC
Outer VLAN
Dest. Source VTEP VTEP IP IP
Only key fields are shown Washington University in St. Louis
UDP VXLAN Header Header
Flags 8b
Dest Source Tenant Ethernet VM VM VLAN Payload MAC MAC
Res VNI Res 24b 24b 8b
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
70
©2014 Raj Jain
VXLAN Encapsulation Format (Cont)
IGMP is used to prune multicast trees 7 of 8 bits in the flag field are reserved. I flag bit is set if VNI field is valid UDP source port is a hash of the inner MAC header Allows load balancing using Equal Cost Multi Path using L3-L4 header hashing VMs are unaware that they are operating on VLAN or VXLAN VTEPs need to learn MAC address of other VTEPs and of client VMs of VNIs they are handling. A VXLAN gateway switch can forward traffic to/from nonVXLAN networks. Encapsulates or decapsulates the packets.
Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
71
©2014 Raj Jain
VXLAN: Summary
VXLAN solves the problem of multiple tenants with overlapping MAC addresses, VLANs, and IP addresses in a cloud environment. A server may have VMs belonging to different tenants No changes to VMs. Hypervisors responsible for all details. Uses UDP over IP encapsulation to isolate tenants
Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
72
©2014 Raj Jain
Stateless Transport Tunneling Protocol (STT)
Ethernet over TCP-Like over IP tunnels. GRE, IPSec tunnels can also be used if required. Tunnel endpoints may be inside the end-systems (vSwitches) Designed for large storage blocks 64kB. Fragmentation allowed. Most other overlay protocols use UDP and disallow fragmentation Maximum Transmission Unit (MTU) issues. TCP-Like: Stateless TCP Header identical to TCP (same protocol number 6) but no 3-way handshake, no connections, no windows, no retransmissions, no congestion state Stateless Transport (recognized by standard port number). Broadcast, Unknown, Multicast (BUM) handled by IP multicast tunnels
Ref: B. Davie and J. Gross, "A Stateless Transport Tunneling Protocol for Network Virtualization (STT)," Apr 2014, http://tools.ietf.org/html/draft-davie-stt-06 http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm Washington University in St. Louis ©2014 Raj Jain
73
LSO and LRO
Large Send Offload (LSO): Host hands a large chunk of data to NIC and meta data. NIC makes MSS size segments, adds checksum, TCP, IP, and MAC headers to each segment. Large Receive Offload (LRO): NICs attempt to reassemble multiple TCP segments and pass larger chunks to the host. Host does the final reassembly with fewer per packet operations. STT takes advantage of LSO and LRO features, if available. Using a protocol number other than 6 will not allow LSO/LRO to handle STT Host
Meta Data
Payload LSO
L2 Header Washington University in St. Louis
LRO
IP Header TCP Header
Segment
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
74
©2014 Raj Jain
STT Optimizations
VM VM
vNIC vNIC
Washington University in St. Louis
NIC
NIC Underlay Network
vSwitch
Large data size: Less overhead per payload byte Context ID: 64-bit tunnel end-point identifier Optimizations: 2-byte padding is added to Ethernet frames to make its size a multiple of 32-bits. Source port is a hash of the inner header ECMP with each flow taking different path and all packets of a flow taking one path No protocol type field Payload assumed to be Ethernet, which can carry any payload identified by protocol type. vSwitch
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
75
vNIC
VM
vNIC
VM ©2014 Raj Jain
STT Frame Format
16-Bit MSS 216 B = 64K Byte maximum L4 Offset: From the of STT header to the start of encapsulated L4 (TCP/UDP) header Helps locate payload quickly Checksum Verified: Checksum covers entire payload and valid Checksum Partial: Checksum only includes TCP/IP headers IP TCP-Like STT STT Header Header Header Payload
Version
8b
Flags
8b
Checksum Verified 1b
L4 Offset
Reserved
8b Checksum Partial
Maximum Segment Size
8b
16b
IP Version IPv4
1b
Washington University in St. Louis
1b
TCP Payload 1b
Priority VLAN ID Context Code Valid ID Point 3b
1b
64b
VLAN ID 12b
Padding
16b
Reserved 4b
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
76
©2014 Raj Jain
TCP-Like Header in STT
Destination Port: Standard to be requested from IANA Source Port: Selected for efficient ECMP Ack Number: STT payload sequence identifier. Same in all segments of a payload Sequence Number (32b): Length of STT Payload (16b) + offset of the current segment (16b) Correctly handled by NICs with Large Receive Offload (LRO) feature No acks. STT delivers partial payload to higher layers. Higher layer TCP can handle retransmissions if required. Middle boxes will need to be programmed to allow STT pass through Source Port (Random)
Dest. Port (Standard)
STT Payload Segment Length Offset Sequence Number*
16b
16b
16b+16b
Washington University in St. Louis
Payload Sequence # Ack Number*
32b 16b *Different meaning than TCP
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
77
Data Offset
©2014 Raj Jain
STT Summary
STT solves the problem of efficient transport of large 64 KB storage blocks Uses Ethernet over TCP-Like over IP tunnels Designed for software implementation in hypervisors
Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
78
©2014 Raj Jain
Summary of Part V
1. 2. 3. 4.
TRILL allows Ethernet to span a large campus using IS-IS encapsulation NVGRE uses Ethernet over GRE for L2 connectivity. VXLAN uses Ethernet over UDP over IP STT uses Ethernet over TCP-like stateless protocol over IP.
Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
79
©2014 Raj Jain
Overall Summary 1. 2. 3.
4.
Virtualization allows applications to use resources without worrying about its location, size, format etc. Ethernet’s use of IDs as addresses makes it very easy to move systems in the data center Keep traffic on the same Ethernet Cloud computing requires Ethernet to be extended globally and partitioned for sharing by a very large number of customers who have complete control over their address assignment and connectivity and requires rapid provisioning of a large number of virtual NICs and switches Spanning tree is wasteful of resources and slow. Ethernet now uses shortest path bridging (similar to OSPF)
Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
80
©2014 Raj Jain
Overall Summary (Cont) 5.
6. 7.
8. 9.
Data center bridging extensions reduce the packet loss by enhanced transmission selection and Priority-based flow control. Make Ethernet suitable for storage traffic. PB Q-in-Q extension allows Internet/Cloud service providers to allow customers to have their own VLAN IDs PBB MAC-in-MAC extension allows customers/tenants to have their own MAC addresses and allows service providers to not have to worry about them in the core switches PBB-TE extension allows connection oriented Ethernet with QoS guarantees and protection Virtual Edge Bridge (VEB) vSwitches switch internally while Virtual Ethernet Port Aggregator (VEPA) vSwitches switch externally.
Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
81
©2014 Raj Jain
Overall Summary (Cont) 10. 11.
12. 13. 14. 15.
SR-IOV technology allows multiple virtual NICs via PCI and avoids the need for internal vSwitch. Fabric Extension and Virtual Bridge Extension (VBE) allows creating switches with a large number of ports using port extenders (which may be vSwitches) TRILL allows Ethernet to span a large campus using IS-IS encapsulation NVGRE uses Ethernet over GRE for L2 connectivity. VXLAN uses Ethernet over UDP over IP STT uses Ethernet over TCP-like stateless protocol over IP.
Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
82
©2014 Raj Jain
Acronyms
ADC API ARP BER BUM CapEx CD CE CFI CFM CPU CRC CSMA/CD DA DCB DCBX
Application Delivery Controller Application Programming Interface Address Resolution Protocol Bit Error Rate Broadcast, Unknown, Multicast Capital Expenditure Compact Disk Customer Edge Canonical Format Indicator Connectivity Fault Management Central Processing Unit Cyclic Redundancy Check Carrier Sense Multiple Access with Collision Detection Destination Address Data Center Bridging Data Center Bridging Exchange
Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
83
©2014 Raj Jain
Acronyms (Cont)
DEI DNS DSCP DVS ECMP ENNI EPL ETS EVC EVP-Tree EVPL EVPLAN EVPN FCoE FEX GB
Drop Eligibility Indicator Domain Name Service Differentiated Services Code Points Distributed Virtual Switch Equal-cost multi-path Ethernet Network to Network Interface Ethernet Private Line Enhanced Transmission Service Ethernet Virtual Channel Ethernet Virtual Private Tree Ethernet Virtual Private Line Ethernet Virtual Private LAN Ethernet Virtual Private Network Fibre Channel over Ethernet Fabric Extension Giga Byte
Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
84
©2014 Raj Jain
Acronyms (Cont)
GMPLS GRE HSRP IANA ID IEEE IETF IGMP IO IP IPSec IPv4 IPv6 IS-IS iSCSI iSCSI
Generalized Multi-Protocol Label Switching Generic Routing Encapsulation Hot Standby Router Protocol Internet Addressing and Naming Authority Identifier Institution of Electrical and Electronic Engineers Internet Engineering Task Force Internet Group Multicast Protocol Input/Output Internet Protocol Secure IP Internet Protocol Version 4 Internet Protocol Version 6 Intermediate System to Intermediate System Internet Small Computer Storage Interconnect Internet Small Computer Storage Interconnect
Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
85
©2014 Raj Jain
Acronyms (Cont)
kB LACP LAN LISP LLDP LRO LSO MAC MDI MPLS MR-IOV MSB MSS MST MSTP MTU
Kilo Byte Link Aggregation Control Protocol Local Area Network Locator-ID Split Protocol Link Layer Discovery Protocol Large Receive Offload Large Send Offload Media Access Control Media Dependent Interface Multi-Protocol Label Switching Multi-Root I/O Virtualization Most Significant Byte Maximum Segment Size Multiple spanning tree Multiple Spanning Tree Protocol Maximum Transmission Unit
Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
86
©2014 Raj Jain
Acronyms (Cont)
MVGRE NIC NNI NVO3 OAM OpEx OSPF OTV PB PBB-TE PBB PBEB PCI-SIG PCI PCIe PCP
Network Virtualization Using GRE Network Interface Card Network-to-Network Interface Network Virtualization Overlay using L3 Operation, Administration, and Management Operation Expenses Open Shortest Path First Overlay Transport Virtualization Provider Bridge Provider Backbone Bridge with Traffic Engineering Provider Backbone Bridge Provider Backbone Edge Bridge PCI Special Interest Group Peripheral Component Interconnect PCI Express Priority Code Point
Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
87
©2014 Raj Jain
Acronyms (Cont)
PE PF PFC PHY pM pNIC PPP pSwitch PW PWoGRE PWoMPLS QCN QoS RAID RBridge RFC
Provider Edge Physical Function Priority-based Flow Control Physical Layer Physical Machine Physical Network Interface Card Point-to-Point Protocol Physical Switch Pseudo wire Pseudo wire over Generic Routing Encapsulation Pseudo wire over Multi Protocol Label Switching Quantized Congestion Notification Quality of Service Redundant Array of Independent Disks Routing Bridge Request for Comments
Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
88
©2014 Raj Jain
Acronyms (Cont)
RSTP SA SDH SID SNIA SONET SPB SR-IOV STP STT TCP TE TLV TP TPI TRILL
Rapid Spanning Tree Protocol Source Address Synchronous Digital Hierarchy Service Identifier Storage Network Industry Association Synchronous Optical Network Shortest Path Bridging Single Root I/O Virtualization Spanning Tree Protocol Stateless Transport Tunneling Protocol Transmission Control Protocol Traffic Engineering Type-Length-Value Transport Protocol Tag Protocol Identifier Transparent Interconnection of Lots of Links
Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
89
©2014 Raj Jain
Acronyms (Cont)
TV UCA UDP UNI VBE VDC VEB VEM VEPA VF VID VLAN VM VNI vNIC VoD
Television Use Customer Address User Datagram Protocol User Network Interface Virtual Bridge Port Extension Virtual Device Contexts Virtual Edge Bridge Virtual Ethernet Module Virtual Ethernet Port Aggregator Virtual Function VLAN ID Virtual LAN Virtual Machine Virtual Network ID Virtual Network Interface Card Video on Demand
Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
90
©2014 Raj Jain
Acronyms (Cont)
VOIP vPC VPLS VPN VRF VRRP VSID VSM VSS vSwitch VTEP VXLAN
Voice over IP Virtual Port Channels Virtual Private LAN Service Virtual Private Network Virtual Routing and Forwarding Virtual Router Redundancy Protocol Virtual Subnet Identifier Virtual Switch Module Virtual Switch System Virtual Switch Virtual Tunnel End Point Virtual Extensible LAN
Washington University in St. Louis
http://www.cse.wustl.edu/~jain/tutorials/nv_hs14.htm
91
©2014 Raj Jain
