Assessing IP/MPLS and MPLS-TP for Transport Networks Document type: Author: Date: Document No.:

White Paper Johan de Bilde April, 2013 XA-W059-E-1

1

Abstract When selecting technology for next generation Transport Networks, “MPLS” is often referred to as the solution. But which “MPLS” is better suited for this application? Will IP/MPLS do the job, and is it fit for transport? Or is MPLS-TP (Transport Profile) a better fit for transport networks, as its name suggests? What are the commonalities of both? Where do they differ? How do they cooperate? This paper is not a tutorial on IP/MPLS and MPLS-TP, and it is assumed that the reader has some familiarity with the basic technology. This paper intends to describe and position both IP/MPLS and MPLS-TP for their specific use in Transport Networks for Railway Operators and Utility Providers, and acts as a guide to help make sound decisions. In the paragraphs below, it becomes clear that when IP/MPLS is the best choice for use in Core Networks, it is not the best option for Transport Networks. Here, MPLS-TP is clearly the better choice: • MPLS-TP adds mandatory OAM functionalities required in Transport Networks, these functionalities are not available in IP/MPLS. • Superfluous functionalities from IP/MPLS that use up a lot of CPU power, are not available in MPLS-TP, yielding a far better performance. • An MPLS-TP network can be run by operators having SDH experience. There is no need to hire expensive Layer 2 (Ethernet) or Layer 3 (IP routing) specialists. • The deterministic behavior of the data paths in MPLS-TP, with symmetrical and congruent data paths in both directions, is essential for use in Transport Networks, and this is quite difficult to achieve with IP/MPLS. • In short, MPLS-TP allows to run Transport Networks easier, with higher quality data streams, requiring less operational personnel that needs less training and hence provides an overall far more cost effective solution. In the text below, color markings are used to indicate the functional fit to Transport Networks: Good fit, standard available, easy to do Can be made to fit, but requires considerable effort Does not fit, cannot be done or very difficult

Page 2 of 15

White Paper • XA-W059-E-1 © OTN Systems NV • All rights reserved

2

Transport Networks Characteristics Telecommunication Networks of Railway Operators or Utility Providers, have a similar architecture to that of Telecom Operators: they consist of a Core Network, a Metro or Transport Network, and an Access Network.

Network Architecture

The Core or Backbone network, is the central part of a telecommunication network. It’s a high capacity network that provides communication paths between applications and different subnetworks. Core networks usually have a mesh topology and provide any-to-any connectivity. Devices in the Core are typically switches and routers, 10s to 100s of them. These Core devices must be able to handle very high volumes of data at very high speeds from any source to any destination. In the early days, all traffic was routed, but IP routing became a limiting factor as networks grew and speeds increased. Routing was mainly done in software, more processing power was needed to process more IP packets faster, and IP Routing hit its speed limitations. Furthermore, IP routing is by definition connection-less, and has unpredictable behavior in large networks with high data volumes. That is why IP/MPLS (Internet Protocol / Multiprotocol Label Switching) was introduced in addition to IP Routing: Incoming packets are classified, labeled and forwarded instead of routed. This forwarding can be performed in hardware and is hence much faster than routing. The introduction of IP/MPLS in the network made the network connection-oriented and allowed Traffic Engineering (TE). Today, IP/MPLS is a widely used technology in Core Networks. The Metro / Transport part of the network is quite different, and has some very specific requirements. It has a large geographical footprint with a high number of nodes, typically 100s to 1000s of them. Network topologies found in a Transport Network include mesh, ring, star, tree and daisy-chain, or any combination of these. Data flows in a Transport Network are quite different from those in the core network. Where Core Networks require flexible any-to-any

XA-W059-E-1 © OTN Systems NV • All rights reserved

Page 3 of 15

connectivity, traffic flows in Transport networks are much simpler and more static, have long times to live (months-years), and must be predictable and very reliable. Railway Operators and Utility Companies deploy various applications that impose specific requirements. Connecting CCTVs, Access Control Systems, Ticket Vending Machines, Management Systems, SCADA, (Railway) Signaling Systems, (Energy) Remote Telemetry Equipment, Backhauling of Mobile Traffic, Voice Servers, Connecting Safety and Security Installations, Safe Corporate Intranet Access, Public Information Systems, etc… demand a Transport Network with very specific characteristics. For Railways, Signaling Systems are critical to keep the trains running safely. For Utilities, the time critical applications are Teleprotection and Differential Delay. The Railways and Utilities equipment must have a lifespan between 10-20 years and require overall stability, absolute latency control and ease-of-use. Solutions must provide SDH-like service and operability, something these operators are well used to. Transport Networks have the following characteristics: • Predictable bandwidth • Symmetrical and congruent paths for both directions • Predictable and symmetrical delay and jitter • Statically provisioned paths • Long-lived connections • Emphasis on manageability and deterministic behavior • Fast restoration in case of failure (less than 50ms) • Powerful Operations, Administration and Maintenance (OAM), in-band OAM End-to-End QoS • Easy to manage by the Transport operations department • Security In the following paragraphs, we will describe the main functionalities of IP/MPLS and MPLS-TP, and map these to the requirements of Transport Networks.

3

IP/MPLS and MPLS-TP Backgrounder As indicated earlier, IP/MPLS was developed to improve performance of IP routed networks, and to introduce Connection Oriented communications. It operates on 3 planes: • Data plane: this is the plane that actually carries the forwarded payload. In this plane, the actual labeled packets are forwarded to their destination. It should be noted that IP packets can also be routed in the MPLS Data plane. The Label Switched Paths (LSP) and Pseudowires (PW) are situated in this Data plane.

Page 4 of 15

White Paper • XA-W059-E-1 © OTN Systems NV • All rights reserved

• Control plane: this is the plane that is responsible for collecting and propagating the information that will be used later to forward incoming packets. Routing Protocols (IGP, BGP) and Label Distribution Protocol (LDP) are parts of the Control plane. The Forwarding Equivalent Classes (FEC) that identify the various LSPs are managed here. The control plane decides which packets belong to which FEC. • Management plane: this is the plane used to manage the network, and includes amongst others Fault and Performance management, Configuration management and Administration. It consists of management functions in the Network Elements (counters, timers, OAM channels) and an external management system to present this information to the operator of the network.

MPLS Planes

For IP/MPLS, the Data Plane and the Control Plane are the most important planes for operating the network. For MPLS-TP, the Data Plane and the Management Plane are the essential planes. In the next paragraphs we will highlight the differences per plane, and the relevance for use in Transport Networks.

XA-W059-E-1 © OTN Systems NV • All rights reserved

Page 5 of 15

3.1

Data Plane IP/MPLS allows both the forwarding (or routing) of IP Packets and the forwarding of Labeled MPLS Packets. As indicated above, IP routing is neither predictable nor symmetrical, and has no predictable delay and jitter, so it is not suited for Transport Networks and hence should not be used in Transport Networks. That is why MPLS-TP only uses the forwarding of labeled MPLS packets.

3.1.1

Label Swith Paths and Pseudowires IP/MPLS uses Label Switch Paths (LSP) and/or Pseudowires end-to-end (PWE3) to transport data through the network. IP traffic is forwarded directly into a LSP, and for transporting legacy data (E1/T1, ATM, FR, Ethernet VLANs, …), pseudowires end-to-end PWE3 are used. In IP/MPLS, paths are uni-directional, and typically have different routes in the forward and reverse directions. This is an important issue, as traffic in Transport networks should flow on congruent paths in both direction. Setting-up congruent paths in IP/MPLS can be done, but must be done by hand, and is a tedious job. The transport of data in MPLS-TP networks is slightly different – but 100% compatible with IP/MPLS. In MPLS-TP, LSPs are bidirectional by definition, so traffic flows in both directions are congruent. Furthermore, all traffic is transported in pseudowires (PWE3), and the PWs themselves run in a LSP. The reason to use PW for all traffic, is that this way user traffic remains unchanged: original source/destination MAC addresses remain intact all the way. In fact, to the user it seems as if he has his own dedicated Leased Line for his service. This is exactly what a Transport Network should do.

Pseudowires in a LSP

3.1.2

Operations, Administration and Maintenance (OAM) aspects OAM in general refers to the framework of processes, activities, tools and standards used for operating, administering, managing and maintaining a telecom network. In order to guarantee the quality of the service, Fault Management (FM) and Performance Monitoring (PM) are key components of any OAM solution. For Ethernet based networks, a number of standards are relevant in this respect:

Page 6 of 15

White Paper • XA-W059-E-1 © OTN Systems NV • All rights reserved

ITU-T Y.1731: defines OAM functions and mechanisms for Ethernet based networks. For Fault Management, the standard defines continuity checks, loopbacks, link trace and alarm suppression for effective fault detection, verification, isolation and notification. For Performance Monitoring, it defines measurements such as Frame Loss Ratio, Frame Delay and Frame Delay variation. IEEE 802.1ag: Connectivity Fault Management (CFM), largely identical to ITU-T Y.1731 but without Performance Monitoring. IEEE 802.3ah: defines Link Level Monitoring in Ethernet First Mile (EFM) for Service Assurance Detection on the native Ethernet service. For IP/MPLS, specific OAM functions are: LSP Ping/Traceroute: Like the traditional IP Ping, LSP Ping is based on echo request and echo reply on either side of the LSP. The LERs of a LSP issue Ping messages, and expect a reply. When the LSP is broken, no echo is generated and an error condition is flagged. LSP Traceroute can then be used to find the location of the failure. VCCV: Pseudowire Virtual Circuit Connectivity Verification allows to use LSP Ping to test the Pseudowire (PW) section of the connection. BFD: Bidirectional Forwarding Detection is, similar to LSP Ping, a mechanism for detecting MPLS data plane failures, but unlike LSP Ping, is not able to verify LSP data plane against the control plane (*). Processing of BFD control packets is relatively faster than the processing of LSP Ping messages. BFD allows faster data plane fault detection and makes it possible to provide this detection on a greater number of LSPs. (*) Note: this verification of the LSP data plane against the control plane is not required in MPLS-TP, since the data plane is separated from the control plane. So the better performance of BFD makes it the preferred OAM function for MPLS-TP. From a Transport Network point of view, LSP Ping/Traceroute and VCCV have a few drawbacks. First, they are on-demand processes, so it is not a continuous dataflow that is monitored as we would expect for Transport Networks. Second, the specific OAM packets do not follow the actual data stream, and hence can have a different timing than the actual data. Furthermore, IP/MPLS does not support Performance Management, and lacks some Fault Management functions (eg. Loopback, lock mechanism). As a result, IP/MPLS OAM functionalities are not sufficient to guarantee fast detection and recovery of less than 50 ms, and hence are not suited for Transport Networks. To overcome these problems, MPLS-TP supports additional OAM mechanisms. The most relevant addition is the in-band OAM capability: the OAM packets (used by BFD or Y.1731) follow the same path as the data flows, in a dedicated channel, called Generic Associated Channel (G-Ach). G-Ach is used in both Pseudowires (PW) and Label Switch Paths (LSP). In the case of LSP, a special label called Generic Alert Label (GAL) is used to flag the G-Ach within the LSP. In the case of PW, this Generic Alert Label is not used. The Y.1731 OAM packets are transported transparently through the PW, so that we can assure end-to-end OAM.

XA-W059-E-1 © OTN Systems NV • All rights reserved

Page 7 of 15

+ Generic Associated Channel G-ACh in PW and LSP

Thus the complete OAM concept within MPLS-TP uses IEEE Y.1731 (for Ethernet Service and Connectivity OAM), BFD (for LSP connectivity OAM) and IEEE 802.3ah (for native Ethernet in Ethernet First Mile). The picture below gives an overview:

Overview OAM in MPLS-TP

Furthermore, OAM add-ons in MPLS-TP include Performance Monitoring, and additional functionalities for Fault Management. As a result, this OAM concept provides Transport Networks deploying MPLS-TP with all the tools required for Fault Management and Performance Management so as to guarantee data throughput, and flag errors quickly and accurately so that path protection can be activated within less than 50ms when needed. 3.1.3

Path Protection In order to guarantee continuous transport in case of failure, path protection is an essential topic. In IP/MPLS, Fast Reroute (FRR) can be used. This mechanism is also called MPLS Local Restoration or MPLS Local Protection, as this is exactly what is says: it is a local restoration

Page 8 of 15

White Paper • XA-W059-E-1 © OTN Systems NV • All rights reserved

network resiliency mechanism. In case of failure, a local Detour Path is activated in less than 50ms. A Point of Local Repair (PLR) redirects the traffic, which merges again to the primary LSP in the Merge Point (MP). Each link or node can be a PLR. In a one-to-one backup approach, the PLRs maintain separate backup paths for each LSP passing through. Maintaining state information for backup paths protecting individual LSPs is a significant resource burden for the PLR. Moreover, periodic refresh messages sent by the PLR, in order to maintain each backup path, may become a network bottleneck. In an attempt to lower this burden, a many-to-one approach is possible, with fewer states to be maintained and refreshed, but still represents a considerable burden for the network. When a network consists of 100s or 1000s or nodes, managing FRR becomes a real challenge. Another method for path protection in IP/MPLS is Global Path Protection. Here the ingress LER sets up a full primary and backup path from ingress to egress. Unfortunately, this comes at a high cost in terms of time: detecting the fault and subsequent reverse notification of the fault towards the ingress LER uses Layer 3 protocols and takes quite some time, far more than the required sub 50ms. During this time, significant packet loss will occur, which is unacceptable in Transport networks. Therefore a two-step approach is often used in IP/MPLS: Primary LSP -> Detour Path (FRR) -> Secondary Path. This causes several switchovers with different delays and jitters that are not really appreciated in Transport Networks. And the signaling is still a considerable burden for the network. Plus it adds complexity.

IP/MPLS Local Protection path (FRR) and Global Protection paths

For path protection in MPLS-TP, the FRR scheme is not used. Instead, Global Path Protection is used, but with a procedure that is compliant with the sub 50ms switchover time. The OAM mechanisms described above signal the LER (typically in less than 10 ms) that there is a problem, and the LER immediately switches to the pre-provisioned backup path. The entire procedure is guaranteed to occur in less than 50ms, and hence fulfills the Transport Network requirements.

XA-W059-E-1 © OTN Systems NV • All rights reserved

Page 9 of 15

MPLS-TP Path Protection

The elimination of FRR and other dynamic functions that prevent end-to-end monitoring (PHP Penultimate Hop Popping, and LSP merge) in MPLS-TP, and the statically provisioned Global Path Protection mechanism provide an excellent protection mechanism, comparable to the SDH protection. 3.2

Control Plane

3.2.1

Control Plane or Management Plane? The Control Plane of IP/MPLS is dynamic by nature: incoming packets are analyzed, and assigned to a LSP by adding a label. The setup of the LSPs is also a dynamic process based on Forwarding Equivalent Classes (FEC): packets with similar characteristics are forwarded in the same way. The LSPs are by definition uni-directional, so the return path is usually different from the forward path. These dynamic, unpredictable and asymmetrical elements pose serious problems in Transport networks, as we want paths to be predictable and symmetrical. Furthermore, the processing of the incoming packets requires considerable processing power in the Network Elements. Also, the signaling of the routing protocols (IGP, BGP) and the Label Distribution Protocol (LDP) again require considerable processing power… Not to mention the extensive manpower with very high skillset in layer 2 and layer 3 protocols to manage this complex network … and all this for a functionality and complexity that is absolutely unnecessary in Transport Networks. In MPLS-TP, the functions of the Control Plane are entirely within the Management System. The Network Elements (NE) are not used for the inspection of incoming packets, setup of LSPs or running routing or label distribution protocols. They do one thing, and one thing only, and they do this extremely efficiently: forwarding Labeled Packets. Primary Paths and Backup paths are statically provisioned by the Management system, end-to-end. The NMS keeps track of available network elements, available links, used and available bandwidth of the entire network, hence it can select the best Primary and Backup Paths with the required parameters.

3.2.2

Traffic Engineering (TE) Traffic Engineering is the process that directs the traffic to where bandwidth capacity is available. It is a method of optimizing the performance of a telecommunication network by analyzing, predicting and regulating the behavior of data transmitted over the network. It reduces the overall cost of operations by using bandwidth resources more efficiently, by

Page 10 of 15

White Paper • XA-W059-E-1 © OTN Systems NV • All rights reserved

preventing a situation in which some parts of the network are over-utilized, while other parts are under-utilized. IP/MPLS allows to perform Traffic Engineering by using some extensions defined as MPLS-TE, but the mechanism is based on Layer 3 (IP Routing) extensions and requires quite some effort and a high skill level to do this right. The key TE mechanisms in IP/MPLS are Explicit Routing (using Resource Reservation Protocol (RSVP) as the mechanism for establishing LSPs), using constrained-based Path Selection Algorithm and extensions to OSPF/IS-IS for flooding resources. Not an easy task, and not a task that can be performed by the Transport Operations department. Instead, MPLS-TP is designed to make Traffic Engineering inherently available, not in the Control Plane, but in the Management Plane. The paragraph below describes the role of the Management Plane, and how the database with the entire network configuration allows to make the best choices for paths with specific requirements, especially the possibility to use the under-utilized links. And all this with just some point and click actions. That is Traffic Engineering at its best. 3.3

Management Plane The importance of the Management Plane is different for IP/MPLS and for MPLS-TP. For IP/MPLS, it is a supplement that helps manage the network in a more user friendly way. Provisioning and configuration of an IP/MPLS network is done using Command Line Interface (CLI), and requires a very high skill level from the operators. They need to thoroughly understand the ins and outs of IP/MPLS, the influences of the various commands and parameters, the topology of the network etc… In many cases, a Network Management System (NMS) is added to assist with these tasks. A graphical user interface helps with the editing of the CLI commands. Scripts can be setup to automate some of the tasks. Alarms are collected and displayed. But still, this NMS is only an aid to assist the operators in the management of their network. The expertise is still with the operators, who, as said before, need a very high skill level in Layer 2 and Layer 3. With MPLS-TP, there is no Control Plane in the Network Elements, and the Management Plane is key to controlling the network. It allows the operators to set up primary and backup paths with the required parameters, much as they were used to doing in legacy transport networks such as SDH/SONET. The Management Plane has a complete view of the entire topology of the network, all existing primary and backup paths, all available and used capacities, and is able to display this in a very graphical way. To create new services the operator need only define from where to where, what kind of service and with which parameters, the Management Plane will then determine the best Primary and Backup path, and provision the network elements. From that point on, the service is created and the network elements run independently. When an error occurs, the LERs automatically switch to the backup path without interference from the Management Plane. This operation method has many benefits in Transport Networks: as only static configurations are needed, with predefined parameters, the end-to-end provisioning by means of a graphical Management Plane is the fastest and most economical way to proceed. Operators need only to be aware of the requirements of the service, the actual provisioning is taken care of by the

XA-W059-E-1 © OTN Systems NV • All rights reserved

Page 11 of 15

management system. So fewer people are required, with lower skills, reducing OPEX quite substantially. Additionally, as the Management Plane has a complete overview of available links and bandwidths, it can select the optimum path, and use resources that would typically not be favored by a dynamic protocol (dynamic protocols typically under-utilize long paths, thus wasting resources).

4

Security Security is a top consideration for all enterprises and telecom operators. Especially in the public domain, e.g. Railway Operators, Utility Providers, Traffic Control etc… safety is a key issue and needs close attention. Attacks through unauthorized access can cause destruction and corruption of data, removal of access, disclosure of information or interruption of service. Both IP/MPLS and MPLS-TP share the same protection mechanism that provides protection against attacks on the Data Plane (Access Control ACLs and authentication, guaranteed end-toend delivery, encapsulation in pseudowires, no denial of service through traffic shaping and traffic engineering), attacks on the Control Plane (abundance of control protocols, not possible to fake signaling or reservation protocols) and attacks on the Management Plane (SNMPv3 management to NE, IPSec on DCN channel, Radius authentication, logging, reporting, audit trail). These security aspects for MPLS and GMPLS networks are described in RFC 5920. The security level of IP/MPLS and MPLS-TP is the same as for SDH networks, and this is more than adequate for use in Transport Networks.

5

Operational Considerations Cost reduction is a constant endeavor for all industries. That is why most of them are consolidating their entire telecom infrastructure onto one packet based network – often referred to as the Next Generation Network (NGN). For this NGN network, MPLS is considered to be the leading connection-oriented packet transport technology. As a consequence, many have already deployed their core networks using IP/MPLS. Given this deployment of MPLS and the desire to align packet networking with more traditional transport operations methods, MPLS-TP is being standardized as a simplified version of MPLS for Transport Networks. It allows companies to run the Transport Network, which typically has an order of magnitude larger than their Core Networks, with fewer people that do not need extensive L2 and L3 skills. Operational savings of MPLS-TP compared to IP/MPLS can be summarized as follows:

OPEX Saving Category Enhanced OAM

Lower Power Consumption

Page 12 of 15

Saving -

Much easier troubleshooting than IP/MPLS Simplified network provisioning

5%

-

No control plane in NE requires less computing power, resulting in Lower Power for same data volume

51%

White Paper • XA-W059-E-1 © OTN Systems NV • All rights reserved

-

No control plane in NE allows higher switching capacity per RU Smaller footprint brings real-estate advantage

68%

-

No need for detailed L2 and L3 trainings Training manual for transport operator 10x smaller than IP/MPLS based manual

80%

-

Transport Operator profile is much cheaper than L2/L3 expert profile

50%

-

As the network grows, the complexity of IP/MPLS networks grows exponentially, requiring a similar growth in operational workforce

70%

Smaller footprint

Lower training cost

Lower hourly labor cost

Smaller workforce

Source: “MPLS-TP based packet Transport Networks” by UTStarcom, inc. USA Source:”A Financial Analysis of the Operational Benefits of MPLS-TP Transport Networks” by Network Strategy Partners Source: OTN Systems 20 years of experience

In both the Utility and the Railways world, the average age of the workforce is high. Many of these companies face a large outflow of competencies in the next decade. In contrast to that, the increase in dataflows and applications in their network means that they will have to do much more with less resources, and that any simplification is a welcome asset. MPLS-TP offers these companies precisely this advantage, as its simplicity will allow their current workforce to be able to not only manage, but also design and maintain these networks. Hiring highly skilled IP personnel is a recruiting challenge that can be avoided in this way.

6

Vendors Support for MPLS-TP MPLS-TP is defined jointly by IETF and ITU, specifically because it was widely recognized that IP/MPLS has some major problems when used in Transport Networks. As a result, MPLS-TP equipment is available from most of the world leading switch/router vendors: CISCO, Juniper, Huawei, Ericsson, ZTE, OTN Systems, IXIA, Hitachi, Metaswitch … The European Advanced Networking Test Center (EANTC) has conducted successful interop tests on MPLS-TP with a large number of vendors, testing both MPLS-TP interworking between different vendors, and also IP/MPLS – MPLS-TP interworking.

XA-W059-E-1 © OTN Systems NV • All rights reserved

Page 13 of 15

7

Summarizing Table Transport Network Requirement

IP/MPLS

MPLS/TP

Predictable bandwidth

MPLS-TE (Traffic Engineering) can be used. Done in Layer 3 as enhancement to standard IGPs such as IS-IS or OSPF.

Designed to provide predictable bandwidth using the Network Management System

Symmetrical Congruent paths for both directions

By definition, all paths are uni-directional, and quite some effort is required to define congruent paths

By definition, all paths are bi-directional and congruent

Predictable and symmetrical delay and jitter

When paths are defined as congruent paths, then their delay and jitter is symmetrical, but requires quite some effort

Congruent paths mean symmetrical delay and jitter

Statically provisioned paths

IP/MPLS allows for dynamically and statically provisioned paths

All paths are statically provisioned by use of a management system

Long lived connections

Statically provisioned paths are possible, but internally many refresh messages to keep these connections alive

All paths are by definition long lived

Emphasis on manageability and deterministic behavior

Emphasis is on any-to-any connections, and a dynamic control plane

This is exactly why MPLSTP was designed

Fast restoration time (