ETHERNET SWITCHES REQUIREMENT OVER IEC 61850 NETWORKS Lucas B Oliviera, GE Grid Solutions Brazil, [email protected] Marcelo Zapella, GE Grid Solution Brazil, [email protected] Andre Sarda, GE Grid Solutions Brazil [email protected] Wilian Zanatta, GE Grid Solutions Brazil, [email protected] Arshad GANI, GE Grid Solutions Singapore, [email protected]

Abstract Network switches for digital substations are normally offered with switching capacity (e.g., 48 Gbps) and interface options as Fast (100 Mbps) or Gigabit (1000 Mbps) Ethernet ports, but it is still obscure how to relate such features with the performance requirements for power utility applications. Many management flow control functions are also offered, such as QoS, CoS, VLAN and so on, but how these functions should be configured to ensure the data flow in situations with high traffic flow in the network. Using real case scenarios, this paper presents the switch performance considering the traffic of GOOSE messages, Sampled Values and PTP over an IEC 61850 network. As conclusion, it is presented how switches must be specified for each application to ensure digital communication performance even in worst case situations. Keywords: Switches performance, IEC 61850, VLAN, QoS, IEC 61850-90-4.

Introduction The substation automation evolution in power systems has been coming in technological waves. Initially the protection systems were composed by electromechanical relays with a protection function each, requiring many devices and much hardwiring to interconnect all of them. In the years passed, the electromechanical relays were replaced for digital relays, which may perform many protection functions reducing the number of devices in a substation. Recently, the hard-wiring used to interconnect the devices has begun to be replaced by Ethernet network, reducing time and costs in installations. Besides, the standard IEC 61850 is pushing substations toward interoperability with intensive use of high-speed digital communication among intelligent electronic devices (IEDs). In this context, the network switches have become one of the most important devices in a power system. Depending on each application, special functions may be required to ensure the switches will perform appropriate marking and treatment of substation automation traffic across the network. Defined in IEEE 802.1Q [2], functions as VLANs and Quality of Service (QoS) are designed to implement flow control, traffic shaping, or queuing differential services in the network. Class of Service (CoS) is used in support of QoS, and it assigns priority values to data messages in order to discipline the network traffic. In protection applications, GOOSE messages used for tripping and inter-tripping has high priority in network traffic and its performance shall be as specified by IEC 61850. Other protocols considered as high priority are the Sampled Value (SV) protocol, which has a high bandwidth, and the Precision Time Protocol (PTP) protocol for time synchronization among the devices in a network. The technical report IEC 61850-90-4 [1] describes the network engineering guideline for power utility automation, and it presents the performance expected for each message type over IEC 61850 networks. Management functions to control traffic flow will be assigned to each message in order to evaluate cases with high traffic flow. As Sampled Values have a high bandwidth, the quantity of Merging Units with IEC 61850-9-2 LE process bus connected to a single switch is also evaluated in this paper, showing the traffic bandwidth each process bus cost to the network. In the case of GOOSE and PTP messages, as these have low bandwidth, the results show how much delay the high traffic flow implies in the messages transmission.

High priority messages over IEC 61850 networks The IEC 61850 protocol is divided into hard real time stack and soft real-time stack. The hard real-time is composed by the services of Sampled Values, GOOSE and Precision Time Protocol, as shown in Figure 1. These protocols rely on the services of the MAC layer, which may support 802.1Q VLANs and priorities, redundancy and even security. Therefore, this paper considers these three protocols (GOOSE, Sampled Values and PTP) as high priority messages over a local IEC 61850 network. The reason MMS protocol is not consider is because it operates at the network layer 3, supporting the soft real time stack.

1

GOOSE messages GOOSE traffic, defined in IEC61850-8-1, provides event data over the entire substation taking advantage of the multicast functionality provided by Ethernet, with a fast transmission mechanism followed by a slow cyclic transmission. In digital substations, GOOSE messages shall be high-availability as it has an important role sending trip commands to protection relays.

Figure 1: Hard real-time stack from IEC 61850 protocol. Sampled Valued (SV) The Sampled Values protocol (specified in IEC61850-9-2) is mainly used to transmit analogue values (current and voltage) from the sensors to IEDs. Unlike GOOSE, SVs are transmitted purely cyclic at 80 samples per cycle for protection applications or at 256 samples per cycle for measurement applications, resulting in high bandwidth consumption. IEEE 1588 – Precision Time Protocol (PTP) The IEEE 1588 - PTP protocol offers a solution for clock synchronization in distributed systems with high accuracy requirements. As important data communication in digital substation, such as GOOSE and SV, PTP also works through Ethernet networks in order to achieve accuracy in the sub-microsecond range.

Performance Requirements for Power Utility The technical report IEC 61850-90-4 focus on the engineering of a local network limited to the IEC61850 requirements for substation automation. It describes the aspects related to protection, as tripping over the network (GOOSE), the multicast data transfer from merging units with large volumes of Sampled Values and it also considers high precision clock synchronization. Among other matters, the technical report outlines different approaches to network topology and redundancy, but this is not covered in this paper. Referring to performance, the technical report shows the expected IEC 61850 traffic and maximum delay of high priority messages, which is shown in table 1.

2

Message

Max. Delay 3 ms

Bandwidth

Application

GOOSE Low Protection (Trip) Sampled Val. 4 ms High Process Bus PTP Low General Phasors, SVs Table 1: IEC 61850 performance for high priority messages. As the PTP protocol calculates its network delay, there is not a maximum delay specified in standard for it. Besides, the PTP network usually has more than one path to deliver the packets, and each path has the delay calculated periodically to check which one is the best one. Process bus performance GOOSE transmits messages cyclically and retransmits spontaneous messages, usually two times, to overcome possible frame losses. One GOOSE application in an IED generates about 1 kbit/s in steady-state and about 1 Mbit/s during bursts. The PTP message burst depends on the network topology, considering the number of IEDs, the Ethernet Switch type (if it supports PTP or not) and mode PTP works, peer-to-peer (P2P) or end-to-end (E2E). Anyhow, PTP burst is comparable to GOOSE messages. While GOOSE and PTP have similar requirements on the process bus network, SV traffic is even more demanding, but it is easily predictable. An SV frame as specified in IEC 61850-9-2LE with a sampling rate of 80 samples per cycle, for protection applications, transmitted at 50/60 Hz grid have an approximate size of 140 bytes, consuming a bandwidth of approximately 5 Mbit/s (50 Hz) or 6 Mbit/s (60 Hz) per source IED. For measurement, the sampling rate is 256 samples per cycle, but 8 points are grouped and sent in a single packet in a lower rate, resulting in a bandwidth of up to 10Mbit/s for systems at 50 Hz and 12 Mbit/s for 60 Hz.

Managed Ethernet Switches In the substation environment, the network infrastructure must operate without the presence of operators and does not have connection to the Internet for remote management. The switches operating parameters must be configured to ensure the flow of information even in situations stress. Furthermore, the internal counters from these switches provide information for analysis and design of the network, showing the communication bottlenecks that need fine tuning. The managed switches allow you to configure its behaviour in various situations of operation, using functions as VLANs and Quality of Service (QoS), which are defined in IEEE 802.1Q. Virtual LANs (VLANs) Virtual LAN technology allows separation of traffic through logical networks. In power system communication, where IEC61850 messages are expected with different priority and usage, there will be only one physical path for each IED and the packets must be separated logically. GOOSE, Sampled Values and Precision Time Protocol messages are multicast, and all of them can be mapped directly at 802.1Q Ethernet frame. In other words, they use the layer 2 protocol. Traffic segregation through Virtual LAN (VLAN) is standardized by IEEE 802.1Q document. The standard added 4bytes in the Ethernet frame, where information about the logical LAN which the host (or message) belongs to. Figure 2 shows an Ethernet frame and the 802.1Q tag position.

Figure 2: 802.1Q Ethernet frame. The information at 802.1Q tag is divided in 4 fields:

3

1) 2) 3) 4)

TPID (Tag Protocol Identifier): 16-bit length, this field presents VLAN protocol and will be equal to 0x8100; PCP (Priority Code Point): 3-bit length, this field presents the priority of the packet at the network; CFI (Canonical Format Unit): 1-bit length, this field is always set to 0 in Ethernet communication; VID (VLAN Identifier): 12-bit length, this field shows explicitly VLAN number identifier which the frame belongs to. It is also called VLAN ID.

When using VLAN traffic segregation, multicast messages are forwarded only onto the VLAN that the multicast message belongs to. Thus, GOOSE, Sampled Values and PTP traffic will flow separately from each other. Finally, as the traffic is separated, IED equipment that expects to receive only GOOSE messages will not have its network interface interrupted by Sampled Values data, for example. An example of expected VLAN traffic segregation is shown in figure 3. Note that a ring physical topology is used only for example propose, as it is a common physical network topology in power system communication. In this example, there are 3 VLAN configured, been one for PTP traffic, one for Sampled Values traffic and another one for GOOSE messages traffic. Merging Unit is the GOOSE and Sampled Values messages supply, and it is the only slave clock of PTP synchronization.

Figure 3: Typical topology in power system communication, (a) Physical network (b) logical network using VLAN. Quality of Service (QoS) Ethernet frames were not designed to prioritize Ethernet traffic. However, as communication is increasing in size and traffic, it is becoming more common situations sporadic traffic (or even the average traffic) can overreach LAN switching capacity for a longer time than network equipment’s buffering capacities. So, data is lost. At power system communication, there is a wide use of communications which does not guarantee the delivery of sent data. NTP time protocol, as an example, uses UDP transport protocol. GOOSE and Sampled Values protocols are mapped directly at Ethernet frame and use multicast transport mechanism. PTP protocol can be mapped directly at Ethernet, as GOOSE and Sampled Values, or at UDP, as NTP protocols. In situations where traffic cannot be dropped, Quality-of-Service (QoS) mechanism must be used to ensure data will not be lost. The QoS spares part of Ethernet port bandwidth to be used only by these messages. When using QoS, if lower priority

4

data reach its bandwidth (which is the Ethernet port bandwidth minus the spared bandwidth to higher priority messages), there will be lost of low priority data. The higher priority data will not be affected, as it has guarantee of bandwidth. Such situation is shown in figure 4.

Figure 4: High traffic flow.

There are several possibilities to separate prioritized traffic from general purposes traffics. Thus, Quality of Service can be performed in different layers and different ways. At data link layer, Priority Code Point (PCP) bits in 802.1Q Ethernet frame is one of QoS mechanism, which is called Classof-Service (CoS). As shown in Figure 5, CoS is a 3-bit field with the possibility to use eight values of priority, from 0 to 7. It is important to highlight the value 1 is the lowest priority mark. Priority 0, at the standard, is mapped as Best-effort quality, above the number 1, in order to ensure that legacy equipment traffic would not be always treated as background traffic when mixed to 802.1Q aware equipment. In addition, even though priority 7 is the highest, it is not recommended to use this prioritization in traffic that does not belong to network control or management.

Figure 5: CoS bits in 802.1Q Ethernet frame.

Real Case Scenarios Considering the switch performance test of high priority messages over an IEC 61850 network, a wide range of scenarios may be set up. Narrowing these options, this paper presents three real case scenarios, which GOOSE, PTP and Sampled Valued were tested separately. All performance tests counted on a managed Ethernet Switch with 24 Gigabit ports and switching capacity of 68Gbps. For GOOSE messages, the real case scenario considers the delay between the sent messages from publisher and received by subscriber. A managed Ethernet switch is placed between the two IEDs (publisher and subscriber), and a high traffic flow is forced to pass through this same switch in order to evaluate the GOOSE messages delay in such a case. The PTP protocol test was performed in a similar way, where the PPS delay between a master clock and a slave was measured. As in GOOSE test, the master and slave were connected through an Ethernet switch that may receive a burst of packets to simulate a high traffic flow. The Sampled Values test scenario to evaluate the switch performance was split in two parts. The first also compares the SV delay between two IEDs when the IEC 61850 network is operating in a normal condition and with high traffic flow. Different from GOOSE and PTP, as Sampled Values have a high bandwidth which is significant to determinate the switch Ethernet port, an additional analysis was performed to evaluate the Ethernet port throughput when SVs are sent to it.

5

Switch performance test for GOOSE messages To test the switch performance considering the GOOSE messages, the scenario presented in figure 6 was set up. The GPS Clock synchronized all equipment and also generated one pulse per minute (PPM) to simulate a GOOSE tripping. The Merging Unit was responsible to send the GOOSE messages to switch, and a PC was monitoring the traces using a synchronized capturing traffic board through a Pulse per Second (PPS) from the same clock. The GOOSE messages were configured into a VLAN. To generate the high traffic flow in the Ethernet Switch, others two PCs were used creating a message burst of approximately 833Mbit/s in fourteen Ethernet ports from switch. The message burst was not in same VLAN as GOOSE message.

Figure 6: Switch performance scenario for GOOSE and Sampled Values messages. The figure 7 presents the delay considering the network with and without high traffic flow. The results are based on the time the GPS Clock generated the Pulse per Minute (PPM) and the time the PC received the GOOSE message traces.

Figure 7: GOOSE messages delay. As result it can be seen that GOOSE performance in Ethernet switch is practically the same considering normal conditions and situations the switch is processing a large amount of data (11.5Gbits/s), considering there is an exclusive VLAN for GOOSE messages. Switch performance test for PTP protocol In PTP test, one GPS clock was configured as PTP grandmaster clock and another as PTP slave. To monitor the PTP delay, each PPS output from both GPS was connected in an oscilloscope. The Ethernet switch is placed between the two clocks, and the messages burst of approximately 833Mbit/s in fourteen Ethernet ports from switch is also present in this test. Figure 8 shows the test scenario.

6

Figure 8: Switch performance scenario for PTP messages. The GPS Clocks were configured to work in Peer-to-Peer (P2P) mode, as the managed Ethernet switch is PTP-aware. Besides, the PTP configuration included VLAN ID and priority as 6.

Figure 9: PTP messages delay, a) normal conditions and b) with high traffic flow. Based on figure 9, as the mean delay in both cases is in nanosecond and the same order magnitude, it can be concluded that PTP performance is not affected by high traffic flow considering it has its own VLAN, just like GOOSE messages. Switch performance test for Sampled Values The first test for Sampled Values is to check the delay in normal operation and with high traffic flow. The scenario for this test is shown in figure 6, where the test set generates a voltage and also receives the SV from the Merging Unit. The Ethernet switch is placed between the test set and the merging unit. The SVs delay is shown in figure 10.

7

Figure 10: SV messages delay, a) normal conditions and b) with high traffic flow. One more time the results from the test shows that even when the switch is dealing with high traffic flow (11.5Gbits/s), there is no delay imposed in those messages that are separately by VLAN. An additional test was performed in order to evaluate the traffic flow when an increasing number of Sampled Values messages are sent through a single Ethernet port. In this test 5 merging units were configured to operate in protection profile, which is in 80 samples per second. Monitoring one single port from Ethernet switch, the merging units were connected one by one in 5 other ports that had its traffic redirected to the monitored port. As result, the figure 117 shows the network traffic increasing when each merging unit is connected.

Figure 11: Network traffic with five Merging Units connected one by one. From this last test, it is confirmed each sampled value message for protection application consumes approximately 0.6 MB/s (or 4.8 Mbit/s) of bandwidth. Therefore, if a fast Ethernet port (100 Mbps) is exclusive for SVs, it can have up to 20 SVs messages for protection applications (80 samples per cycle), or up to 8 SVs messages if the application is for measurement (256 samples per cycle).

8

Conclusions Over IEC 61850 network, the correct switch sizing and configuration is critical to the proper functioning of the entire system. Applicable in protection, automation and control systems, switches need to have unique features such as high performance when subjected to high traffic and give priority to certain types of packets. In power system communication, IEDs must be compatible with VLAN mechanisms and traffic prioritization at least to digital (GOOSE), analogue (Sampled Values) and synchronization (PTP) packet messages separately, as they are mapped directly at the data-link layer. QoS mechanism with CoS bits is enough to guarantee prioritization for these messages, as the VLAN ID can be mapped directly per message type. IEC 61850-90-4 Technical Report recommend mapping the CoS bits as shown in figure 12. In addition, the document specifies the usage of priority High for Sampled Values data and priority Medium or High for synchronization messages. QoS configuration is applied when legacy equipment is connected as it does not allow prioritization itself. So, if there is no legacy equipment without CoS capabilities, such configuration might not be necessary.

Figure 12: CoS classification as shown in IEC 61850-90-4 Technical Report.

Finally, with well-established communication and reliable equipment systems, it is possible to have a protective architecture, automation and control distributed where the various physical devices substation are logically interconnected by the network environment.

References [1] “Communication networks and systems for power utility automation part 90-4: Network engineering guidelines”, IEC 61850-90-4 Technical Report, Edition 1.0, August 2008. [2] “IEEE Standard for Local and metropolitan area networks, Virtual Bridged Local Area Networks”, IEEE 802.1Q-2005, December 2005. [3] “Reason Switches Technical Manual”, REASON-SWITCHES-TM-EN-2v1, August 2015. [4] C. A. Dutra, I. H. da Cruz, S. L. Zimath. “Característica do barramento de processo em uma rede de comunicação baseada em IEC 61850”, XXII SNPTEE, October 2013.

9