Congestion Sensitivity of Real-Time Residential Internet Applications Richard Woundy, SVP Software & Applications Yiu Lee, Distinguished Engineer Anthony Veiga, Consultant Carl Williams, Consultant Comcast Cable Address: 1701 JFK Boulevard, Philadelphia, PA 19103 Phone: 215-286-7326, 215-286-5894 Email: [email protected], [email protected]

Abstract Bandwidth usage on residential Internet access networks has increased substantially over the last few years. This has been amplified by the rapid growth of peer-to-peer and other highbandwidth applications such as remote backups. Even social networking services offering picture sharing and audio/video streaming can use a significant amount of bandwidth in a user’s access network. Such competition for bandwidth can cause other applications to experience jitter and packet loss thereby producing varying, detrimental effects on the user-experience. The upstream path on the broadband access network is the place that has traditionally seen the least amount of provisioned bandwidth, and as a result has suffered the most from congestion. This study characterizes the behavior of commonly used real-time applications in the presence of upstream congestion. Its goal is to determine the threshold of bandwidth consumption where the user-experience degrades. Congestion impact measurements can inform ISPs about what is happening on their networks, and how they should react to congestion. Furthermore, this study considers three potential techniques to mitigate the effects of upstream congestion in DOCSIS® cable networks, and the trade-offs in choosing a particular technique. Disclaimer: This technical paper explores various potential new technologies enabled for broadband service providers, but does not represent a discussion of Comcast business plans or priorities.

1. Experimentation Purpose and Overview 1.1.

Upstream congestion impacts

Congestion will happen when applications send more traffic than the bandwidth available. These applications can consume access link bandwidth shared with other user(s) and applications. For example, social networking sites are supporting bandwidth hungry services such as picture Page 1 of 23

sharing and audio/video streaming, and leveraging AJAX and web services that increase their traffic loads even further [20]. Also, most video activity now occurs using HTTP, meaning it is now part of the Web browser experience. Peer-to-peer gaming has expanded into a richer experience for gamers and also consumes more bandwidth. Peer-to-peer applications such as P2P file distribution are another significant source of Internet traffic. The large amount of data that home backup systems push to remote servers will further add to upstream congestion. All these compete for the same available access bandwidth. Compounding the problem is the asymmetric nature of broadband networks on which upstream bandwidth is usually lower than the downstream bandwidth. In this paper, we look at upstream congestion in the access network and its potential impact on a wide spectrum of real-time applications.

1.2.

Relationship with previous studies

Studies have been done on the impact of congestion for a wide range of applications [1,2,3,8,19,24,25,26,31]. Much of this analysis has been done in isolation for a specific application. Armitage and Stewart [1] discussed the challenges of creating a control environment that provided different network conditions to study the impact of jitter and packet loss to public online game servers. Sheldon, et. Al. [2] studied a Real Time Strategy (RTS) game called Warcraft III. This work examined the statistical correlations with jitter and packet loss in terms of user interactions. Lang [3] examined the Xbox game Halo. This work covered the user experience when trying to play under highly variable delay or high-packet-loss conditions. ITUT G.114 [8] focuses on the fundamental understanding of conversational quality for digital voice systems. Chambers, et. al. [19] report on research from a comprehensive analysis of a collection of on-line game players and game workloads using data from real gaming servers. There were a number of findings showing that gamers are extremely demanding users, and unless game servers are properly set up and provisioned, gamers quickly choose to go elsewhere. The experiments reported in this paper were motivated by the need to understand bounds on the quality of user-experience for a variety of real-time networking applications operating under varying congestion conditions caused by competing traffic. Due to the asymmetric nature of broadband networks where upstream bandwidth is lower than the downstream bandwidth, the focus was on congestion on the outbound flows (upstream direction). In today’s broadband networks, congestion may be self induced by means of other parallel application flows within the user’s networks. In the case of cable networks, multiple parties share the same upstream bandwidth, thereby also allowing for neighbor induced congestion. The application response to these congestion conditions will impact the user-perceived quality of their session. A user tolerance level will vary depending on the degree of usability that in turn is a function of the application.

1.3.

Determination of users’ impression of application quality

The range of user reactions parallels the range of users’ impressions of application quality. For example, when the application is performing as intended, the user is able to function without Page 2 of 23

impediment and continues along. On the other hand, as quality degrades, the application may cease to operate, or remain active but at such a low level of usability that the user decides to terminate the application. There are other intermediate points where quality is not perfect and the user can either continue operating the application, albeit at a lesser level of usability and satisfaction. In most if not all of these scenarios, the typical user response is to contact the ISP’s support team. This paper identities and plots “inflection markers” that describe the session quality bounds of popular networking applications as perceived by a user when varying degrees and types of congestion exist in the upstream bandwidth of the access network. These inflection points can be used to better understand similarities and differences in how a wide spectrum of applications responds to outbound congestion.

2. Experimental Configuration To simulate congestion and artificial jitter, we deployed a known testing tool that accurately creates congestion characteristics of jitter and packet loss. The setup of the test environment is described in the following section.

2.1.

Network setup

Our experimental testbed uses the widely available kernel-resident dummynet module on FreeBSD. FreeBSD is stable and widely available via public download. The dummynet kernel module allows for easy configuration of packet loss and jitter. Dummynet has been shown in previous studies to be reasonably precise as a delay and loss rate tool [4,5]. The FreeBSD PC was used as an Ethernet bridge with dummynet functionality providing for delay and packet loss on the outbound traffic for the ingress interface. Application clients and software reside on the so-called home network portion of the setup while the server and Internet reside on the egress connection. Figure 1 depicts our experimental testbed setup which consists of PCs and other client application hardware connected on a private network subnet. For each application we measured user-perceived quality while the application was experiencing jitter as a result of congestion. We varied the packet loss independently for each application. Dummynet commands were invoked to create both kinds of congestion.

Page 3 of 23

Figure 1 – Experimentation Setup

The dummynet bridge was put in front of the home network and was used to add jitter (variable delay) and packet loss on the upstream only. Dummynet uses the FreeBSD’s internal packet filter to create jitter and packet loss through the identification of traffic flows in terms of pipes. One-way jitter was introduced by delaying the forwarding of bridged packets through the dummynet bridge for a specified amount of time. No jitter was introduced on the return link. This simulates congestion in the upstream direction. Different jitter values were used on the upstream link. Loss was introduced by dropping the received packets according to a set packet loss probability. Unlike simple delay and loss, jitter is harder to produce reliably because it is a statistical variation in latency rather than a completely independent variable. Dummynet does not have an inherent jitter command, and the recommended method to emulate jitter is to alter the amount of delay applied to packets during testing. To create jitter in our experiments, we, therefore, continuously varied the dummynet pipe rule to cause fluctuations in the configured latency.

2.2.

Measuring user-impressions with MOS scoring

In voice communications, the mean opinion score (MOS) provides a numerical measure of the quality of human speech at the destination end of the circuit [21]. While the MOS rating system was initially developed for phone calls, the concept of a MOS can be adapted to other medium such as video and gaming applications [22]. Having MOS-like estimates of user impression qualities for popular networking applications will help ISPs measure the application-level quality of service offered to their customers. In voice applications, MOS uses a scale of 1 to 5 based on how "annoying" a particular quality of the phone call is to the user. A listener is required to give each sentence a rating using the following rating scheme [21]: • A MOS 5 means the quality is excellent and the impairment is imperceptible. Page 4 of 23

• • • •

A MOS 4 means the quality is good and the impairment is perceptible but not annoying. A MOS 3 means the quality is fair and the impairment is slightly annoying A MOS 2 means the quality is poor and the impairment is annoying. A MOS 1 means the quality is bad and the impairment is very annoying.

In extending the concept of a MOS to other media, such as video and gaming, we use the same scale to measure different qualities of the experience. We calculated the MOS by noting users’ comments and observed trends during and after our user studies. For voice telephone calls, there are also physical tools that can be attached to the phone lines to measure the MOS analytically. For video, we adapted the MOS measurements to include features such as choppiness, video artifact, quality, and overall ability to communicate a concept. For gaming, we included reactions to environmental interaction, response to user actions, and perception of movement quality. We did not focus on the gaming technical issues but on the user experience when trying to play under high delay or high packet loss conditions. We also included some specific qualities for individual games, such as the ability to actively target and shoot other players in First Person Shooter games, or the ability to interact with Non-Player Characters (NPCs) in Role Playing Games (RPG). Each of these was broken down into a total MOS for each test session and graphed accordingly. The testing was designed to determine when the user-experience would degrade to the point of having the user cease to use the application and call customer service. To aid in this assessment, the MOS scale mapping was changed to the following: • • • • •

A MOS 5 is a normal application experience. A MOS 4 is an acceptable experience with some degradation. A MOS 3 is tolerable application experience in which the application is still usable; this experience is likely to trigger user complaints to customer support. A MOS 2 is a very poor application experience at which a user would turn the application off. A MOS 1 is the point where the application no longer functions.

The following sections describe the measurement of the MOS in our experiments for each of the applications under consideration.

3. Application Experimentation This section describes the various applications under investigation and the user-perceived results (MOS) for different congestion characteristics in terms of jitter and packet loss. For each application, graphs of MOS score results for jitter and packet loss are provided. In each graph, the y-axis gives the MOS value on a 1 to 5 scale. The x-axis plots the jitter value as the average delay within the delay variation range in units of milliseconds. The average delay is the value given to the script as a base amount of delay. This is also the mean, median and mode average of all delay values for the duration of the script. Results for packet loss are graphed with the y-axis as the MOS score and the x-axis as a percentage of packet loss. Discussion of overall results is provided in Section 5. Page 5 of 23

The application spectrum in our experiments includes Voice over Internet Protocol (VoIP), interactive real-time time video, outbound video streaming, and gaming. The selection is based on popular real-time time applications. There are commercial over over-the-top top voice providers, such as Vonage [9] and Skype [10]. Both were used in the experiments. Skype multi-person multi video conferencing was also tested. In addition, a fairly recent broadband service that is growing rapidly is the ability for a user to stream media over the Internet to his mobile phone, laptop, or PC. The popular Slingbox Pro device and Sling Player software were used [11]. Gaming applications were also included, as their constant increase in popularity and users’ high call volume to ISPs support teams when they don’t function properly make them an obvious choice. Classes of network games such as First Person Shooters (FPS) and Role Playing Games (RPG) differ in their user interaction model. Representative games from each were incorporated. For FPS, the popular Xbox 3600 game Call of Duty: Modern War Warfare fare 2 (MWF 2) was used as well as a the PC-based based games Crysis Wars and Team Fortress 2. For RPG, World of Warcraft was used [12,13,14]. Forza Motorsport 2 was selected, as it represents a genre of games (high performance vehicle racing) that is one of the staples of the computer gaming world w [15]. Street Fighter II is also used. In addition addition, MAG: Massive Action Game was used in our analysis. MAG is a multiplayer-only first--person person shooter video game developed by Zipper Interactive for the Sony PlayStation 3.

3.1.

Vonage

A human perceived MOS test was done for Vonage Vonage. Test procedures for determining human perceived MOS for Vonage involved two users. One called from a Vonage Analogue Terminal Adapter (ATA) under test to a normal land land-line line phone. The initiating caller would read a short sentence, ntence, and the listening user would attempt to interrupt with the word that he is currently hearing. This would allow the first user to gauge the round trip delay. Another method for gauging the delay was to have the users count together, as the first ccaller aller would speak all even numbers, and the second user would speak all odd numbers. The user must wait to hear the preceding number before speaking the next. In this manner, the users are able to determine whether or not they understand each other clear clearly ly enough and swiftly enough for a call to be useful for real-time time communication. Res Results are shown below in Figures 2 and 3. 3 5

MOS

4

3

2

1 0

100 300 400 Average200 Delay (milliseconds)

500

Figure 2 – Vonage – MOS – Jitter

Page 6 of 23

5

MOS

4 3 2 1 0

5

10

15

Packet Loss (percentage dropped) Figure 3 – Vonage – MOS – Packet Loss

3.2.

Skype video & voice oice conferencing

For the Skype Video testing, two clients were connected to make a Skype call. The first Skype client is a Windows PC with a headset and webcam. This PC is connected to the internal side of the dummynet bridge. The second Windows PC is connected to the exte external rnal network with a headset and webcam. Once in the call, MOS is measured by waiting approximately one minute for the call conditions to settle and then having the two users count. User one counts odd numbers and user two counts even numbers, but the user must not speak the next number until the previous has been heard over the call. This provides a subjective method for determining acceptable delay. Next, users clap in front of the camera to determine audio-to-video audio synchronicity. Then, the users make a motion in front of the camera that involves some speed and the other user understanding a message. In our testing, we quickly flashed a number using our fingers, and switched the number twice. The user on the other end speaks the number shown. By conducting ing the experiment in this manner we are able to determine the degree of delay and choppiness of the video, and to make a determination of whether or not the user-perceived user video quality is sufficient enough to provide a means for communicating ideas. the MOS score for jitter. Figure 5 graphs the MOS score for Figure 4 provides the graph of th selected packet loss points.

Page 7 of 23

5

MOS

4 3 2 1 0

500

1000

1500

2000

2500

Average Delay (milliseconds) Figure 4 – Skype Video Conferencing – MOS - Jitter 5

MOS

4 3 2 1 0

5

10

15

20

Packet Loss (percentage dropped) Figure 5 – Skype Video Conferencing – MOS – Packet Loss

3.3.

Slingbox

The setup for this test involved a Slingbox wired to a Linksys WRT54GL, connected to the dummynet bridge. The external port of the dummynet bridge connects to a Cisco 7606 router. On another subnet of the Cisco 7606 router is a Linksys WRT610N. Behind the WRT610N is a hard-wired PC. The Slingbox was connected to the local high definition cable television signal, and tuned to local unscrambled video. In the experiment, it was used to stream the signal to the PC in real-time with no recording involved. MOS was calculated based on user perceived quality on a scale of one to five based on degree of changes observed. The MOS parameters for this test looked at skipping, frame rate, video artifacts, video resolution, readability of text, audio synchronicity with visual observation and quality of the audio stream. The MOS assessments were performed by real users rather than standardized software to measure digital video quality. For the objectives of this test it was determined that a user's opinion was more valuable since this was the factor that determined whether the video quality was acceptable, and, if not, would likely trigger a service call. Figure 6 provides the MOS graph for jitter. Figure 7 plots the MOS for various degrees of packet loss. Page 8 of 23

6 5 MOS

4 3 2 1 0 0

200

400

600

800

Average Delay (milliseconds) Figure 6 – Slingbox – MOS - Jitter 6 5 MOS

4 3 2 1 0 0

2

4

6

8

Packet Loss (percentage dropped) Figure 7 – Slingbox – MOS – Packet Loss

3.4.

Gaming – Call of Duty: MWF 2

The setup for this test involved an Xbox 360 connected to the dummynet bridge. Call of Duty: Modern Warfare 2 is an Xbox Live game, meaning that it connects to a central server to facilitate the creation of a multiplayer game. Once a game session is established, it becomes a peer-hosted session where one user’s Xbox 360 becomes the game host. Twelve-to-twenty users were active at any given time during game play. Since Call of Duty is a First Person Shooter, the game is based around attacking members of the opposing team with ranged and melee weapons. Since the game involves multiple users, we measure the MOS as a weighted value against the perceptions of other players. One way the MOS was weighted was to view the “Killcam” which shows the perspective of another player before he kills the user’s character. It was possible to compare the actions of your own character shown in this “Killcam” against the perceived actions from your own gameplay to determine whether or not your actions were received by the host and other players, and ultimately determine the quality of the gaming session. Figures 8 and 9 plot the graphs for jitter and packet loss respectively. The results for packet loss are quite different from those for increased delay. The MOS was measured from the perspective of the tested user. Packet loss rates start to impact the game performance from the perspective of non-tested users Page 9 of 23

because packets from the tested user to the server get lost and the non-tested users are unaware of the location information of the tested user. This was determined by the fact that the non-tested users began to terminate the sessions in the middle of game playing because they were not receiving correct information about the tested user. The tested user has a distinct advantage over the non-tested users. Since the packet loss is only on the forward link and not the return, the tested user receives all location information of non-tested users. Despite or rather because of its “unfair” play advantage, the tested user was eventually left alone in the game. The MOS accounted for this negative result in the scoring of the user quality. 5

MOS

4

3

2

1 0

50

100

150

200

250

300

350

400

450

Average Delay (milliseconds)

Figure 8 – Gaming: Call of Duty: MWF 2 – MOS – Jitter 5

MOS

4

3

2

1 0

5

10

15

20

Packet Loss (percentage dropped)

Figure 9 – Gaming: Call of Duty: MWF 2 – MOS – Packet Loss

3.5.

Gaming – Crysis Wars

The setup for this test involved the first PC with the Crysis Wars game client connected to the dummynet bridge. The Crysis Wars client will associate with a locally installed game server on the external subnet of the dummynet bridge. The second external PC was used for a second local player. MOS was measured by comparing the actions of a character with the other user’s screen, to determine if these actions were representative of what the initial user intended. In order to Page 10 of 23

determine the playability of the game under load conditions, the player whose network was under duress was asked to move in erratic patterns through the game world while the external user attempted to hit the character with a sniper rifle. This is difficult when movement is erratic, but impossible if the graphical representation of the character’s location does not match the actual location of the character. In this way, Crysis Wars had an interactive game play and, therefore, a weighted MOS. Figures 10 and 11 plot the graphs for MOS with jitter and packet loss respectively. 5

MOS

4 3 2 1 0

200

400

600

Average Delay (milliseconds) Figure 10 – Gaming: Crysis – MOS - Jitter 5

MOS

4 3 2 1 0

20

40

60

80

100

Packet Loss (percentage dropped) Figure 11 – Gaming: Crysis – MOS – Packet Loss

3.6.

Gaming – Forza Motorsport 2

The setup for this test involved an Xbox360 connected to the internal port of the dummynet bridge. The external port of the dummynet bridge connects to another subnet of the 7606 where a second Xbox 360 resides. Although Forza is a multiplayer game, the Xbox Live service is again used to facilitate the setup of a game session, at which point a user’s Xbox 360 becomes the host. For this test, it was possible to create a locked match in which only the two local Xbox 360 machines were able to join. This enabled a more controlled environment for testing. Again, Page 11 of 23

since this game involves other users, a weighted MOS was used. One of the criteria measured was the interaction of the two vehicles. At high loss, it became possible for one driver to drive through the space a vehicle was currently occupying, even though the game rules state that there should be a collision between the two vehicles. Figures 12 and 13 plot the graphs for MOS with jitter and packet loss respectively. 5

MOS

4 3 2 1 0 100200300400500600700800900100011001200 Average Delay (milliseconds) Figure 12 – Gaming: Forza – MOS – Jitter 5

MOS

4 3 2 1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Packet Loss (percentage dropped) Figure 13 – Gaming: Forza – MOS – Packet Loss

3.7.

Gaming – World of Warcraft (WoW)

The setup for this test involved a PC with World of Warcraft installed and connected to the internal port of the dummynet bridge. The external port of the dummynet bridge connects to another subnet of the Cisco 7606 where a second PC with WoW resides. Since WoW is a Massively Multiplayer Online Role Playing Game, both clients connected to the same realm, or collection of game servers. Due to the nature of the game, there were a large number of other players connected to the same game realm. This application also used a weighted MOS based on interactions, including the perception of “rubber-banding,” a common phenomenon in similar games that shows a character’s location abruptly change to a much larger distance than their Page 12 of 23

capabilities would suggest possible, due to a loss or delay of location transmission. Figures 14 and 15 plot the graphs for MOS with jitter and packet loss respectively. 5

MOS

4 3 2 1 0

200

400

600

800

Average Delay (milliseconds) Figure 14 – Gaming: WoW – MOS – Jitter 5

MOS

4 3 2 1 0 5 10 15 20 25 30 35 40 45 50 55 60 Packet Loss (percentage dropped) Figure 15 – Gaming: WoW – MOS – Packet Loss

3.8.

Gaming – Team Fortress 2

Team Fortress 2 is a team-based, first-person shooter multiplayer video game. The setup for this test involved a PC with Team Fortress 2 client installed and connected to the internal port of the dummynet bridge. The external port of the dummynet bridge connects to another subnet of the Cisco 7606 where a second PC with a Team Fortress 2 client resides. Users from both the clients joined a Team Fortress 2 server that had no other players. MOS scoring involved moving both players to the same location. One character in TF2 was used to observe the movement of the other character. Scoring involved moving, jumping and firing of weapons. Figures 16 and 17 plot the graphs for MOS with jitter and packet loss respectively.

Page 13 of 23

5

MOS

4

3

2

1 0

50

100

150

200

250

300

350

Average Delay (milliseconds)

Figure 16 – Gaming: Team Fortress 2 – MOS – Jitter 5

MOS

4

3

2

1 0

10

20

30

40

50

60

Packet Loss (percentage dropped)

Figure 17 – Gaming: Team Fortress 2 – MOS – Packet Loss

3.9.

Gaming – MAG

MAG, previously titled MAG: Massive Action Game, is a multiplayer-only first-person shooter video game developed by Zipper Interactive for the Sony PlayStation 3 (PS3). The setup for this test involved a PS3 gaming console connected to the internal port of the dummynet bridge. The dummynet bridge connects to another subnet of the Cisco 7606 where a second PS3 gaming console resides with a MAG game user. Both MAG users joined the same gaming session with other external players. MOS Scoring involved moving both players to the same location. Scoring involved moving, jumping and firing of weapons. Figures 18 and 19 plot the graphs for MOS with jitter and packet loss respectively.

Page 14 of 23

5

MOS

4

3

2

1 0

100

200

300

400

500

600

Average Delay (milliseconds)

Figure 18 – Gaming: MAG – MOS – Jitter 5

MOS

4

3

2

1 0

5

10

15

20

Packet Loss (percentage dropped)

Figure 19 – Gaming: MAG – MOS – Packet Loss

3.10. Gaming – Super Street Fighter IV (SSF 4) Super Street Fighter, commonly abbreviated as SSF, is a series of fighting games developed in Japan in which the players pit the video games' competitive fighters from around the world, each with his or her own unique fighting style, against one another. Two PlayStation 3 gaming consoles were installed with Super Street Fighter IV (SSF 4) hosting an endless match. Only one character was played to observe the movement on both display screens. MOS scoring was determined by time sensitive moves such as combo attacks and the use of a complex set of strung-together commands. Figures 20 and 21 plot the graphs for MOS with jitter and packet loss respectively.

Page 15 of 23

5

MOS

4

3

2

1 0

50

100

150

200

250

300

350

Jitter (average milliseconds)

Figure 20 – Gaming: SSF 4 – MOS – Jitter 5

MOS

4

3

2

1 0

5

10

15

Packet Loss (percentage dropped)

Figure 21 – Gaming: SSF 4 – MOS – Packet Loss

4. Mitigation Techniques in DOCSIS Networks The above study characterizes the behavior of commonly used real-time applications in the presence of upstream congestion. We were able to determine the threshold of bandwidth consumption where the user-experience degrades. These congestion impact measurements can inform ISPs about what is happening on their networks, and how they should react to congestion. In particular, we explore three general techniques for the mitigation of upstream congestion impacts on real-time applications in DOCSIS networks: 1. Steer real-time application traffic to a second DOCSIS service flow that is configured with Quality of Service and a higher priority than the default service flow. 2. Steer real-time application traffic to a second service flow that is configured with the same best-effort service and priority as the default service flow. 3. Reduce the queue length of the default service flow within the cable modem.

Page 16 of 23

4.1.

Technique 1: Use second service flow with QoS/higher priority

The first technique is to create a second service flow for latency-sensitive traffic, using QoS and a higher DOCSIS traffic priority than the default service flow. This second service flow would be very similar to the service flows used for those used in MSO Digital Voice products deployed today. DOCSIS defines a number of service flow scheduling types to enable QoS: non-real-time polling service, real-time polling service, unsolicited grant service, and unsolicited grant service with activity detection [34]. One benefit of this technique is its usage of existing, deployed technology. Another benefit is its effectiveness in the presence of both self-congestion (congestion caused solely from the subscriber’s own traffic) and shared-congestion (congestion caused by the aggregate traffic load of subscribers sharing the same network upstream). The downside of this technique is that network engineering must allocate bandwidth carefully to the new service flows, and must be careful to manage capacity planning for the upstream networks. In particular, one must be careful not to starve the best-effort service flows for other Internet traffic, in order to accommodate the new service flows for latency-sensitive traffic.

4.2.

Technique 2: Use second service flow at best-effort priority

The second technique deals with the buffer management strategy that is in operation at the upstream service flow queue located in the cable modem. The cable modem implementation may leverage a large upstream queue length in an attempt to ensure maximum bandwidth utilization for TCP flows over the default service flow; however, a full upstream queue can impact application flows that are latency sensitive [30]. In this second technique, we introduce a modified upstream service flow to handle the real-time application traffic. In this approach, a second service flow for latency-sensitive traffic is created, but at the same best-effort priority as the default service flow. We tested this technique in our lab with self-congestion, with very good results. The real-time application traffic was classified (identified) either by the incoming Ethernet port (e.g. for Xbox and PS3 game consoles) or by the TCP or UDP port numbers (e.g. for specific online games running on the PC), and the traffic was marked with particular DiffServ code points [33]. While the lab used a Lenovo netbook S10e running Ubuntu for the traffic classification and marking, a home gateway compliant with the Home Gateway Initiative specification [32] could have been used instead. The Vonage ATA generated traffic with the same DiffServ code points, so its traffic did not need to be marked. On the cable modem, traffic marked with the DiffServ code points was redirected to the second service flow. One benefit of this technique is the usage of existing, deployed technology – the same as used for the best-effort service flow. Capacity planning is also simplified compared to the first technique (using QoS), because all of the Internet traffic is still treated as best effort.

Page 17 of 23

One downside of this technique is that network engineering must still allocate bandwidth to the second service flow, in addition to the bandwidth allocated to the default service flow. Another downside is that this technique may not be as effective as the first technique (using QoS) for shared-congestion cases.

4.3.

Technique 3: Enable shorter queue length for default service flow

The third technique is to modify the operation of the default service flow to use a shorter queue length within the cable modem implementation. The variation of queue lengths for default service flows has been modeled and studied in [30], and simulation results indicate that an upstream queue length beyond a certain point (i.e. the “bandwidth delay product” or BDP) was non-productive for TCP throughput (i.e. longer queues did not improve TCP bandwidth consumption) and was counter-productive for VoIP quality (i.e. longer queues degraded VoIP quality due to queuing delays). The benefit of this technique is that there are no changes needed to bandwidth allocation or capacity planning, because the existing default service flow is used. The downsides are that the queue size is not currently a DOCSIS configurable parameter (so this is not current technology), and that this technique may not be as effective for shared-congestion cases as the first technique (using QoS).

5. Conclusions Below is a summary of the various inflection points indicating where each type of network effect degrades the user experience to change it from acceptable to unacceptable. The inflection points for the tested applications are compared to each other in single graphs. Figures 22, 23 and 24 provide a survey of inflection points. Figure 22 shows a MOS score of 2 for each application, where the amount of jitter and loss would cause the user’s experience to become unacceptable. Figure 23 shows a MOS score of 3 and Figure 24 shows a MOS score of 4. These graphs are provided to show the amount of jitter and loss that are acceptable for an application to perform at each plotted quality level. Figure 22 shows the network conditions that would cause a user to quit the application. Figure 23 shows conditions in which the application experience is tolerable enough to continue playing, but is likely to trigger user complaints to customer support. Figure 24 shows conditions which, while not desirable for user-experience, would still allow applications to function. Each of the tested applications is unique in the way in which it deals with network degradation. Although many of the applications had fairly close points of inflection for each type of network degradation, some were outliers. As an example, Crysis was able to function with a very large amount of packet loss (50%) compared to Call of Duty and Forza (10%). World of Warcraft was also acceptable at higher levels of loss, though this can be due to the fact that it was the only application tested that used TCP. TCP has a built-in mechanism to retransmit lost packets to Page 18 of 23

convert packet loss to delay. Because of this, World of Warcraft may have been able to interpret loss as another form of delay, and cope appropriately. Failure points for various applications Vonage

80

Call of Duty

Packet Loss (percentage dropped)

70

Skype 60

Crysis 50

Forza Motorsport World of Warcraft

40

Slingbox 30

MAG 20

TF 2 10

SSF 4

0 0

100

200

300

400

500

600

700

800

900

1000

1100

1200

1300

1400

1500

Delay (milliseconds)

Figure 22– MOS 2 for Tested Applications Failure points for various applications Vonage

60

Call of Duty

Packet Loss (percentage dropped)

50

Skype Crysis 40

Forza Motorsport World of Warcraft

30

Slingbox 20

MAG TF 2

10

SSF 4

0 0

100

200

300

400

500

600

700

800

900

1000

1100

1200

1300

1400

1500

Delay (milliseconds)

Figure 23– MOS 3 for Tested Applications

Packet Loss (percentage dropped)

Failure points for various applications 45

Vonage

40

Call of Duty Skype

35

Crysis 30

Forza Motorsport 25

World of Warcraft

20

Slingbox 15

MAG 10

TF 2

5

SSF 4

0 0

100

200

300

400

500

600

700

800

900

1000

1100

1200

1300

1400

1500

Delay (milliseconds)

Figure 24– MOS 4 for Tested Applications

Page 19 of 23

One general finding from the aggregation of the application experience inflection points is that video game applications perform better than other types of applications when presented with large amounts of packet loss. Of the applications tested, most applications became unacceptable between five and ten percent packet loss. There was no common range at which applications became unacceptable for values of delay. While all of the voice and video applications and half of the games were within the five and ten percent range for loss, the fact that voice, video and gaming applications all had strongly varying reactions to delay suggests that no common range of acceptable delay can be inferred. In order to prevent unacceptable experiences with home applications, the access networks’ goal should be to prevent loss from reaching 10 percent, and to provide the best customer experience loss should not exceed five percent. The level of acceptable jitter when measured by MOS methodology varies from application to application. No common trends or boundaries can be deduced over the spectrum of applications under study. Our research shows that mitigating delay and packet loss generally will improve user experience; however, some applications are more sensitive to delay and packet loss than others. For different real-time applications, we show that user-perception of quality varies differently even when under similar types and levels of congestion. Having application profiles for the MOS scores of popular real-time applications makes ISPs aware that they must take into account the totality of application spectrum rather than responding in isolation to specific application performance. This work concludes with a survey of those profiles and provides preliminary insights into a complicated landscape of congestion. The research was done for the most pressing congestion scenario of access link upstream traffic flows. Our research also identified three potential mitigation techniques, to support a good user experience for real-time applications in the presence of upstream congestion. One technique is to leverage a second service flow with QoS and higher priority (similar to service flows used for Digital Voice products), but that technique requires careful capacity planning. We tested a second technique, leveraging a second best-effort service flow; this technique works well for self-congestion conditions, but may not be ideal for shared-congestion conditions. A third technique, using shorter queue lengths for the best-effort service flow, shows interesting results in simulations, but is not yet available in production networks.

6. Acknowledgements The authors would like to acknowledge the contributions of Alain Durand of Juniper Networks, who was instrumental in the design of the lab experiments, and was a contributor to earlier versions of this technical paper.

7. Abbreviations and Acronyms AJAX ATA BDP DiffServ

Asynchronous JavaScript and XML Analog Telephony Adapter Bandwidth-delay Product Differentiated Services Page 20 of 23

DOCSIS FPS FreeBSD HGI HTTP ISP MAG MOS MWF NPC P2P PC QoS RPG RTS RTT TCP UDP VoIP WoW Xbox

Data Over Cable Service Interface Specification First-person shooter Free Unix Operating System Distributed via the Berkeley Software Distribution Home Gateway Initiative Hypertext Transfer Protocol Internet Service Provider Massive Action Game Mean Opinion Score Call of Duty: Modern Warfare 2 Non-player Character (controlled by the gamemaster in role-playing games) Peer-to-Peer Personal Computer Quality of Service Role-playing game Real Time Strategy Round Trip Time Transmission Control Protocol User Datagram Protocol Voice over Internet Protocol World of Warcraft Video Game Console Manufactured by Microsoft

8. References [1] G. Armitage, L. Stewart, “Some Thoughts on Emulating Jitter for User Experience Trial,” in Proceedings of the 3rd ACM SIGCOMM workshop on Network and system support for games, August 30, 2004, Portland, Oregon, pp. 157 – 160. [2] N. Sheldon, E. Girard, S. Borg, M. Claypool and E. Agu, “The Effect of Latency on User Performance in Warcraft III,” Technical Report WPICS-TR-03-07, Computer Science Department, Worcester Polytechnic Institute, March 2003. [3] T. Lang, “User Experience while Playing Halo with network delay or loss,” CAIA Technical Report 031205A, http://caia.swin.edu.au/reports/031205A/CAIA-TR031205A.pdf, December 5, 2005. [4] L. Rizzo, “Dummynet: a simple approach to the evaluation of network protocols,” ACM Computer Communication Review, Vol. 27, No 1, pp. 31-41, January 1997. [5] W. A. Vanhonacker, “Evaluation of the FreeBSD dummynet network performance simulation tool on a Pentium 4-based Ethernet Bridge,” CAIA Technical Report 031202A, http://caia.swin.edu.au/reports/031202A/CAIA-TR-031202A.pdf, December 2003. [6] FreeBSD, http://www.freebsd.org (as of August 2010).

Page 21 of 23

[7] G. Appenzeller, I. Keslassy, and N. McKeown, “Sizing router buffers,” in Proceedings of ACM SIGCOMM04, 2004. [8] ITU-T G.114 (5/2003) SERIES G: TRANSMISSION SYSTEMS AND MEDIA, DIGITAL SYSTEMS AND NETWORKS, Figure 1, page 3, One way transmission time. [9] Vonage, www.vonage.com (as of August 2010). [10] Skype, www.skype.com (as of August 2010). [11] Slingbox, www.slingmedia.com (as of August 2010). [12] Call Of Duty (COD), www.callofduty.com (as of August 2010). [13] Crysis, www.ea.com/games/crysis (as of August 2010). [14] World of Warcraft (WoW), www.worldofwarcraft.com (as of August 2010). [15] Forza, www.forzamotorsport.net (as of August 2010). [16] hping, www.hping.org (as of August 2010). [17] lighttpd, www.lighttpd.net (as of August 2010). [18] Minacom, www.minacom.com (as of August 2010). [19] C. Chambers, W. Feng, S. Sahu, and D. Saha, “Measurement-based Characterization of a Collection of On-line Games”, Internet Measurement Conference, Proceedings of the 5th ACM SIGCOMM conference on Internet Measurement, 2005. [20] H. Jayathilaka and H. Porawagama, “Network Resource Utilization and Social Networking”, http://people.apache.org/~hiranya/SocialNetworking.pdf. [21] ITU-T P.800-199608, SERIES P: TELEPHONE TRANSMISSION QUALITY, Methods for objective and subjective assessment of quality, http://www.itu.int/rec/T-REC-P.800199608-I/. [22] Recommendation ITU-R BT.500-11, Methodology for subjective assessment of the quality of television pictures, http://bellerofonte.dii.unisi.it/dottorato/corsi/corsi2003/Perceptual%20Image%20Modeling/F urther_readings/Part4/ITU-R_BT.500-11.pdf. [23] Recommendation ITU G.107, The E-model, a computational model for us in transmission planning.

Page 22 of 23

[24] P.Pragtong, K.Ahmed, T.Erke, “Analysis of speech data rate in voice over IP conversation”, IEEE TENCON, Thailand, vol.1, pp.143–146, Nov. 2004. [25] L.Roychoudhuri, E.Al-Shaera, G.B.Brewstera, “On the impact of loss and delay variation on Internet packet audio transmission”, Computer Communications, vol. 29, No.10, pp.15781589, June 2006. [26] J. But, S. Burriss, G. Armitage. Priority Queuing of Network Game Traffic over a DOCSIS Cable Modem Link. In Proceedings of Australian Telecommunications and Network Applications Conference (ATNAC), December 2006. [27] Peterson, J. and A. Cooper, "Report from the IETF Workshop on Peer-to-Peer (P2P) Infrastructure, May 28, 2008", RFC 5594, July 2009. [28] L. Zhang and D. Clark, Oscillating Behavior of Network Traffic: A Case Study Simulation, Internetworking: Research and Experience, Vol. 1, pp. 101-112, 1990. [29] Hidenari, S., Y. Hori, H. Sunahara and H. Sunahara, 1997. Characteristics of UDP packet loss: Effect of TCP traffic. Proceeding of the 7th Annual Conference of the Internet Society, June 24-27, Kuala Lumpur, Malaysia. http://www.isoc.org/inet97/proceedings/F3/F3_1.HTM. [30] Martin, J., Westall, J. Shaw, T., White, G., Woundy, R., Finkelstein, J., Hart, G., 2010, Cable Modem Buffer Management in DOCSIS Networks, Proceedings of the IEEE Sarnoff 2010 Symposium, Princeton NJ, April 2010. [31] Carsten Griwodz , Pal Halvorsen, The fun of using TCP for an MMORPG, Proceedings of the 2006 international workshop on Network and operating systems support for digital audio and video, November 22-23, 2006, Newport, Rhode Island. [32] Home Gateway Initiative Report HGI-GD013-R2, HGI Guideline Document: QoS White. http://www.homegatewayinitiative.org/publis/HGI-GD013-R2.pdf. [33] Nichols, K., Blake, S., Baker, F., and D. Black, “Definition of the Differentiated Services Field (DS Field) in the IPv4 and IPv6 Headers”, RFC 2474, December 1998. [34] CableLabs, “MAC and Upper Layer Protocols Interface Specification CM-SPMULPIv3.0-I13-100611”, 2010.

Page 23 of 23