A NGEWANDTE M ATHEMATIK UND I NFORMATIK ¨ ZU K OLN ¨ U NIVERSIT AT

Report No. 98.334 FEC Supported Congestion Control in One to Many Reliable Multicast by Frank Brockners August, 1998

Center for Parallel Computing (ZPR) University of Cologne Weyertal 80, D–50923 K¨oln

Keywords: Congestion Control, Forward Error Correction (FEC), Reliable Multicast

FEC Supported Congestion Control in One to Many Reliable Multicast Frank Brockners Center for Parallel Computing (ZPR) University of Cologne [email protected] Technical Report ZPR-TR 98-334 August, 1998

Abstract

1 Introduction

This paper describes the design of a new FEC fueled rate and congestion controller (RCR) which is targeted primarily at one to many reliable bulk multicast data transfer. The concept of RCR is validated by analytical and simulation results. The heuristic analysis is based on a new extended model for flows, which implement a congestion control algorithm similar to TCP. The main goal was to develop an algorithm which is TCPfriendly and competes with the control loop of TCP. Large delays in the control circuit, scaling issues and high packet loss probabilities are among the major challenges of a reliable multicast rate controller implementing TCP-fair congestion control. The controller presented in this paper shows that layered forward error correction (FEC) redundancy coding not only reduces the control- and retransmissions in reliable multicast environments using automatic repeat requests (ARQ) to trigger retransmissions but also helps a congestion controller to compete with TCP. FEC permits the receivers to tolerate losses in a way that they only need to signalize loss levels which are significant for them. This high-pass filtering of the loss signal reverts the disadvantage of reacting slowly to loss into an advantage in direct comparison to TCP. The greater responsiveness with respect to rate increment ensures that TCP always receives its share of the total available bandwidth. Simulations and analytical analysis for multicast as well as unicast setups show that already a very moderate level (some %) of redundancy suffices to strengthen a connection suffering from long delays and high loss probabilities. Keywords: Congestion Control, Forward Error Correction, Reliable Multicast.

Congestion control is vital for the internet and is even more important for reliable multicast setups. Besides fairness and throughput considerations, flow and congestion control has been recognized as a major problem for reliable bulk multicast data transfer. Existing approaches for congestion control schemes in reliable multicast can be devided into closed-loop and open-loop control schemes. All closed loop control schemes have in common that receivers send some kind of feedback information (either positive or negative acknowledgements) back to the source. The source evaluates the feedback information and accordingly adjusts its emission rate. A great varience of the present reliable multicast frameworks use closed-loop congestion control. In open-loop scenarios a control path from the receivers to the source does not exist. Open-loop solutions employ a preconfigured rate, high levels of redundancy coding or repeated transmissions. Repeated transmissions denotes the sending of the same information elements multiple times. This may either be at different times or to different multicast groups. The controller presented in this paper primarily relies on close-loop congestion control but also employs aspects of open-loop control, as it utilizes redundancy coding to increase protocol efficiency. We take a short glance at the different approaches for flow and congestion control for reliable multicast before studying our approach which integrates TCP-fairness and TCPcompetitiveness. Sender-initiated approaches for flow control that use positive acknowledgments (ACKs) always face the ACK implosion problem while receiver oriented schemes do so too in the absence of an NAK aggregation or suppression scheme. In [1] Towsley et al. demonstrated that receiver-initiated (NAK 1

based) error recovery usually is superior to sender-initiated error recovery. Depending on the way how receiver initiated protocols solve the NAK implosion problem, they can be divided into (1) timer based, (2) deterministic and (3) structure (tree) based approaches. Examples include SRM [2] for a timer based, DTRM [3] for deterministic and RMTP [4] as well as TMTP [5] for tree based approaches. In each of these approaches the emission rate is either restricted to a small tolerable level or a TCP-like congestion control is implemented in order to prevent or resolve congestion. RMP [6] and RMTP serve as examples for TCP-like window controlled congestion control schemes. Both rely on positive ACKs, either to the toke site (RMP) or to the designated receivers (RMTP). NAK based schemes significantly reduce protocol control traffic but lack a control signal to clock themselves (like ACKs in TCP do). Instead they usually employ some kind of of rate control. Explicit rate adjustment is common for most of the approaches. Be it close-loop congestion control (RMTP [4], TMTP [5]) or open-loop control with a fixed emission rate like SRM. Layered multicast schemes, including the one based on FEC which transfers the TCP-principles of congestion window increase and decrease to group joins and leaves of the receivers [7] are examples for open-loop congestion control. A problem common to all controllers interpreting loss as a congestion signal is, that packet loss is very likely to occur: For a heuristic argument one may think of 50 uncorrelated flows on disjoint paths in the network, each with a packet loss probability of 1%. The overall packet loss probability amounts to as much as 40%. Loss rates of this magnitude destroy any kind of TCP-like congestion control [8] – A rate controller would end up with the minimum emission rate permitted. [9] and [10] give an in depth analysis of the loss pattern of the MBone. A possible solution of this problem is to tolerate a certain amount of loss. [11] discusses a monitoring based approach whereas LTRC [12] directly measures the loss rate and adjusts the rate increase-decrease function of the controller. The flow- and congestion controller RCR presented in this paper also uses a TCP-like emission rate control and redundancy (FEC coding) to implement loss tolerance. FEC enables each receiver to filter the loss signal locally and to react to the occurance of loss individually. The active use of redundancy in the control circuit compromises between TCP-friendliness and TCP-competitiveness. This paper is structured as follows: After a short overview about the distributionmodel, section 2 focuses on some aspects of reliable congestion control and discusses the fairness aspect. The discussion of the design of the controller in section 3 is followed by an analytical model of the controller for the later analysis in section 5. Section 5 compares the results obtained

from the model with those of a prototypical implementation within the ns-2 [13] network simulation tool.

1.1 Overview of the distribution model We follow our protocol design given in [14] of a one to many distribution model which runs on top of the IP-multicast service and is tree based. It is reliable in a way that a receiver either receives all data correctly or is able to detect an unrecoverable situation. The protocol achieves reliability by using both - redundancy coding (FEC) and automated repeat requests (ARQ). It follows a complete receiver initiated retransmission scheme and uses the ttl hop count to form local groups. The burden for correct reception of all packets is carried by the receivers. By means of an expanding ring search each receiver finds a so-called leader, who is closer to the source and has the capability to locally retransmit packets with a limited hop count value. This implies that the IP multicast tree has to be symmetric. Receiver feedback (e.g. ARQ) is sent to the leader only, who decides whether to forward, take actions, or simply discard it. The sender does not keep any state information about the set of receivers. The controller this paper is about only uses [14] as framework. The tiny and straight forward rate control algorithm described in [14] was completely replaced. Therefore, the results and conclusions of this paper can easily be transferred to other multicast and even unicast scenarios.

2 Some design issues 2.1 Fairness considerations Stability and robustness of the internet heavily depend on congestion controlled flows. A controller has to interpret packet loss as a congestion signal and back off its emission rate in conditions of loss. With a constantly growing internet, end-to-end congestion control becomes a vital factor. In [15] Floyd and Fall show the negative effects of flows which are unresponsive to loss and emphasize the need for continuing the use of end-to-end congestion control. Therefore, they propose router mechanisms to restrict the bandwidth of best effort flows to a disproportionate share of the overall available bandwidth. Up to 90% of today’s internet WAN traffic uses TCP [16] as transport protocol. This motivates new protocols to take TCP as reference and adjust their congestion control to be TCP-friendly. [15] defines a flow to be TCP-friendly, if the arrival rate of the flow does not exceed the bandwidth of a corresponding TCP connection under the same circumstances. Basically this may be achieved by a control algorithm similar to the method employed by TCP, i.e. by reducing the emission rate to half 2

of the original value in case of congestion and by increasing the emission rate to no more than one additional packet per round trip time (RTT) otherwise. The congestion control algorithm of TCP constantly checks the network for additional bandwidth by linearly increasing the congestion window. In case of loss, TCP backs off exponentially by reducing its congestion window to half of its original size. The fast response of TCP to losses keeps the competition among large sets of TCPconnections fair. The nature of reliable multicast as a point to multipoint control scheme may have a significant negative impact on the overall throughput of the network. A reliable multicast protocol running in parallel with TCP primarily faces two facts: (1) TCP reacts to loss quite rapidly and (2) TCP (New-RENO/SACK) recovers from a drop (quite) fast. While the first one makes TCP (and the overall network) vulnerable to applications which do not implement congestion control the second capability makes TCP a tough competitor for bandwidth.

his individually observed loss signal and only reacts if loss exceeds the level which could be tolerated through FEC coding. If such a scheme is run with a couple of TCP connections in parallel, we observe the following: If loss is distributed equally among all active connections, TCP will also discover the loss situation and immediately correspond to it. Immediate means faster than the reliable multicast connection running in parallel. Once the high-pass (FEC) filter produces an output signal, a receiver issues a congestion notification packet which is forwarded towards the source leading the source to drop its emission rate to half the current rate. Additionally, the source adjusts the gradient of the emission rate to half the current value. Figure 1 shows a RM source with a far slower control circuit than a TCP connection operating in parallel. In the absence of redundancy, the source has to react to every packet loss in the network. TCP recovers much faster from a loss and wins the competition for the bottleneck bandwidth.

2.2 Partial decoupling of congestion and retransmission control An environment which uses rate control at the source and relies on local repair retransmissions for loss recovery faces a principle problem: If the receive window of receivers, which are elected to do local retransmissions, is not smaller than the transmit window of the sender, sets of receivers may lose track of the transmission, although they have sent negative acknowledgements in time. Therefore, retransmission control and congestion control has to be decoupled partially. This results from the fact that retransmission control often has only local scope whereas congestion control always has to take into account the whole group.

Figure 1: RM source without FEC support in a setup with RTTreliablemulticast  RTTTCP . In particular tree-based reliable multicast protocols which aggregate feedback at the tree nodes introduce further delay into their control circuit. Delay and timers keep them from overreacting to certain constallations because they virtually have to react to the ‘wishes’ of thousands of receivers. Therefore they are not in the condition to be highly adaptable to changes. But in order to be TCP-friendly they have to exponentially back off their rate on discovering heavy loss. Additionally, it is desirable to restrict the amount of expensive control traffic to a minimum. This results in a very conservative increment of the emission rate of the source after rate back off. One possibility to solve this contradiction between speed and fairness is to implement forward error correction. FEC works as a high pass for the loss signal. In case of low, uncorrelated loss the source will not be requested to slow down. With FEC, every receiver filters

Figure 2: Network example to explain the necessity of a partial decoupling of congestion and retransmission control. The elected protocol control tree and local buffer sizes are shown. Consider a network constellation like in figure 2 with three receivers (3,4,5) and one source (0). Two receivers are connected to the source through a bottleneck link. The control tree is shown with dashed lines. Node 4 works as leader (this is a 3

receiver, which is elected to aggregate and forward NAKs as well as to do local retransmissions) for nodes 3 and 5 and is responsible for NAK and retransmission control for these nodes. Assuming, that loss will only occur on the bottleneck link, receivers 3 and 5 will force receiver 4 to retransmit lost packets. As receiver 4 cannot notice any loss situation by itself, it does not need to contact its leader which is source in the given example. Besides, the problem of increasing congestion through the retransmissions of receiver 4 this scenario might lead to a situation where one of the receivers 3 or 5 requests the retransmission of a packet which was already removed from the buffer of 4. This is the first time, 4 needs to contact its leader that neither holds the packet in its buffers any longer, as its buffer has the same size as that of receiver 4. A naive solution for this problem is to ask for very large buffers at the source — buffers that are larger than the buffer of any receiver. This is impractical because the buffer capabilities are usually not known by the sender. The suitable solution for this problem is to partially decouple congestion and retransmission control: Another control message is used to signal congestion to the source. These congestion notification messages (CGN) are also transmitted through the control tree but are always forwarded to the source. In addition, the control tree is used to aggregate the congestion notification messages to avoid a CGN implosion problem. Congestion notification messages inform the source about the first packet lost and the amount of loss the receivers observed.

or rate decrease. Otherwise the packet-scheduler clocks the controller. Although being designed for a reliable multicast environment (because big latencies in the control process are common for reliable multicast) the concepts of the controller apply to unicast environments as well. Source and receivers participate in the congestion control scheme of RCR. Receivers issue NAKs or congestion notification messages (CGN). NAKs are sent to the individual leader of a receiver, while CGNs are directed towards the source. Both are unicasted and both follow the control tree formed by the election process. The following sections describe the part of the sender as well as the part of the receiver of the control process in detail. The following sections describe the rate control at the sender and at the receivers as well as the determination of the parameter “round trip time” in detail. While congestion control at the sender closely follows the design principles of TCP, FEC usage allows fundamental modifications at the receivers. The resulting controller suits the need for a controller which is TCPfriendly and TCP-competitive.

3.1 Sender The flow and congestion control of the sender consists of two parts. During the start phase the sender multiplicatively increases the send rate. In the absence of any concrete knowledge of the network or the receivers the sender starts at a given minimum rate and lowers the inter packet time gap t by a constant factor after each timer expiry. After being notified of congestion in the network the sender switches to the main phase and henceforth only linearly increases the rate.

3 The design of the rate controller The rate controller (RCR - a Rate Controller supported by Redundancy) achieves TCP-friendliness and TCPcompetitiveness by implementing a TCP-like control process which is supported by redundancy through FEC coding. TCP-friendly rate controllers [17] have been proposed for unicast [18] and also reliable multicast [12] environments. LTRC, as presented in [12] achieves competitiveness by adjusting the reaction of the controller to loss depending on the amount of loss encountered by the receivers. RCR is the first controller to actually implement and analyze the effects of FEC deployment within the controller. FEC can be used to high pass filter the loss signal in a very efficient way to be loss tolerant. In contrast to controllers implementing loss tolerance at the source, FEC focuses loss tolerance on the receivers. This reduces the susceptibility of the controller to delay in the control circuit and the amount of control traffic from the receivers which would otherwise be aggregated (e.g. in tree-based protocols like RMTP [4]) or suppressed (e.g. in SRM [2]). In contrast to TCP (which is clocked by ACKs) the RCR is clocked by time as long as the time interval between two packets is smaller than the time-outs for rate increase

Figure 3: Congestion control state diagram of the sender At transmission startup time, the sender initializes the inter packet gap t to

t

tmax = rate8s

min

(1)

seconds. s denotes the packet size in bytes, ratemin the minimum emission rate allowed. ratemin as well as the maximum emission rate allowed ratemax are configurable parameters . A timeout interval of TMODECR controls the decrement of the 4

emission rate. After a minimum time interval of TMODECR seconds the inter packet gap is reduced to

t

Dstart t

Besides the actual rate drop, the changing to INCREASE state is accompanied by an adaptation of gradient of the emission rate.

(2)


R 2l (l + 1) mod Z


R l

with Dstart representing a constant factor. We use Dstart = 0:8 in our simulations. The first NAK or CGN packet reaching the sender leads him to switch to the main phase and enter the INCREASE state. Cycle count variable  ( 2 f0; 1; : : :; Z ? 1g) and the addend
R determining the following rate increment are initialized. The cycle count Z partially determines the gradient of the rate increment after a significant drop and will be explained and discussed in a forthcoming section. The value of
R depends on the timeout value TMODECR for reducing the inter packet gap t and the round trip time r. The determination of r will be discussed in section 3.2.

(5) (6)

The stepwise cutback of the gradient of the emission rate aims at a reduction of the packet losses in consequence of the rate control. So that the overall control traffic is reduced. Gain and cost of the variable gradient are discussed in section 5.2. The cycle of length Z circumvents that the controller reduces its emission rate to the minimum in a steady state model because the gradient approaches zero. One particularity of the INCREASE state not mentioned in figure 3 concerns the direct congestion notification through a DECR
R (3) CGN packet. CGN packets carry the sequence number of the r2 first missing packet. If the sender still keeps this packet in Figure 3 outlines the state diagram for the main rate control his transmission window it checks his scoreboard to obtain the phase of the sender. Whenever the sender receives a CGN or emission rate, the packet was originally sent at. The controller NAK packet and at the same time at least TMOINCR seconds then uses the maximum of this value torig and the one calcuelapsed since the last switch to INCREASE state, the sender en- lated by the usual formula (t 2t) as the new inter packet ters the INCREASE state. With this transition the value t rep- gap. resenting the inter packet gap doubles: t 2t1.

TMO

(

(7)

This enhances the stability of the control circuit. As torig is completely independent from influences of the network or the control circuit, it removes any delays from the control circuit. In this case the rate is no longer determined by the measurements of the round trip time but the observations of the sender. This is actually more than the usual ‘go back N ’. It therefore stabilizes the control circuit and reduces oscillations. If the sender does not receive another CGN or NAK packet and the accordant timeout condition is fulfilled the sender changes to DECREASE state with the next packet scheduler step. DECREASE describes a reduction of t which is equivalent to a rate increment. After a time interval of at least TMODECR seconds the inter packet gap t is reduced to

: loss > ? (n?n k) (4) Islow t : loss  ? (n?n k) with Ifast = 2 and Islow = 1:3. ? denotes a factor which weights the fraction (n ? k)=n. The quantitative loss signal can easily be included in the NAK t

maxftorig ; 2tg

t

1 At this point one may think of varying the direct TCP-analogy. Instead of doubling t, it may be multiplied by a loss dependent factor, so that t f (loss)t. f (loss) is to enable the controller to react to different loss rates differently and therefore be loss tolerant. LTRC [12] basically relies on this mechanism to achieve loss tolerance. In our simulations we studied the following scenario, inspired by the idea to reduce the emission rate by a factor smaller than 2 if the loss is close to a value a (n; k) FEC block code (We always use n and k as the FEC coding constants. An (k; n) erasure code is capable of recovering the original k data packets from any k different packets out of the n coded packets.) could cope with: Ifast t

packets. Our simulations have shown, that determining reasonable values for loss and ? is a difficult task in complex network setups. The amount of loss and the time of detection are not always correlated. There might be receivers which experience low loss rates and are positioned close to the source whereas other receivers positioned far from the source may experience high loss rates but their NAKs are suppressed because a retransmission is already in progress. Furthermore, bursty losses, which are common with drop-tail routers, make the computation of a mean loss rate, e.g. with a low pass filter like an exponential weighted moving average (which works well for estimating the average queue length of a RED router), very difficult. Although the proposed RCR is able to adjust to different loss rates it does not do so in practice. We therefore chose a constant function f (loss) = 2 for all simulations presented in this paper. The simulation results presented show that it is far better to use FEC to implement loss tolerance than to include the tolerance into the dynamic process of rate adaptation at the sender. This is due to the fact that the sender needs to find a rate which suits all the receivers needs, whereas FEC gives each receiver a

t

t 1 +
Rt

(8)

On approximating the non continuous progression of the emission rate (which is a staircase function in reality) by a continuous function, the emission rate has a constant gradient. ‘tolerance interval’ which allows him to recover from loss locally without penetrating the whole system.

5

3.2 The round trip time r

3.3 Receiver

The value used for the packet round trip time r has a significant influence on the overall behavior of the control circuit. It determines the gradient of the emission rate and subsumes buffer effects in the network as well as propagation and processing times. While the definition of the round trip time is straight forward in the unicast case, things are much more complicated in the multicast case because there is no unique round trip time. A close timely relationship between sender and receivers like in TCP does not exist. To determine a value analogously to the round trip time in the unicast case, RCR uses the maximum of the round trip times noticed by the sender r = RTTmax = maxfRTTg. This results in a robust and TCP conformable control circuit. As with TCP, it assures that the effects of a rate increase are noticeable by the sender within one round trip time. Values r < RTTmax have the disadvantage that the sender increases the emission rate before a response of the last rate increase can reach him. This leads the sender to overestimate the available bandwidth and gives rise to large oscillations. By choosing r > RTTmax the RM flow is reduced more than necessary. r = RTTmax already means a discrimination, because r is much larger than the RTT of a TCP connection on the same network path. This results from the fact that the NAK and CGN packets do not take the direct route to the sender but follow the control tree. The computation of RTTmax follows the principles of the NTP algorithm [19] and is similar to the one used in [2]. Due to the lack of positive acknowledgments, RCR uses the NAK and CGN packets to transport the timing information. As the frequency of occurrence of the NAK and CGN packets cannot be compared to that of TCP-ACK packets, the computed value for r contains a significant amount of insecurity. Note that an overestimation of r is less dangerous than an underestimation. This is reflected in the sender’s evaluation of the RTT measures rp . If rp is larger than the actual used one, r is assigned its value. To respond to network dynamics, smaller measures are also taken into account but will lower significance. Again, an exponential weighted moving average is used in this case.

r



rp : r p > r (1 ? $r )rp + $r r : otherwise

Corresponding to the election process at the beginning of each transmission the set of receivers falls into two groups, i.e. nodes that were elected to retransmit (leaders) and nodes that receive only. As retransmissions also may have a severe impact on overall network performance, the leaders also need some kind of rate control.

Figure 4: Congestion control state diagram of a leader. A state diagram of a leader resembles that of the sender. As shown in figure 4 a third state named SET is added to the existing states INCREASE and DECREASE. The CGN packet is sender specific and therefore left out. SET couples the retransmission rate to the packet emission rate of the sender. Each leader permanently evaluates the data packet arrival rate – again as inter packet gap
t(i;i+1) and computes an estimated inter packet gap trmrecv :

trmrecv

(1 ? $)
t(i;i+1) + $trmrecv

($ = 0:2)

(10) New data packets reaching the leader in correct sequence always force the emission rate of the leader to reflect the incomtrmrecv . The parameter of the cycle count l is ing rate: t initialized to 0. When receiving an NAK packet, the leader reduces its retransmission rate to half of the current value. As long as no new data packets are received the rate control of the leader totally resembles that of the sender. In the absence of any reasonable definition of round trip time at the receivers a constant value of one second is used. A receiver has two different options to signalize loss and congestion to his leader. Negative acknowledgements (NAK) report missing packets to the leader and request their retransmission. NAKs simultaneously act as a congestion signal. A NAK packet may contain reports for more than one missing packet. Their structure resembles that of TCP-SACK packets [20] and draws a detailed picture of the receiver’s receive window. Congestion notification packets (CGN) are direct loss signals from a receiver to the sender. A CGN signal decouples congestion

(9)

One important fact to note when using negative acknowledgment packets for the RTT computation is that this setup automatically measures the RTT of the connection suffering from congestion. If for example the congested path is close to the source, the sender receives a congestion signal quite fast and therefore computes a small round trip time. NAK-packets from receivers which noticed the loss later than the one close to the source are suppressed and do not influence r. 6

notification from the retransmission mechanism, as shown in section 2.2. Receivers detect lost packets through gaps in the packet number sequence space. Each entry in the receive window is accompanied by an entry in a scoreboard, which keeps state information like timer and status information for that packet. The scoreboard is checked immediately if a new data packet is received. Otherwise it is consulted in a timely manner. Out of sequence arrival is easily detected by comparing the value of snnext holding the sequence number of the next packet expected to the sequence number snrmact of the packet which just arrived. If snrmact > snnext the scoreboard is checked immediately and the receiver decides whether to react with a CGN/NAK packet or not.

1. Usage of FEC redundancy: By interpreting packet loss as a signal FEC may be used as a high pass filter. The parameters of the FEC block code (n; k) determine the overall behavior of the controller. If the receiver uses FEC to the full extend for being loss tolerant, it tolerates n ? k lost packets per block bi without issuing an NAK or CGN packet. If #bi denotes the number of packets already received for block bi and #act is the packet sequence number relative to the start of the block. The formal condition for #bi reads

#bi  #act ? n + k

(11)

Section 5 shows on the basis of simulations and calculations that the amount of redundancy has to be chosen very carefully to achieve TCP friendliness as well as TCP competitiveness. A full exploitation of the redundancy is not always applicable. Figure 5 gives four examples of filter functions f(#act ). The x-axis shows the number of packets sent by the source, the y-axis the number of packets reaching the receiver. If the number #bi of packet received drops below the threshold f(#act ), the receiver stops tolerating losses and requests retransmissions via an NAK. The curves shown with bold lines describe the situation if one resigns to use FEC (curve A) or uses FEC to the maximum (curve D). Possible filter functions belong to the gray area. Curves B and C are examples for alternative functions. Functions similar to B are applicable in scenarios with a high level of redundancy (n ? k)=n deployed. Only part of the redundancy is used for congestion control purposes in order to preserve TCP fairness. A receiver can react in two possible ways: If #bi < f(#act) and #bi > #act?n+k the receiver only issues a CGN packet, otherwise it has to send an NAK packet.

Figure 5: FEC fueled filter capabilities at the receiver. Leaders that receive an NAK packet consult their scoreboard to find out whether the requested packet resides in their receive window, an NAK for this packet has already been sent to their leader or timer constraints exist, which refuse a reaction to the ACK. Assuming that no timer restrictions exist, a leader forwards the ACK to his leader if it does not have the requested packet(s) in memory or in the other case schedules a retransmission for the packet(s). Leaders never show any response to a CGN packet. CGN packets are always forwarded towards the sender – or suppressed by timers. The frequently mentioned timers assure that a minimum time interval TMO exists between two retransmissions for the same packet, two CGN, or two NAK packets. The decision, whether to send an NAK or CGN packet is based on two criteria:

The function types represented by curve C ( f(#act ) = #act nk ) are more sensitive to loss when they start filling a FEC block than at the end of a block. This results in a good responsiveness of the controller and is suggested for large blocks with a moderate level of redundancy (around 10%).

2. Available buffer space: NAK and CGN transmission relies on UDP as transport protocol and is therefore not reliable. Hence the size of the receive window determines the amount of time after which a receiver decides to repeat the NAK or CGN packet if it does get any response. Choosing a short time interval bears the danger of NAK implosion, choosing a long time interval may drive the receiver out of buffer space. In the absence of a timely relationship between source and receiver the receiver uses 7

control circuit that uses multiplicative decrease and linear increase. Further on the model assumes RED gateways. The assumption of uncorrelated packet loss does not hold for drop tail gateways. The model presented in this paper generalizes those in [23] and [24]. It contains their results as a special cases. The model is strictly targeted at the analysis of a steady state and does not try to be correct in a microscopic view. The model assumes that both connections in figure 6 use the multiplicative decrease, linear increase algorithm for their rate control. The gateways detect congestion if the sum of the incoming rates exceeds the maximum transfer rate of the gateway. The outgoing link of gateway 2 determines the maximum transfer rate in the example discussed here. When detecting congestion, the gateway drops a packet of one of the connections. This makes the referring connection drop its rate to half of the current value. The TCP connection cannot rely on FEC to recover from the loss and therefore has to react to the loss situation and reduce the size of the congestion window cwnd to half of its original size. The RM connection only has to react to a loss with a certain probability, depending on the level of redundancy. The packet drop probability of a connection is proportional to the fraction of the available bandwidth the corresponding connection receives.

a static timeout value which takes into account the size of the local receive window. If a receiver i has a receive window of size #Bi FEC blocks and tmin denotes the minimal inter packet gap used by the sender, the minimal time interval TMOCGN;NAK is chosen as follows:

TMOCGN;NAK = a1 (1 + rand(0:05)) #Bi n tmin

(12)

where a denotes the largest number of lost NAK or CGN packets tolerable for the receiver and rand(0:05) denotes a random number 2] ? 0:05; 0:05[. RCR uses a value of a = 12. In the best case the receiver tolerates the loss of 12 NAK/CGN packets in sequence. Randomization of TMOCGN;NAK lowers synchronization effects among the receivers [21], [18]. Note that in RM environments that do implement a reliable NAK transport mechanism like [22] a timely repetition of NAKs is superfluous. But the buffer management still remains a problem. Hence, there still is a need for a mechanism to signalize congestion (like the CGN packets). Additionally, CGN packets are also necessary for filter functions comparable to curve B in figure 5.

4 A steady state model of the controller 4.1 Simplifying assumptions For simplicity we impose the same assumptions as in [23] and [24]. Let W be the maximum throughput of the gateway, measured in bit/s. The rate with which packets are dropped at the gateway is approximated by an independent poisson process. Hence, the packet losses are uncorrelated and equally distributed. TCP denotes the average percentage of the total bandwidth the TCP connection consumes, when a packet is dropped. So pTCP = TCP is the probability that a packet of the TCP connection is dropped [23]. The assumptions are not realistic but have been shown to be able to describe the behavior of TCP-SACK and (partially) TCP-Reno in situations where congestion is the only source of packet loss, TCP timeouts do not occur and the system is in a steady state [24]. For a direct comparison, the window controlled emission rate of TCP has to be translated into an equivalent emission rate, represented by the inter packet gap. The emission rate is considered to be continuous. Let ri be the round trip time of a TCP connection i. This connection increases its window by roughly one packet of size s (in bit) per round trip time. For two TCP connections competing for bandwidth at one gateway it follows [23]

Figure 6: Simple test network used for analytical controller analysis. In this section a model of the rate controller is developped and mapped onto a very simple network scenario with only two connections: One TCP and one RM connection, both competing for bandwidth on a single link. The setup is presented in figure 6. The goal is to analyze the FEC influence on the overall throughput as well as on parallel TCP connections. As the RM connection is also point to point in this example, the later results apply to TCP connections as well. This model yields an insight into the effects of introducing redundancy and a cycle (for adjusting the gradient of the emission rate) into a rate 8

the event space Ei to the interval [0; 1]. E.g. With i = 3 a possi(13) ble event vector could be e = (; ; ) which means that first, 1 the TCP connection registers a packet drop and then the RM 2 connection registers two packet drops in sequence. For i = 3 This result may be directly mapped to the case discussed here, we obtain the event space E3 = f(; ; ); (; ; ); (; ; ); because RM and TCP connection are following exactly the (; ; ); (; ; ); (; ; ); (; ; ); (; ; )g. same algorithm if we apply the simplifying assumptions and The corresponding interation formulas for each  (e) and i no FEC is used. i (e), which are deduced in the next section, are applied according to the event sequence defined by e. (0) = 0 is randomly chosen from the interval ]0; 1[. pTCP = TCP We obtain the expected value E() for  as

1 = 1 +1 r r

= 1 + 1r = r = pnoFEC TCP RM

For equal round trip times rTCP ability is 1=2.

1 = ilim i) !1 E( X Pr(e)i(e) = ilim !1

(14)

= rRM the packet loss prob-

e2Ei

(15)

with Pr(e) = pn qi?n denoting the probabilility of event e. n signifies the number of significant packet drops the TCP connection registers: n = #fj j ej = g.

4.2 Generalized model The generalized model introduces FEC redundancy and the cycle count Z to adjust the gradient of the emission rate after each rate drop. We now give up the assumption, that the connections react on every packet loss to investigate the influence of FEC on a flow and congestion controller similar to TCP. The other assuptions made in section 4.1 still hold. Especially, we still employ poisson distributed losses but change the way we react to losses. Because of the introduction of the cycle count into the control process, the system is no longer stateless and can no longer be interpreted as a markov process. The markovian property that the present state must be independent of the past states is violated. Again let W be the bandwidth of the bottleneck link in bit/s and i; i the percentage of bandwidth occupied by TCP and the RM connection respectively, when a significant packet drop happens. The ith significant packet drop takes place at time ti . i and i are only defined for the discrete times ti. At time ti we have i + i = 1. A packet drop is said to be significant if either the TCP or the RM connection react to the drop. We use “” to represent the event that the TCP connection registers a significant packet loss and “” to represent the event that the RM connection registers a significant packet loss. Let the probabilities for the two events be p and q: Pr() = p; Pr() = q, with p + q = 1. Further, let
ti+1 = ti+1 ? ti be the time elapsed between two packet drops. 1 and 1 denote the expected values of  and for the steady state, i.e. for i ! 1. Each i (and analogously i ) is defined by a corresponding event vector e = (e1 ; : : :; ei) out of the event space Ei. Ei = f(e1 ; : : :; ei) j ej 2 f; gg. i(e) defines a function mapping

Figures 8 and 7 illustrate the meaning of different variables defined above. Figure 7 shows the situation, where TCP connection reacts to lost packets two times in a row. In figure 8 the RM connection is the first one to detect a lost packet. After this the TCP connection reacts to loss and finally it is the turn of the RM connection again. In contrast to figure 7 a cycle count Z different from zero was chosen ( Z = 1): At time ti the gradient of the emission rate is reduced to half of the original value and is set back to this value at time ti+2 . Z denotes the cycle count with  = 0; 1; : : :(Z ? 1). After Z ? 1 bisections of the emission rate the initial value of 1=r2 is adopted again. After Z ? 1 steps the gradient reaches its minimum s=(2Z ?1r)2 .

Figure 7: Example of the rate control of the sender for a reaction series f TCP, TCPg and Z = 0.

9

With a relationship between
 and
t

W
 =
t rs2

(20)

1

we obtain the expression determining the iteration step from i to i+1 for the condition that the TCP connection drops its rate at ti

0

1 A

(i+1); = i 21 @1 + 1 r 1 + 2 r

Figure 8: Example of the rate control of the sender for a reaction series f RM, TCP, RM g and Z = 1.



2

2 1

2 2

= S  i

0  1@ with S  = 2 1 +

1

1 + 2 r r

Other interesting expected values include the transferred include the amount of data D during time T . The bandwidth of the TCP connection in steady state results to W = D=T . Inserting this result, we obtain for
t The expected values for throughput and duration of the TCP connection are computed with a similar formular, as both are
t = Ws i+1 2 1 1 functions of i. In steady state we obtain: +

D1 = E(D) = ilim !1 T1 = E(T ) = ilim !1

W1 = DT 1 1

X

e2Ei

X

e2Ei

r12

2 1

2

1(21) A

2 2

(22)

22 r22

From this result we can derive the transferred amount of data

Pr(e)D(i (e))

(16)

Pr(e)T (i (e))

(17) figures 7 and 8.

D;i for the TCP connection during
t with respect to i+1 . D;i is represented by the gray area in the examples shown in DTCP

(18)

;i

=

Zt

i+1

ti

W(t)dt

= i+1; W
t ? 21
t2 rs2

4.3 Iteration steps for ; ; D; T

 1r 2 2 2 3+2 = 2i+1; W2sr1   2 r22 r

The determination of the interation formulas falls into two parts, dependent on the connection which has to react on a significant packet loss.



2+

1

2

(23)

1

2 r2

Case A: The TCP connection registers a significant packet The transferred amount of data for the RM connection is obloss (Event “”) If the TCP connection registers a packet tained analogously loss, it reduces its actually consumed bandwidth from Wi to Wi=2. The RM connection keeps its part W i of the bandti+1 width. During the time
t between two consecutive signifiRM = D W (t)dt ;i cant packet drops at times ti and ti+1 the TCP connection ret i  i ceives an additional part of W
 = W(i+1 ? 2 ) of the overall bandwidth W , whereas the RM connection receives an = W( i+12; + i )
t additional part of W
= W( i+1 ? i ). The index  of
t denotes the actual value of the cycle count. At time ti+1 the following equation holds for the bandwidth of W 2 i+1;q; 2 ? i+1;q; ) ? 1+2i+1?1;q; 2 2 2 r1 the congested link: 1+

Z

1 CA

0 B@

W 2i + W
 + W i + W
= W

=

(19) 10

22

2?22 )

2s( r + r 1

2

r22

=

Case B: The RM connection registers a significant packet loss (Event “”) The computations here are completely similar to the previous case. Here the resulting bandwidth equation is

Wi + W
 + W 2i + W
= W

=

(24)

Note that the values of
 and
are different from those in the previous paragraph.





(i+1); = 1? S  +i S  with

0  1@ S = 1 + 2


t = Ws (i+1); 1 1 2 r + 2 r Z ti RM = W (t)dt D;i 2 1

2

1

1+ 2rr

2 2 2 2 1

2 + r

0 W 2(1 ? (i+1); ) B @1 + (i+1); ? 2(11+? 1 2 2 2

e2Ei+1 ei+1 =

X



e2Ei+1 ei+1 =

X

e^2Ei



q Pr(^e) (1? S + S i(^e)) 



Pr(^e) S +q(1? S ) +

X

e^2Ei

Pr(^e)i(^e) 





4.3.1 Evaluation of E() with the assumption Z





xn+1 = (p S +q S )xn + q(1? S ) which converges to the fixed point xf of the function f(x) =      (p S +q S )x + q(1? S ) if f 0 (xf ) < 1; f 0 (x) = p S +q S . For xf we obtain

2

(i+1); 1 2 22 r2 1+ r12

(25)

This iteration defines a Banach series

2

W(t)dt

2 1



p Pr(^e) S i(^e) +



1

2s( r1 + 2 r?  )

Pr(e) i+1(e)

= p S E(i ) + q(1? S ) + q S E(i )    = (p S +q S ) E(i ) + q(1? S ) 1

=

e2Ei+1 ei+1 =



2 2



ti

X

qS

 2 r 2 2  2 3 + 2 = (1 ? (i+1) )2 W (22s r2)   r 2 2 2r i+1

X



ti

Zt

Pr(e) i+1(e) +

e2Ei+1 ei+1 =

= pS

1 A

+1

TCP = D;i

X



xf = q(1? S )  1 ? p S ?q S

1 )C A

(26)

The series xn converges as

2 2 RM + (2 ? p)rTCP < 1; p 2]0; 1[ f 0 (xf ) = (p + 1)r 2r2 + 2r2 RM

TCP

For the expected value of  we obtain with q

=0

= 1 ? p:

 With the assumption of a stateless process (Z = 0) we can p)(1? S ) evaluate equation 15 which defines the expected value for . 1 = (1 ? (27)    1? S +p(S ? S ) We will show that in this special case the bandwidth will be distributed according to equation 13 which was derived in [23]. If we utilize 1 = p and solve equation 27 for 1 we finally To compute the expected value E() we first derive an iter- obtain ation formula for E(i+1) as a function of E(i+1). With a given event vector e = (e1 ; : : :; ei+1 ) 2 Ei+1 we denote 1 = 1 + 1rTCP (28) the corresponding event vector formed out of the first i comrRM ponents as e^: e^ = (e1 ; : : :; ei) 2 Ei . as the only resonable solution for the quadratic equation 27. This result proves that our model is consistent with those in [23] and [17] and contains their results for the special case E(i+1) = Pr(e) i+1(e) Z = 0. e2Ei+1

X

11

In the absence of an interaction free representation for equation connection limz!1 E(TCP). Figure 9 shows how to derive 15 with Z 6= 0 we realized the iteration formulas as functional the corresponding formula. programs and evaluated them with Mathematica [25]. It is now possible to describe the effects of varying drop probabilities but we still lack a description of the FEC influences. The next section links the actual influence of FEC usage to the model. The derived relationship between p and (n; k) assumes a complete usage of the redundancy n ? k for the purposes of congestion control – This corresponds to curve D in figure 5.

4.4 Relationship between packet loss probability p and the parameters (n; k ) of the block code Let V be the total number of losses at the gateway 2. Again we assume a setup like in figure 6 with two connections, one TCP and one RM competing for bandwidth on a congested link. The total number of losses is divided into the number lost packets belonging to the TCP connection VTCP and those belonging to the RM connection VRM . It is V = VTCP + VRM . For the packet loss probability of the TCP connection we obtain pnoFEC = VTCP =V and in analogy 1 ? pnoFEC = VRM =V . If FEC is used for the flow control of the RM connection, Figure 9: Stepwise deduction of pFEC . Each node represents n?k VRM of the original VRM packet losses are no longer siga packet drop. The left branch denotes  events, the right n nificant. Note that ignoring packet loss leads to further losses, branch  events. Due to packet level redundancy the fraction since neither of the two connections reduces its emission rate. b(n?k)?1 VRM of all packet drops is not significant. b(n?k) V k Besides, the relative redundancy n? n , the absolute size of n and k is also significant for the process. The absolute size of n determines the amount of data D(
t) being transferred between two subsequent packet drops. D(
t) is represented by pFEC = zlim the gray area the example in figure 8. !1 E(TCP)



D(
t) = 1 +21=2
t r2 = 43 1 1 2s 2 r2  3 = 8 1s

(29)

Dividing the number of packets D(
t)=s by n results in the number of transferred FEC blocks b in the time interval
t.

3 21 r2 c + 1 b = b 8n s2

(30)

If the assumption of poisson distributed losses holds (section 5 shows that this is a real simplification), b(n ? k) ? 1 of the b(n ? k) packet losses are tolerable during
t. Now we can calculate the loss probability of the TCP connection, which equals the expected value for a reaction of the TCP 12

VTCP + b(n ? k) ? 1 VRM VTCP = zlim !1 V b(n ? k) V V !  b(n ? k) ? 1 V 2 V TCP RM + b(n ? k) V V + ::: z  b(n ? k) ? 1 V i?1! X V RM TCP = zlim !1 V i=1 b(n ? k) V 1   = VTCP b ( n ? V 1 ? b(n?k)k?) 1 VV 1 1  = (31) 1 ?1 1 + b(n?k) p RM

noFEC

The value of pnoFEC is determined by the model chosen and the RTT of the connections. Figure 10 shows p as a function of k=n for different cycle counts. Especially the area with relatively small FEC redundancy (k=n > 0:9) is very interesting: Minor changes of the redundancy level result in major changes

5.1 FEC-Redundancy Flow control in reliable multicast faces the requirement to be TCP-fair and competitive at a time. TCP fairness may be achieved with a multiplicative decrease, linear increase control algorithm which is similar to TCP congestion control. FEC permits an RM controller to be more competitive. Besides, FEC reduces the control traffic of the protocol. This section analyzes the influence of FEC onto the control circuit for the setup given in figure 6. As both connections are point to point, the results apply to a TCP connection using redundancy as well. Figure 10: pFEC as a function of the coding parameters with k = 30.

of the packet drop probability of the TCP connection. The RM connection already becomes less resilient for small values n?k and reduces TCP throughput significantly. The reason for this behavior can also be derived from the TCP control loop, which has a strong sensitivity to a change of the packet loss probability. Jacobson shows in [8] that for given round trip time r, Figure 11: Throughput of the TCP flow as a function of the FEC window size w the throughput is a function X(p) of the packet coding parameters and the FEC block size used. loss probability p: X(p) = r(1+wpw) . Therefore a small loss percentage suffices to reduce throughput significantly in case of large window sizes.

4.5 Extending the model to n competing flows In appendix A we extend the model from 2 to n competing flows at one gateway, derive the corresponding formulas and prove the convergence for steady state. Besides, we present various results of an example setup with 10 and 20 competing flows.

5 Heuristic analysis

Figure 12: Throughput of the RM flow as a function of the FEC coding parameters and the FEC block size used.

This section compares the results of the model with those obtained from a prototypical implementation of the congestion controller. We extended the network simulator ns-2 [13] to support our reliable multicast framework. The implementation was done in C++ and OTcl. First we analyze the influence of different parameters on the overall behavior of the controller. Then we present some simulation results of a bigger network with randomly distributed RM and TCP connections.

Figures 11 and 12 visualize the influence of the redundancy on the throughput of the two connections. The dashed lines describe the theoretic results we obtained by using the formulas developped in the previous section. The iteration formulas were evaluated with Mathematica using i = 20 iteration steps to compute the expected values for , troughput etc. Triangles show the simulation results. For the sake of clearness, the data points are connected by lines. The values are mean values over

13

a simulation time of 400 seconds. The packet size was chosen to be 536 bytes for both connections, the gateway used RED queue management with a queuelength of 30 packets. The cycle number was Z = 0, i.e. the process was stateless. n ? k can be interpreted as a measure for the TCP disconformity. In a setup where both connections receive the same throughput, a small amount of redundancy suffices to discriminate the TCP connection. 10%-15% redundancy are enough to reduce the throughput of TCP to the half of the original value. The very good responsiveness bears dangers when higher levels of redundancy are deployed. The less resilient the RM connection is, the more prejusticed parallel TCP connections are. Therefore the coding parameters have to be chosen very carefully. Otherwise one may break the control circuit of TCP. If a high level of redundancy is used to minimize control traffic and retransmissions, one should not use the full redundancy for the control circuit but deploy a filter function utilizing only a part of the redundancy (like curve B in figure 5). The larger the FEC blocks are, the more effective is the use of redundancy [26]. This rule also holds for the situation described here as well, as figure 11 shows. If the block length n grows while the relative level of redundancy is kept constant, the number of recoverable packets n ? k which are crucial for the behavior of the controller grows as well. So the controller becomes even less sensitive to packet loss for growing n if k=n is kept constant. In order to prevent this, we suggest choosing relatively small n. This is also motivated by the current software codecs based on linear block codes. They perform well for k up to k = 32 ([27], [28]).

Figure 13: Blocking effects under the assumption of poission distributed drops.

Figure 14: Measures in which the number of cycles Z (0,1,2,3) and the FEC parameters (k; n) were changed.

parameters k=n, with k = 100 and cycle count Z = 1. n varied between 100 and 130. Measured values are again represented by triangles whereas the solid continuous curve is the result of the analytic model. The measured values are usually slightly bigger than the calculated ones. This is partly due to the assumption of poisson distributed losses. The nature of statistical fluctuation is such that a random pattern of losses usually exhibits some clustering of the losses - some blocks contain more than the average number of losses, some blocks contain less. So it happens that more than n ? k losses fall into one block. This reduces the amount of packets actually transferred between two significant losses. In figure 13 we varied the amount of data (with FEC blocks as unit) being transferred between two significant losses. We used multiples of the number of blocks b (from equations 30 and 31) as parameter. The results are shown with dashed lines in figure 13. Most of the measured values fall into the region between the curves for b and b=2. As we are more interested in the phenomena involved in the process, we will continue to use the plain poisson model in the following calculations. But one has to be aware of the fact, that this choice is a major simplification and is useless for setups of greater complexity [29].

5.2 Cycle count Z

In the previous sections, we have introduced the cycle count for reducing control traffic. This section discusses advantages Figures 11 and 12 indicate that the model developed in the pre- and disadvantages of the approach. The cycle count controls vious section is well suited for a heuristic analysis of the con- the gradient of the emission rate after a drop. Beginning with troller. A small number of iterations already suffices to de- an increase of one packet per RTT (which equals s=r2 ) the grascribe the effects qualitatively and quantitatively. The limita- dient of the emission rate is halved after each significant loss tions of the model become obvious in figure 13. It shows the until (after Z ? 1 steps) it reaches s=(2Z ?1r)2 . The next drop throughput of the TCP connection as a function of the coding resets the gradient to s=r2 again. 14

throughput. Again, the effect is most obvious for small n ? k. Transmissions, which are not targeted at the maximum possible speed may choose to use a cycle. The cycle count increases the fairness of the protocol with respect to the other connections operating in parallel but lowers the throughput of the connection per time interval.

5.3 A quantitative measure of the TCP-fairness

Figure 15: Influence of the cycle number Z meters on the transmissions duration.

To complete the discussion of the tiny network setup this section takes a short glance at the quantitative fairness of the RM and the FEC para- protocol with TCP. Besides min-max fairness [30], the fairness n n index F = ( i=1 xi)2 =(n i=1 x2i ) with xi being the resource usage of i ([31], [32]) is frequently used [23]. It allows a quantitative analysis of fairness and therefore meets our needs.

P

P

Figure 16: Influence of the cycle number Z and the FEC parameters on the mean bandwidth consumed by the RM flow Figure 17: Fairness index F as a function of the packet drop (k = 30). probability.

The cycle compromises between a good adaptation to the present bandwidth situation and the need to reduce control traffic. The simulation results in figure 14 match the expectations: The bigger Z the less throughput the TCP connection receives and the more throughput the RM connection receives. Already a small percentage of redundancy is able to compensate for the cycle count effects. Figure 15 and 16 again obtained by an evaluation of the formulas developped in section 4 with Mathematica, give insight into the possible benefits. Figure 15 shows the relative prolongation for cycle counts 1; 2 and 3 as a function of the relative redundancy k=n (TZ denotes the expected value for a time interval with a fixed amount of significant losses). Cycle counts reduce the number of significant losses per time interval. As FEC and the cycle count are expected to work hand in hand, the effect of the cycle count is maximized for small n ? k. Not surprisingly, this advantage is not for free: Figure 16 shows up to what percentage the cycle count reduces 15

Figure 18: Fairness index F as a function of the relative level of redundancy k=n. We define the expected value of the mean bandwidth W as the resource to be distributed fairly. In the absence of any cycle count or redundancy F is maximized for x1 = x2 = W =2 in

our simple test setup, with F = 1. We obtain the same result for min-max fairness. To evaluate the effect of redundancy and cycle count quantitatively figure 17 shows F as a function of the packet loss probability p of the TCP connection for different cycle counts. Depending on the cycle count, F is maximized for p = pnoFEC;Z = 1;Z . If we include the results of figure 18, which shows F as a function of the relative redundancy k=n, we notice that the operating point of the control circuit is usually positioned close to, or right of the maximum (because p > 0:5).

ulation results focus on a single backbone link (the link which connects the ‘backbone’ nodes 0-24), we have chosen no internetwork redundancy at WAN level for the test network. Besides the multicast traffic, which originates at node 109 and is targeted towards 20 destinations, we randomly created 100 TCP connections connecting 100 randomly chosen ftp-clients to a set of 20 randomly chosen ‘servers’. This random setup resulted in a total number of 38 flows on link 0-24. The routers use RED queue management with a queue size of just 10 packets.

5.4 Simulation of a bigger network The previous sections focussed on the behavior of the protocol in a simple unicast environment to distill the central features. This section puts attention to bigger networks, where analytic modeling becomes impossible and simulation is the only means of evaluation.

Figure 20: FEC influence on throughput in medium sized networks. Figure 20 shows the mean bandwidth that each of the 38 flows receives for a simulation time of 400 seconds. Nine different bars for each flow represent the throughput of the flow for a different level of redundancy used by the server. The leftmost bar shows the situation in the network without any reliable multicast data transfer, whereas the rightmost bar reflects the situation for a multicast data transfer with (n; k) = (30; 37). The measured values for the RM connection are shown with wider bars in order to emphasize them. The outcome of the simple model with just two connections qualitatively holds true for this larger setup: The higher the level of redundancy, the larger the average throughput of the RM connection. The effect on the TCP connections is no longer easy to quantify. The same is true for the cycle count. As shown in figure 21 the throughput of the RM connection is nearly independent of the value of Z . As a consequence one Figure 19: Automatically generated, medium sized test netcan draw the conclusion that the influence of the redundancy work. Backbone link 24-0 was traced. Node 109 was set to be is much bigger than that of the cycle count – as long as a conthe RM source. Nodes 34, 41, 71, 77, 86, 87, 89, 90, 91, 95, 96, stant gradient (leading to a linear rate increment) is used after 97, 99, 100, 104, 105, 114, 117, 124, 126 are RM receivers. each rate drop. This fact justifies the design decision described earlier. It turns out that it is more suitable to fuel the rate conWe used the topology generation tool tiers [33] to create a sim- troller by FEC than to actually tune the control process by difple medium sized test network, shown in figure 19. As the sim16

References [1] Towsley D., Kurose J., Pingali S., “A Comparison of Sender-Initiated and Receiver-Initiated Reliable Multicast Protocols,” IEEE Journal on Selected Areas in Communication, vol. 15, pp. 398–406, April 1997. [2] Floyd S., Jacobson V., McCanne S., Liu C.G., Zhang L., “A Reliable Multicast Framework for Light-weight Sessions and Application Level Framing,” in Proceedings of the ACM SIGCOMM Conference ’95, pp. 342–356, October 1995. Figure 21: Influence of FEC and cycle number in the medium size test network. Again link 24-0 was traced.

[3] Grossglauser M., “Optimal Deterministic Timeouts for Reliable Scalable Multicast,” IEEE Journal on Selected Areas in Communication, vol. 15, pp. 422–433, April 1997.

ferent parameters – either through a loss tolerant adaptation of the rate decrement (by a factor different from 2) as response to packet loss or a cycle count to modify the increment.

[4] Paul S., Sabnani K., Lin J., Bhattacharyya S., “Reliable Multicast Protocol (RMTP),” IEEE Journal on Selected Areas in Communication, vol. 15, pp. 407–421, April 1997.

6 Conclusion and future work

[5] Yavatkar R., Griffioen J., Sudan M., “A Reliable Dissemination Protocol for Interactive Collaborative Applications,” in Proceedings of ACM Multimedia ’95, pp. 333– 344, 1995.

This paper describes a new TCP conformant congestion control scheme primarily designed for reliable multicast data transfers. It is the first to integrate TCP-friendliness and TCP[6] Whetten B., Montgomery T., Kaplan S., “A High Perforcompetitiveness. mance Totally Ordered Multicast Protocol.” Theory and TCP shows up to be a hard competitor for bandwidth, as rePractice in Distributed Systems, Springer Verlag LNCS liable multicast transfers usually suffer from high packet loss 938. probabilities and large delays in their control loop. The use of packet level redundancy (FEC) within the rate controller makes [7] Vicisano L., Crowcroft J., “One to Many Reliable Bulkit TCP-competitive. It reduces the mentioned disadvantages as Data Transfer in the MBone,” in Proceedings of the Thrid shown by the simulations and analytical results in this paper. International Workshop on High Performance Protocol TCP-friendliness is achieved by a TCP conformant rate conArchitectures HIPPARCH, Uppsala, Sweden, June 1997. troller interpreting packet loss as a congestion signal. Besides the design of the controller, the primary focus of this [8] Jacobson V., “Issues in Gigabit IP-Networking.” Tutorial presented at INET 91, Copenhagen, Denmark, June 1991. paper is the analysis of the FEC introduction to the control loop. Therefore we developed a new, extended model for [9] Yajnik M., Kurose J., Towsley D., “Packet Loss Correlaflows with a congestion control scheme simliar to TCP. As tion in the MBone Multicast Network,” in Proceedings of shown, already moderate levels of redundancy (up to 10% the IEEE Global Internet Conference, London, Novemwith FEC-block sizes of about 30 packets) may significantly ber 1996. change the bandwidth distributions among competiting flows. This raises the question how to detect the level of redundancy [10] Handley M., “An Examination of MBone Performance.” USC/ISI Research Report: ISI/RR-97-450, January that compromises best between TCP-friendliness and TCP1997. competitiveness. Further more, one may discuss the idea of dynamically adapting the loss filter function of the receivers as [11] Sano T., Shiroshita T., Takahashi O., Yamashita M., well as the level of redundancy of the sender to the needs of the “Monitoring-Based Flow Control for Reliable Multicast data transfer and those of the network. The necessary classifiProtocols and its evaluation,” in Proceedings of the IEEE cation of the controller needs further analysis in complex netInternational Performance, Computing and Communicawork setups as well as for other reliable (multicast) protocols. tion Conference (IPCCC ’97), February 1997. 17

[12] Montgomery T., “A Loss Tolerant Rate Controller for [25] Wolfram S., The Mathematica Book. Cambridge UniverReliable Multicast,” tech. rep., West Virginia University, sity Press, 1996. NASA-IVV-97-011, August 1997. [26] Blahut R.E., Theory and practice of error control codes. [13] “UCB/LBNL Network Simulator ns version 2.” Addison-Wesly Publishing Company Inc., 1983. http://www-mash.cs.berkeley.edu/ns, 1997. [27] Rizzo L., “Effective Erasure Codes for Reliable Com[14] Brockners F., “Bulk multicast data transfer – toputer Communication Protocols,” in Computer Commuwards the integration of FEC and ARQ using a nications Review, April 1997. lightweight feedback control tree,” tech. rep., University of Cologne, Center for Parallel Computing, [28] Rizzo L., “On the feasibility of software FEC,” Tech. Rep. LR-970131, DEIT, 1997. TR97-279, July 1997. available via http://www.zpr.unikoeln.de/people/frank/doc/summer.ps.gz. [29] Paxson V., Floyd S., “Wide-Area Traffic: The Failure of Poisson Modeling,” ACM Computer Communication Re[15] Floyd S., Fall K., “Promoting the Use of End-to-End view, vol. 24, Oct. 1994. SIGCOMM ’94 Symposium. Congestion Control in the Internet.” Submitted to IEEE/ACM Transactions on Networking, February [30] Bertsekas D., Gallager R., Data Networks. Prentice-Hall 1998. International, Inc., 1992. [16] Jacobson V., “Congestion Avoidance and Control,” Com[31] Jain R., Chiu D., Hawe W., “A Quantitative Measure puter Communication Review, vol. 18, pp. 314–329, Auof Fairness and Discrimination for Ressource Allocation gust 1988. in Shared Computer System,” tech. rep., DEC DEC-TR301, September 1984. [17] Mahdavi J., Floyd S., “TCP-Friendly Unicast Rate-Based Flow Control.” Technical note sent to the end2end[32] Jain R., “Fairness: How to Measure Quantiatively?.” interest mailing list, January 1997. ATM Forum Document 94-0881, September 1994. [18] Rejaie R., Handley M., Estrin D., “RAP: An End-to-end Rate-based Congestion Control Mechanism for Realtime [33] Donar B.D., “A Better Model for Gernerating Test Networks,” in Proceedings of IEEE Global TelecommunicaStreams in the Internet,” in submitted to ICNP 98, 1998. tions Conference GLOBECOM 96, London, November [19] Mills D.L., “Network Time Protocol (Version 3).” IETF 1996. RFC (Request For Comments) 1305, 3 1992. [20] Mathis M., Mahdavi J., Floyd S., Romanov A., “TCP Selective Acknowledgment Options.” IETF RFC (Request For Comments) 2018, October 1996. [21] Floyd S., Jacobson V., “Traffic phase effects in packetswitched gateways,” Computer Communications Review, vol. 21, pp. 26–42, Apr. 1991. [22] Speakman T., Farinacci D., Lin S., Tweedly A., “PGM Reliable Transport Protocol Specification.” IETF Internet-Draft draft-speakman-pgm-spec-01.txt, January 1998. [23] Floyd S., “Connections with Multiple Congested Gateways in Packet-Switched Networks, Part 1: One-way Traffic,” Computer Communications Review, vol. 21, Oct. 1991. [24] Mathis M., Semke J., Mahdavi J., Ott T., “The Macroscopic Behavior of the TCP Congestion Avoidance Algorithm,” Computer Communication Review, vol. 27, July 1997. 18

A Extended model for n competing flows This section extends the model described in section 4.2 from 2 competing flows at one gateway to n competing flows at a single gateway. Again a state space exploration is used to compute the share of the bandwidth of each flow. vi;j (i = 1 : : :n) denotes the percentage of bandwidth occupied by flow i when a significant packet drop happens.n The j th significant packet drop takes place at time tj . The vi;j are only defined for the discrete times tj . At time tj we have i=1 vi;j = 1. We use “i ” to represent the event that the connection i registers a significant packet drop. Further, let the probabilities for an event i be pi: Pr(i) = pi . Like in section 4.2 we use
tj +1 = tj +1 ? tj to describe the elapsed time between two consecutive packet drops. vi denotes the expected value of vi;j for steady state, i.e. for i ! 1. Like in the case with only two flows, each vi;j is defined by a corresponding event vector e = (e1 ; : : :; ej ) out of the event space Ej . Ej = f(e1 ; : : :; ej )jek 2 f1; : : :; ngg. vi;j (e) defines a function mapping the event space Ej to the interval [0; 1]. Here we obtain the expected value E(vi ) for vi as

P

vi = jlim E(v ) = lim !1 i;j j !1 A.1

Iteration steps for v

X

e2Ej

Pr(e)vi;j (e)

(32)

i

As all connections i follow the same model we develop the iteration formulas for one connection i = k. For the sake of simplicity we resign to use a cycle count. Case A: Event k - Connection k registers a significant packet loss. If connection k registers a significant packet loss, it reduces its actually consumed bandwidth from Wvk;j to Wvk;j =2. All the other connections keep their part of the bandwidth Wvi;j for i = 1 : : :n; i 6= k. During time
t between twovconsecutive significant packet drops at times tj and tj +1 connection k receives an additional part of W
vk = W(vk;j +1 ? k;j 2 ) of the overall bandwidth W , whereas each other connection i receives W
vi = W(vi;j +1 ? vi;j ). For the bandwidth of the congested link we obtain at time tj +1

W vk;j 2 + W
vk +

n X i=1 i6=k

(Wvi;j + W
vi) = W

(33)

With a relationship between
vk and
t

W
vk =
t rs2

(34)

k

we obtain the expression determining the iteration step from vk;j to vk;j +1 for the condition that the connection k drops its rate at tj

1

0

vk;j +1

= vk;j 12 @1 + Pn1 rk A i=1 ri = Sk vk;j 0 1 Sk = 12 @1 + Pn1 rk A i=1 r 2

2

2

2

i

Case B: Event i, i 6= k

We use v^j to describe the ‘virtual’ connection formed out of all the connection except vk;j : 19

(35) (36)

v^j =

n X i=1 i6=k

vi;j = 1 ? vk;j

(37)

For the connection i which registers the significant packet drop we can again use the relationship between
ti and
vj .

vi;j +1 = v2i;j + ci
ti

 s Pn

= risW i = 1 : : :n. c^ is defined analogously as c^ = W

with ci

2

v^j +1

i=1 i6=k

 1 = Pn c . i r 6

(38)

i=1 i=k

2

i

n X v i;j = 2 + ci
ti + (vk;j +
ti ck ) k =1

k6=i

n n X X = ? vi;j + v +
t cl i 2 l l;j l l6 k l6 k v i;j = v^j ? 2 +
tic^ If we insert
ti = c1k (vk;j +1 ? vk;j ) and v^j = 1 ? vk;j we obtain for vk;j +1 vk;j +1 = vk;j + vi;j 12 1 c^ = vk;j + vi;j Tk 1 + ck =1 =

1 1 2 1+ cc^k

=

=

(40)

1 rk2 2 Pni=1 r12 ; Sk i

vk;j +1 = A.2

(39)

1

= 12 + Tk . As iteration formula we finally obtain for vk;j +1: with Tk

=1 =



vk;j Sk : k vk;j + vi;j Tk : i ; i 6= k

(41)

Evaluation of E (v )

We now form an interation formula for the expected values E(v). The interation formula finally allows us to compute the limit of the series defined by the iteration formula.

E(vk;j +1) =

X e2Ej+1

Pr(e)vk;j +1(e)

0 1 n X X B@ Pr(e)vk;j +1(e)C = A e2Ej +1 ej +1 =i

i=1

0 1 n X X = pi Pr(^e) (vk;j (^e) + vi;j (^e)Tk )C pk Pr(^e) vk;j (^e)Sk + B @ A 2 2   0 6 1 n X X X X B@ Pr(^e) + pi Pr(^e)vk;j (^e) + piPr(^e)vi;j (^e)Tk C = pk Sk A X

i=1 i=k

e Ej +1 ej +1 = k

e^2Ej

i=1 i6=k

e2Ej +1 ej +1 =i

20

e Ej +1 ej +1 = i

e2Ej +1 ej +1 =i

= Sk pk E(vk;j ) + = Sk pk E(vk;j ) +

0 n X @ i=1 i6=k

n X i=1 i6=k

pi E(vk ; j) + piTk

pi E(vk ; j) +

n X i=1 i6=k

= (1 ? p2k + Tk pk )E(vk;j ) + Tk

n X

e^2Ej

1 Pr(^e)vi;j (^e)A

pi Tk E(vi ; j) n X

= Sk pk E(vk;j ) + (1 ? pk )E(vk;j ) + Tk = (1 ? pk + Sk pk )E(vk;j ) + Tk

X

i=1 i6=k

piE(vi ; j)

piE(vi ; j)

i=1 i6=k

n X i=1 i6=k

pi E(vi ; j)

(42)

We obtain the formulas for E(vi ), i = 1 : : :n analogously. The complete iteration formalizes to a Banach series

Axj = xj+1 with

(43)

0 1 ? ( 1 ? T )p T p


rT pn 2 r 1 r 2 B rT p1 1 ? ( 12 ? rT )p2


rT pn A=B B @ ..................................................... 2 1

2 1

2 1

2 2

2 2

T rn2 p2

T rn2 p1

If we define r~ = ( r12 ; : : :; r12 ), p = (p1 ; : : :; pn) and matrix 1

n

2 2




1 ? ( 12 ? rTn )pn

D(u) as ..

0 .

1 CA

(45)

0 pn we can calculate the fixed point v of the Banach series from Av = v Av = I v ? 12 D v + T < p; v > r~ = I v Solving this equation for v leads to 1

p1

0 ..

0

(44)

2

0p 1 B D(u) = @

0 v = cv D?1(p) r~ = cv B @

1 CC CA

.

1

pn

10 CA BB @

1

r12 .. .

12

rn

1 0 CC = c BB A v@

(46)

1

p1 r12 .. .

12

pn rn

1 CC A

(47)

The constant cv 2 IR can be computed with the constrain that the sum of all drop probabilities equals 1. Therefore we have jjvjj1 = 1. As fixed point we finally obtain exactly one feasible solution

v = Pn 1

12

i=1 pi ri

21

0 BB @

1

p1 r12 .. .

12

pn rn

1 CC A

(48)

In order to obtain a relationship between the ri and vi we make use of the assumption that the percentage vi of the bandwidth a connection receives when experiencing a drop equals its drop probability pi in steady state: pi = vi . If we insert pi = vi into equation (48) we obtain

n 1 X = r12 i 2 v r i j =1 j j

v2

(49)

and can compute a relationship between any two vi ; vj .

vi2 ri2 = vj2 rj2 As round trip times and drop probabilities are always positive vi jjvjj1 = 1 we obtain for vi

(50)

= rrji vj is the only feasible solution of this equation. Using

1

vi = Pnri

1

(51)

i=j rj

Convergence of the Banach series (x )

A.3

i

A

We show the convergence in several steps. First we compute the characteristic ploynomal of and show that  = 1 is a zero of order one. Unfortunately all zeros different from 1 are difficult to compute. We therefore prove that the norm of all eigenvalues is lower or equal to 1 by computing jj jj1. Additionally, we decompose using a fitting decomposition IRn = E( = 1)  Image( ? ) and show that Image( ? ) contracts to zero under .

A A I

A I

A.3.1

A

A

Characteristic polynomal A ()

A

To analyze the eigenvales of we compute the characteristic polynomal A (). We show that A (1) = 0 is a zero of order 1. Further, we define T = 12 Pn 1 1 . i=1 r 2 i

A () = det(A ? I) = 1 ? ( 12 ? rT )p1 ?  rT p2


rT pn rT p1 1 ? T )p2 ? 


T pn 1 ? ( 2 r r = .T. .p. . . . . . . . . . . . . . . . . . .T. p. . . . . . . . . . . . . . . . . . .
.
.
. . .1. ?. . .(.1. ?. . . T. .)p. . . .?. .. rn 1 2 rn n r  1 rnT 2  T 1 ? ( 2 ? r )p1 ?  p2  


pn r n T 1 T 1 ? ( 2 ? r )p2 ? 


pn = QnT r2 p1 i=1 i . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . p1 p2


rTn 1 ? ( 12 ? rTn )pn ?  2 1

2 1

2 1

2 2

2 2

2

2 2

2

2 1

2

2 1

2 2

2 2

2



r  1 T    T 1 ? ( 2 ? r )p1 ?  ? p1 p2 ? rT 1 ? ( 12 ? rT )p2? 


0 0 r 1 ? ( 1 ? T )p ?  ? p


0 2 T 2 r 2 n T = Qn r2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . i=1 i 0 0


pn ? rT 1 ? ( 21 ? rT )pn ?  p2


r 1 ? ( 1 ? T )pn ?  p1 2

2 1

2 1

2 2

2 2

2 2

2 2

2

n

2

n

T

22

2

n

2

rn2

?f () f () 0


0 0 0 1 ?2f () f () 00


0 0 2 3 n 0 T 0 ? f () f ()


0 0 3 4 = Qn r2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . i=1 i 0 0 0


?fn?1() fn () p0 p2 p3 p4


pn?1 ?fn () + pn 1 1 11 00 0 n n n n Y X Y =  QnT r2 B @ B@pi fk CACA  fk CA @B i=1 i i=1 k=1 6 8 1 0 =:'() > z }| { > n p !C > >  Q T BBBQn fk () 1 ? X i C < CC :  62 f1 ? p2 ji = 1; : : :; ng: (i) k =1 B r f () => i A @ i=1 > >   > : : (ii)  Q T r pk Qn fk () : fk () = 0 k=1 k=i

n n 2 i=1 i

i

n n 2 i=1 i

(52)

i=1 k6=i

with functions fi () defined as follows





2 fi () = pi ? rTi 1 ? ( 21 ? rT2 )pi ?  i  2 r p i = Ti  + 2 ? 1 (53) Qn In case (ii) A () does not have any zero at  = 1. Therefore we concentrate on case (i): In case (i) k=1 fk () is always different from 0, as all functions fi (x) yield results different from 0. We can verify that A (1) = 0. Pn We take a closer look at the derivative of function '(x) = 1 ? i=1 fip(ix) to show that '0 (1) > 0 which proves that the zero A (1) = 0 is of order 1.

n p X i 0 2 (x) f (x) f i i=1 n 2 X =  r ? ppi
2 rTi i=1 x+ ?1

'0(x) =

2

i

i

T

Especially, we have '0 (1) = A.3.2

Pn

4T i=1 ri2 pi

> 0.

2

> 0 for x 62 f1 ? p2i ji = 1; : : :; ng

Computation of an upper bound for the eigenvalues of

A

To obtain an upper bound for the absolute value of all eigenvalues i we compute jjAjj1. Let ai;k be the coefficients of

(X n

) jai;kj k = 1; : : :; n jjAjj1 = max i =1 9 80 = < XT 1 1 T = max :@ r2 pk A + 1 ? ( 2 ? r2 )pk k = 1; : : :; n; k i6=k i 23

(54)

A.

(

X 1 = max 1 ? p2k + Tpk ri2 k = 1; : : :; n i   = max 1 ? p2k + Tpk 2T1 k = 1; : : :; n = 1 A.3.3

Decomposition of

A and convergence of A.

)

(55)

A I

We employ a fitting decomposition, so that IRn = E( = 1)  Image( ? ). This implies the earlier result that  = 1 is an eigenvalue of order 1. Let w 2 Image( ? ). We will show that jj wjj1 < jjwjj1 which proves that Image( ? ) contracts under . w = ( ? )y with y = a1 u1 + a2u2 + : : :anun . The ui denote the unit vectors of IRn . Further let wk = ( ? )uk = ( rT2 pk ; rT2 pk ; : : :; ? p2k + rT2 pk ; : : :; rT2 pk ).

A

A I

A I A I

1

2

A



A I

n

2



X

jjwkjj1 = ? p2k + rT2 pk + T pk r12 k i= 6 k i X (1) pk = 2 ? rT2 pk + T pk r12 i6=k i

k

X1 = p2k ? 2T p + T p k k 2 2 rk i ri pk p + = p2k ? 2T k 2 r 2 

k

= pk 1 ? 2T rk2

At step (1) we needed that rT2

k

A



1  2 2 2 2 rk rk +1+ r2rk +:::+ rr2k + + ::: + 2 r12 r22 rk n ?1 k+1

=  rk 2

2

(56)

< 21 .

Awk )k of Awk and the computation

The computation of jj wkjj1 falls into two parts: The computation of the kth component ( of the j th component with j 6= k.







(Awk )k = rT2 pk rT2 p1 + rT2 pk rT2 p2 + : : : ? p2k + rT2 pk 1 ? p2k + rT2 pk : : : + rT2 pk rT2 pn n k 1 2 k k k k 2 2 2 2 2 X pi k Tpk T pk T = ? p2k + p4k ? Tp rk2 + rk2 + rk2 rk2 + rk2 pk i6=k ri2 2 2 X pi = ? p2k ? rT2 p2k + p4k + rT2 pk + Tr2 pk 2 k k k i ri

!



(57)



(Awk )j = rT2 p1 rT2 pk + rT2 p2 rT2 pk + : : : 1 ? p2j + rT2 pj rT2 pk + : : : rT2 pk ? p2k + rT2 pk + : : : + rT2 pn rT2 pk n 1 2 j j j j j j k 2 2 2 X = rT2 pk ? 2rT 2 pj pk + rT2 r2 pk pj ? 2rT 2 p2k + rT2r2 p2k + Tr2 pk rp2i j j j j j j k j i6 k i =

i6=j

T 2 p X pi ? p ? p ) + = 2rT 2 pk (2 j k {z } rj2 k i ri2 j | >0

(58)

24

> 0 therefore j(Awk )j j = (Awk )j

(59)

Awk)jj1.

Now we are able to compute jj(

jj(Awk )jj1 =

X j

j(Awk )j j

XT X T 2 X pi ! = j(Awk )k j + 2r2 pk (2 ? pj ? pk ) + 2 pk 2 i ri j 6=k j j 6=k rj X T 2 X pi ! 1 T XT T 2 p X pi p ? p (2 ? 2p ) ? = j(Awk )k j + 2r2 pk (2 ? pj ? pk ) + k k k 2 2 2 rj rk2 k i ri2 j j i ri | 2 rk j {z } =:R ! X X X pi 2 X 1 = j(Awk )k j + (2 ? p2k )Tpk r12 ? Tp2 k pr2j + 2 T pk 2 +Rk j rj j j j j i ri | {z } | {z } = 0 0 1 1 = X X = j(Awk )k j + (2 ? 4pk )pk ? T2 pk @ pr2j A + T2 pk @ pr2j A + Rk k

1 2T

1 2T

j

j

= j(Awk )k j + (2 ? 4pk )pk + Rk

j

j

(60)

Awk)k cannot be estimated in advance we take a look at both possible cases.

As the sign of (

Awk)k > 0:

Case (

2

2

jj(Awk )jj1 = ? p2k ? rT2 p2k + p4k + rT2 pk + Tr2 pk k k k

2 X X pi (2 ? pk)pk 1 T + ? pk (2 ? 2pk ) ? T pk pi

i

= 0

ri2

4

2 rk2

rk2

i

ri2 (61)

Awk)jj1 is no feasible solution.

Therefore jj(

Awk)k < 0:

Case (

2

2

jj(Awk )jj1 = p2k + rT2 p2k ? p4k ? rT2 pk ? Tr2 pk k k k

2 X X pi (2 ? pk)(pk ) 1 T + ? pk (2 ? 2pk ) ? T pk pi

ri2 2 2 X pi 2 ? pk ? 2T p ? 2T p = pk + 2T p k k rk2 2 rk2 rk2 k i ri i

4

2 rk2

rk2

ri2 (62)

Awk)jj1 ? jjwkjj1 < 0:

We check for jj(

!



2 2 2 ? pk ? 2T p ? 2T p X pi ? p 1 ? 2T p jj(Awk)jj1 ? jjwkjj1 = pk + 2T k k k 2 2 2 rk 2 rk rk k i ri2 rk2 2 2 ? 2T p X pi = ? p2k + 2T p k 2 2 rk rk k i ri2 25

i



2 2 X pi = ? 2Tr2 pk 4Trk2 ? T1 + p1 k i ri2 k

0

X 1 !2

! 1

1 X pi A + ? 2 2 2 2 i ri i ri pk i ri 1 X 1 !2 1 X 1 1 X pi A 2 ? 2 r2 2+ 2 2 k i ri rk pk i ri i ri

2 = ? 2Tr2 pk @rk2 k

X1

0 = ?2T 2 pk @ 0 1 !2 X X 1 p 1 1 1 i A = ?2T 2 pk @ ? r4 + r2 p 2 ? r2 2 r r k i i k k k 0 i 1i B CC !2 B n X X B 1?1 + 1 pi C 2p B C = ? 2T k B 2 2 2 | {z } B i ri rk rk pk ri2 C 0 >0 | {z } i=1 i=k

>0

< 0 For y

(63)

= a1 u1 + a2u2 + : : :anun we obtain

jjA(A ? I)yjj1 =