A Time Model for Distributed Multimedia Applications Winfried E. K¨ uhnhauser1 and Martin S¨ ußkraut2 1 2

Technical University at Ilmenau, Germany Technical University at Dresden, Germany

Proc. Intl. Conference on Distributed Computing and Networking - ICDCN 2006 c LNCS 4308, pp. 491-496. Springer-Verlag, 2006 Abstract. The significant resource requirements of distributed realtime multimedia applications often push today’s system platforms to their limits. As a consequence, efficient, economic and adaptive management of resources is a major issue in distributed multimedia systems. This paper outlines a model-based time service encompassing three different aspects of time that are fundamental for sophisticated strategies within cross layer failure semantics, flow control, and synchronization.

1

Introduction

Distributed multimedia applications such as traffic control systems, video conferencing, or telemedicine systems exhibit communication patterns that in many ways differ from contemporary communication paradigms. Media streams are sequential, ordered infinite sequences of discrete typed data objects that continuously flow from some source to some destinations. In general, media streams also are quite voluminous, encompassing for example HDTV video streams with a bit rate of up to 2.8 GBit/sec. Media streams have real-time properties, using time to specify synchroneity, periodicity, ordering and timeliness. Last but not least, communication failures affecting single stream fragments are not fatal in general; they may be tolerated depending on the importance of the fragment as well as the application’s quality requirements. Contemporary communication models such as RPC or RMI do not apply well to such communication patterns. Distributed multimedia applications thus often build their own communication abstractions directly on top of low level transport protocols such as TCP, UDP, SCP, or RTP/RTCP. This paper discusses a time model to support the design and implementation of stream communication abstractions. It focuses on four different aspects of time and evaluates how they can be exploited in communication failure semantics, flow control, and synchronization. The model has been implemented by the time service of the Noja middleware framework [1, 2].

2

Time in Distributed Multimedia Applications

Time in distributed multimedia systems serves many different purposes.

2

Temporal Ordering. Objects in multimedia documents are ordered by their creation time. This time in general is relative to the begin of the document. For example in a DV-coded PAL video document frame ordering is implemented by a time stamp carried by each frame. When media documents are communicated, media objects are wrapped into stream objects which may be ordered differently, depending on the order in which they are processed by the receiver. Timeliness. In distributed real time multimedia applications arrival times of media objects are of major importance. Media objects arriving too early require buffer resources, while objects arriving too late may become useless. Here, time is used to synchronize stream processing with real time; incorrect synchronization may result in extensive resource usage or loss of media objects. Synchroneity. Multimedia documents such as multi camera and multi audio track recordings consist of single media documents that are related to each other. Whenever the document is processed as a whole, interdependent streams have to be synchronized with respect to a common time base. Periodicity. In many media document formats any two media objects have a fixed distance in time. PAL video documents for example consist of 25 frames per second, resulting in a frame distance of 1/25 second. Audio documents have a much smaller periodicity, e.g. 1/44.000 for a stereo document in CD PCM format. Periodicity of a media document for example allows to predict resource requirements and temporal ordering of future media objects in a media stream. 2.1

Model Requirements

This section discusses the role of time in four major building blocks of a communication model: failure semantics, flow control, synchronization and adaption control and summarizes the results in fig. 1. A communication model’s failure semantics specifies guarantees given to an application even in the presence of communication failures. Failure semantics simplify application level failure handling by providing easy-to-use abstractions restricting the visibility of complex low-level communication failures. In order to detect timing failures, a failure semantics’s implementation must compare the arrival deadline of a media object with its actual arrival time (question (1)). More sophisticated schemes also strive for predicting future timing failures (question (2)), thus allowing for failure prevention policies or for early discarding of media objects that are predicted to miss their deadline. Both, recovery and prevention policies will only be successful if they meet their objectives within the affected object’s deadline (question (3)). Flow control balances the flow of media streams between the components of a distributed multimedia application, including buffering of early stream objects (question (4)) as well as reordering stream objects that arrive in disarray (question (5)). Additionally, jitter compensation requires answering question (6). Synchronization control manages the temporal relations between different media documents and/or different components of a distributed multimedia application. Questions (7) and (8) arise whenever multimedia documents encompassing more than a single media document are processed.

3

Adaptation control manages dynamic changes of application requirements, stream properties, or resource availability. Additionally to being triggered externally, adaptation control watches internal indicators such as variations of the total average media object processing time, the processing time within individual application components, or transmission times of communication channels (questions (9) - (11)). (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11)

Is a given stream object in time? Can a given stream object reach its destination in time? Is there enough time for error recovery or prevention policies? Is a given stream object too early? What is the ordering of two given stream objects? What is the temporal distance between two given stream objects? What is the processing deadline for two given stream objects so that they will arrive synchronously at their destination? Which stream objects from different media streams must be processed synchronously? What is the average processing time of stream objects of one media stream in the multimedia application? What is the average processing time on one processor? What is the average transmission time on a given communication channel? Fig. 1. Questions to be answered by the time service

3

The Model

The time model introduced in this section is a concise, precise and – with respect to the requirements summarized in figure 1 – complete foundation for exploiting and managing several different aspects of time in distributed multimedia systems. The heart of the model are three basic time types together with methods to handle corresponding time objects. This section introduces these time types and methods and illustrates how they are combined in order to provide answers to the question catalogue in figure 1. 3.1

Basic Sets and Functions

We model a distributed multimedia application by a directed graph, its node set P representing individual application components (processors), its edges (∈ P × P ) modelling the flow of media streams among the processors. Nodes as well as edges have attributes that describe time related dynamic properties such as current channel bandwidth, network load, or CPU and memory availability. In order to calculate and predict the timing behavior of an application, these attributes are monitored during application runtime. Timing properties relate to observation points. Important observation points where streams enter and leave processors will be represented by the set OP = {In, Out}. As an example, the observation point at the output port of processor p

4

where p connects to some processor q is denoted by (p, Out), while its counterpart at q is (q, In). At processors q with more than one incoming edge, each edge may have its individual observation point which then will be identified by (p, In, q); the same holds for outgoing edges, respectively. When a multimedia document is wrapped into a media stream for communication, each media object is encapsulated by a stream object. The set of all stream objects managed by a multimedia application is represented by the set S, and the set of all media objects by the set M . Unwrapping a media object from a stream object is modelled by a function mo : S → M . Each stream object belongs to a media stream ms ∈ MS , M S denoting the set of all media streams of an application. Mapping a stream object to its media stream is modelled by stream : S → MS . All stream objects of a media stream ms use the same path through the application graph: from the processor source(ms) to the processor sink (ms). Time values are of type T = R; negative numbers denote negative temporal distances. The end-to-end-time for a media stream is defined by ete : MS → T . 3.2

Time Types

Media Time tm . Any media document is an ordered sequence of media objects. Ordering is based on generation time stamps carried by each media object. In periodic document formats (such as PCM audio files) ordering may be implicit, given by the order objects appear within the document. Time stamps define temporal ordering as well as temporal distances. For each time stamped media object, the function tm : M → T results in its media time. For each media object from the same document, tm is strictly monotonous. For periodic documents with objects mi and period p ∈ T , media time stamps compute by tm (mi ) = p∗(i−1). Streaming Time ts . Dynamic and proactive policies for adaptation, error correction and prevention require monitoring and prediction of the communication system’s timing behavior. Error correction for example will ponder the time required for corrective actions against deadline misses (question (2)). To this end, stream objects take up time stamps at each observation point. For each stream object, the function ts : P × OP × S → T results in its stream time at some observation point. Stream-global Logical Clock tg . In asynchronous distributed systems we cannot assume a global clock, and time stamps based on unsynchronized local clocks would not compare. However, clocks can be synchronized cheaply by an NTP-like synchronization scheme where the application graph serves as the NTP clock synchronization tree. A stream source serves as stratum-0 server, and explicit synchronization among stratum-0 servers becomes necessary only if more than a single source exists. Synchronization among lower strati then uses downstream synchronization by messages piggy-backed on the stream. Note that the resulting global clock is relative to the root of the synchronization tree and is global with respect to a given stream, only. By synchronizing the root node of the stream’s synchronization tree with an NTP stratum-0 server the whole

5

tree then becomes synchronized with UTC time. The function tg : MS → T represents the stream-global logical clock within the model. 3.3

Questions and Answers

We now combine the time types tm , ts and tg into functions answering the question catalog in fig. 1. Failure Control. With respect to question (1), the boolean function ontime(s) computes by comparing the expected total processing time tpt of s with the stream’s end-to-end time: ontime(s) ⇔ (tpt(s) = ete(stream(s)). If the observation point is the entry point of the sink for s, tpt computes easily by tpt = ts (sink, In, s) − ts (source, In, s). However, in general an observation point is somewhere on the path between source and sink where ts (sink, In, s) is still unknown and has to be predicted. In this case, the more general form of tpt includes a predicted remaining processing time ppt related to (p, op, q) : tpt(s) = ts (p, op, s) + ppt(p, op, sink, In, s) − ts (source, In, s). With ppt discussed below, ontime answers questions (1) and (2). In order to answer question (3), the duration of the failure handling policy must be known. Policy-individual time-to-repair values are modelled by a policyindividual function ttr : P ×OP ×P ×OP ×S → T which any failure handling policy is required to provide. Then, the estimated arrival time eta for a stream object s, repaired at some observation point (p, op) with assistance of a node p0 at observation point (p0 , op0 ) is eta(p, op, p0 , op0 , s) = tg (stream(s))+ttr(p, op, p0 , op0 , s)+ ppt(p, op, sink, In, s). If eta(p, op, p0 , op0 , s) ≤ ts (source, In, s) + ete(stream(s)) holds, s can be repaired at p without violating end-to-end time guarantees. Flow Control. We already know that a stream object is on time if ontime(s) holds at observation point (p, op). For the same reasons, if ts (p, op, s) − ts (source, In, s) + ppt(p, op, sink, In, s) < ete(stream(s)) then s is too early, respectively if ts (p, op, s) − ts (source, In, s) + ppt(p, op, sink, In, s) > ete(stream(s)) then s is too late (question (4)); additionally, the difference between both sides of the equation quantifies punctuality. The order of two stream objects s, s0 ∈ S (question (5)) at some observation point (p0 , In) is defined by the order s and s0 left (p, Out), where (p, p0 ) is an incoming edge at p0 . Thus the relation s ≤ s0 at observer point p is defined by s ≤ s0 ⇔ ts (p, Out, s) ≤ ts (p, Out, s0 ). Note that due to stream object dependencies stream object order and media object order may differ. For question (6), the temporal distance between two stream objects s and s0 at observation point op on processor p computes by distancets (p, op, s, s0 ) = ts (p, Out, s0 ) − ts (p, Out, s). Note that distancets is a value relative to an observation point, suited for jitter compensation with respect to the last edge only. The absolute temporal distance distancetm between two media objects computed by distancetm (s, s0 ) = tm (mo(s0 )) − tm (mo(s)) can be used for (absolute) jitter compensation only if the difference between media and stream order is known. Synchronization Control. For two stream objects s and s0 to arrive at their destination synchronously (question (7)) their expected arrival times

6

must be the same: eta(p, op, sink, In, s) = eta(p, op, sink, In, s0). The optimum processing time opt for each stream object is computed by opt(p, op, s) = eta(p, op, sink, In, s) − ppt(p, op, sink, In, s). Because the processing time of any media object depends on its media time tm , any two media objects s and s0 from different streams have to be processed synchronously (question (8)) if and only if their media times are the same, i.e. the relation dub holds: s dub s0 ⇔ tm (mo(s)) = tm (mo(s0 )). Adaptation Control. The interim processing time ipt of a stream object s between two observation points op, op 0 on processors p, p0 ∈ P is 0 ipt(p, op, p0 , op0 , s) = ts (pP , op0 , s) − ts (p, op, s). Then, the average processing time 0 0 0 −1 0 0 0 of stream objects beapt(p, op, p , op , S ) = s∈S 0 ipt(p, op, p , op , s)(|S |) 0 0 tween op, p and op , p is computed based on statistical data collected from the observation of stream objects from some subset S 0 ⊆ S, which in an implementation will become stable only after a certain application runtime. This answers questions (9) - (11). Predicted Processing Time ppt. Some of the above equations assume that stream times at both observation points are known. If we want to predict the ipt at some earlier processing stage, the more general predicted processing time ppt(p, op, p0 , op0 , s) = apt(source, In, p0 , op0 , s) − apt(source, In, p, op, s) applies. ppt thus assumes that observations in the past allow to foretell the future. Within a given implementation environment, ppt’s quality will depend on the extent this assumption is honored.

4

Summary

The goal of our work was to develop a time model to support the design and implementation of stream communication abstractions in middleware frameworks for distributed multimedia applications. Our approach was to analyze the roles of time within a communication model’s failure semantics, flow, synchronization and adaptation control components, and to develop a stringent time model in which many roles of time time are casted into a small set of well-defined time functions. Implementing the formal model within the time service of the distributed multimedia framework Noja [1, 2] was a straight-forward affair. For a description of the implementation and experimental results with respect to its resource requirements and real-time behavior we refer the reader to the long version of the paper available from the authors.

References 1. Eichhorn, A., K¨ uhnhauser, W.E.: A Component-based Architecture for Streaming Media Applications. In: Proceedings of NET.OBJECTDAYS 2001 – 7th Workshop on Multimedia Information and Communication Systems. (2001) 273–286 ISBN 300-008419-3.

7 2. Eichhorn, A.: Modelling Dependency in Multimedia Streams. In: Proceeedings ACM Multimedia 2006, ACM (2006) 3. Hoffmann, M., K¨ uhnhauser, W.E.: Towards a Structure-Aware Failure Semantics for Streaming Media Communication Models. Journal of Parallel and Distributed Computing 65(9) (2005) 1047–1056