AN EFFECTIVE APPLICATION-LAYER CONTROL FOR REAL-TIME SCREEN RECORDING Shin-Hung Chang ‡, Shao-Ting Lee and Jan-Ming Ho ＋

＋

＋

＋

Institute of Information Science, Academia Sinica, Taipei, Taiwan ‡ Dept. of Computer Science & Information Engineering, National Taiwan University, Taipei, Taiwan E-mail address: [email protected] ABSTRACT: Distance learning service has become a more and more popular medium for the

public to obtain knowledge. Through advances of broadband network and video compression technologies, conventional products in the distance-learning category can provide a real-time streaming broadcast of live contents or an on-demand streaming of archive contents. However, due to the low resolution of a video camera, images of the projected high-resolution computer screen in the video are usually blurred. This makes it difficult for remote participants to recognize most of the characters in the instructor’s operations only by looking at the video. Therefore, real-time screen recording is proposed to record full motions on the computer screen for learning-supporting materials. Using conventional screen recording system, an instructor may find it difficult to operate application programs smoothly. This is due to the fact that most of CPU utilization is taken up by the screen recording routines. In this paper, we propose an effective application control, called "sporadic control (SC)" mechanism, to regulate the frequency of screen recording routines to protect the system and network from instantaneous heavy workload. From the experimental results, we show that the CPU utilization can be reduced by more than 15%. This improvement is significant. KEYWORDS: distance learning service, screen recording and media streaming. has become a key application in distance learning services [1][2][3][4][5][6][7]. At present, various Distance learning service has become a more commercial products exist, including those from and more popular medium for the public to obtain Real Media, Microsoft Media and Quick Time knowledge. The ultimate goal of a real-time technologies. The majority of these products distance-learning service is to provide remote provide a real-time streaming broadcast of the students with the same capability and experience live content (classes or lecture talks) or an onthat in-class students can receive. demand streaming of the archive content. In these distance-learning systems, videos of instructors Streaming Server and their computer operations through projection Encording Projected Materials equipment are taken, then digitized, compressed, Process multiplexed, and streamed to remote clients, as Speaker shown in Fig. 1. However, the resolution of a video camera is Network low compared to a computer screen, therefore, Video Camera images of the projected screen in the video are usually blurred. This makes it difficult for remote Clients Clients Clients students to recognize most of the characters in the instructor’s operations only by looking at the live Fig. 1: An illustration of the conventional distance video. Nowadays, a large number of instructors learning system. generally use electronic materials (computerWith advances of broadband network and based demonstration and presentation) in their video compression technologies, video streaming classes or lecture talks. Therefore, real-time screen recording is proposed to record full 1. INTRODUCTION

motions on the computer screen for learningsupporting materials. To date, two major design criteria have been used to record full motions on the computer screen. Design (1): This design is implemented at the application layer. It captures bit-map images of a computer screen and joins them into a sequence to give the appearance of full motions. Typical commercial products using this design are Winstructor, HyperCam and Microsoft Media Encoder [8][9][10][11]. A screen recording system adopting this design method is independent of system hardware and software configuration. However, due to the limited bandwidth of the backplane bus (such as PCI, AGP, or ISA bus), frame rate of these products is usually low. The low frame rate will result in jittery presentation in the receiving clients. Design (2): This design is implemented at the driver layer. It captures the data sent to the display driver and saves it into a proprietary streaming and archive format. A typical commercial product using this design method is ScreenWatch [12]. With this design method, an extra display driver must be embedded. Therefore, the configuration of system hardware (e.g. device driver, chip or display card) and software (e.g. screen or color resolution) will be strongly restricted. Additionally, if the data sent to the display driver is huge (such as a video playback or animation demonstration), this embedded driver may be inefficient for capturing data and cause the whole screen recording system to halt. In order to avoid incompatibility of system configurations and to decrease the probability of the screen system malfunction, most screen recording systems are implemented at the application layer. However, the frequency of capturing the full screen cannot be substantially increased, due to the limited bandwidth of the backplane bus. Therefore, when recording full motions of a computer screen at the application layer, a good mechanism is needed to improve the recording frame rate. Obviously, not all regions of a computer screen changes at any time when a computer system is running. In Fig. 2, several changed regions (video playback, dynamic web pages, and clock timer) on a computer screen are illustrated. If the update region can be clearly identified when a screen changes, the recording data can be significantly reduced and the frame rate can be dramatically increased. Most computer window systems, including the Macintosh, MS-windows and X-window systems are event- or message-driven. Only parts of these events or messages correlate with screen updates.

From the parameters associated with these events or messages, we can capture the updated regions on the computer screen. However, not all screen updates trigger system events or messages, so the full screen must be captured entirely from time to time. Dynamic web pages

Video

Clock

Fig. 2: An illustration of computer screen update.

Therefore, our screen recording system uses the following two basic operations, (1) intercepting system messages and tracking for display variations, (2) capturing part of or the whole screen. The continuous operation of these two approaches is sufficient for less time-critical applications. However, by using this approach for distance-learning application, instructors may find it difficult to operate their application software. This is due to the fact that most of the CPU utilization is taken by the screen recording routines. Additionally, when the application software creates a large number of display variations and event messages, instantaneous heavy workload usually slows down the system and the network and thus degrades the quality of distance learning services. The goal of this paper is to avoid these instantaneous workloads yet to guarantee the screen recording quality at the same time. We present a novel application-layer control mechanism, called "sporadic control (SC)", to regulate the frequency of screen recording routines to protect the system and the network from instantaneous heavy workload. To implement this notion, we model our screen recording kernel as being composed of the event interceptor (EI) and the periodic refresher (PR). In addition, these two components are coordinated by the SC mechanism. With the addition of the SC mechanism, the CPU utilization can be reduced by more than 15%. The improvement is significant.

The rest of this paper is organized as follows. Problems of screen recording are stated in section 2. Our proposed control mechanism is presented in section 3. The analysis and experimental results are presented in section 4. Finally, we state the conclusions of this paper in section 5. 2. PROBLEM SETTING 2.1 Computer Display Architecture Bus interface is a set of electronic signal pathways that allows information and signals to travel between components inside or outside of a computer. This transmission medium is like a highway used to transfer packets of data between the graphics card and the computer's chipset or motherboard. Bus interface plays an important role in influencing the efficiency of a computer system. Fig. 3 presents the typical display architecture in the current computer system. On the right of the figure, a chipset (such as the north-bridge chipset) connects the processor (Central Processor Unit, CPU) and system memory (Dynamic Random Access Memory, DRAM) through the system bus. On the left of the figure, the graphic module and other peripheral devices are connected with this chipset through the backplane bus. Computer Moniter

Screen Recording

AGP PCI

Chipset

(Backplane Bus)

(System Bus)

Other Pheriphrals

Instead of only capturing full screen, we model our screen recording kernel as being composed of the following components. Event Interceptor (EI): The EI collects all events generated by the kernel of window system. Event Filter (EF): The EF receives whole events (inspected by the EI) and filters those correlated to screen updates. From the information included in these screen update events, the data of changed regions on the screen is captured and sent to the URB (Update Region Buffer) for archiving. Update Region Buffer (URB): The URB is for collecting and storage of whole update region data and for archiving these data in a proprietary format. Periodical Refresher (PR): The PR periodically captures the current screen, and compares it with the previously captured one. By computing modified parts, the PR captures the changed region data and sends them to the URB for archiving.

(2) System Memory

Graphic Module

2.2 Our Screen Recording Architecture

The detail architecture is shown in Fig. 4.

(1) Display Memory

direction (1) is more efficient than that of direction (2).

Periodical Refresher (PR)

Update Region Buffer(URB)

Archive

Processor

Fig. 3: A typical display architecture in the current computer system.

In this figure, two data transmission directions through buses are labeled. Direction (1) indicates the data flow (from system memory to display memory) of the system display. Direction (2) represents the data flow (from display memory to system memory) of the screen recording. At present, the chipset and bus controller in many commercial products usually optimize the process of direction (1). Therefore, the transmission of

Event Filter(EF)

Event Inspector (EI)

Fig. 4: The original architecture of our screen recording system.

Continuous operations of the EI and the PR operations are sufficient for less time-critical applications. However, the CPU utilization is almost fully taken up by the screen recording routines and results in the fact that other application programs cannot be operated smoothly. Therefore, a control mechanism is needed to effectively reduce the CPU utilization for running the various application programs

smoothly and guaranteeing the screen recording quality at the same time.

rate of screen capture routines is regulated under a stable state.

3. PROPOSED APPLICATION-LAYER CONTROL MECHANISM

Frequency Control

Perodical Refresher (PR)

3.1 Screen Recording System with an Integrator To reduce the CPU utilization, we add several extra components and separate the EI and PR to run in two independent procedures, as shown in Fig. 5. Integrator: Because some update regions generated by the EI component may contain the overlap portion, the Integrator is used to combine the overlap region and reduce the number of capture routine. Update Region Filter (URF): The URF is used to regulate the generation of update region. This component can protect the screenrecording system from the instantaneous heavy workload. Perodical Refresher (PR) Update Region Filter(URF)

Update Region Buffer (URB)

Sporadic Controller (SC) Update Region Update Region Archive Buffer (URB) Filter(URF)

Event Filter (EF)

Event Inspector (EI)

Event Rate Feedback

Fig. 6: The architecture of screen recording system with the Sporadic Controller.

With the addition of the SC, the CPU utilization is reduced significantly. Many applications with high CPU utilization (such as playback video, play game, demo animation) can be operated smoothly and are not affected by screen recording routine. The instructor will not be aware that the screen recording is running. 4. EXPERIMENTAL RESULTS

Archive

Integrator

Event Filter (EF)

Event Inspector (EI)

Fig. 5: The architecture of screen recording system with the Integrator.

In the following, we test the efficiency of capturing full screen of a computer under Window XP. The hardware and software configurations of our screen recording system are shown in Table 1. Experimental results are presented in Fig. 7. Table 1: The hardware and software configurations of our screen recording system.

3.2 Screen Recording System with the Sporadic Controller

Sporadic Controller (SC): The SC makes highlevel decisions, including when to archive data of update region, when to combine update regions, when to compare frames and when to produce intra frame. From the basis of event rate feedback reported by the EI, the SC can adjust the capturing frequency of the PR. By coordinating these components, the processing

Values

CPU Chipset OS

Pentium III 700 Intel 82443BX Windows XP

AGP Bus Interface Screen Resolution

In order to further reduce the CPU utilization, we add a Sporadic Controller into our screenrecording system, as shown in Fig. 6.

System Configuration

1280x1024 32 bits 1024x768

16 bits

800x600 0

2

4 6 Frame Rate (fps)

8

10

Fig. 7: Efficiency of capturing full screen with "ATI 128 AGP 2x" VGA chipset.

In this experiment, the process of capturing full screen runs in best effort. From the basis of experimental results in Fig. 7, we find that the frame rate of capturing full screens is usually low ( < 10 frames per second). However, in a screen recording system, captured frames usually need to be compressed for streaming and archiving. Therefore, the number of real-time captured frame may even less. Therefore, we design our screen recording system with another architecture. By using the "performance monitor" software in Microsoft Windows System, we test the effectiveness of our proposed control mechanism. In Fig. 8, we present the results of these experiments. In figures 8-a and b, we show typical traces of CPU utilization when the screen recorder is running without and with the sporadic control mechanism respectively. In figures 8-c and d, the screen recorder is running while the lecturer is receiving a remote video-streaming program. In these figures, the average CPU utilization of these 4 scenarios are 76%, 37.6%, 85%, and 69.2%, respectively. Compared with the screen recording system without the SC mechanism, the CPU utilization of our system with the SC mechanism is reduced by more than 15%.

(c) Run video application without SC mechanism.

(d) Run video application with SC mechanism. Fig. 8: The CPU utilization of running screen recording system with or without the SC mechanism.

5. CONCLUSIONS

(a) Without SC mechanism.

(b) With SC mechanism.

It is a current trend to use screen recording technology to improve the efficiency of distance learning service. In this paper, we present our design of a screen recorder with emphasis on improving the responsiveness of applications used to present the lecture. In other words, we focus on decreasing computing and network overhead such that user applications run without severe performance degradation. The key idea is to control the period in processing the event queue and thus to refresh screen updates. We also partition the screen into several blocks and periodically update each screen block with the update period being dictated by the sporadic controller. Our experiments verifies that our design effectively decrease system load and allows a lecturer to smoothly operate his/her demonstrations without noticeable interruptions. The experiments also show that the proposed control mechanisms are able to decrease CPU utilization by more than 15%. We also implemented a prototype system which streams both audio/video and screen recording. We are also studying the design of error control protocols to enhance system reliability.

6. REFERENCES [1]

Mark Gellevij and Hans Van Der Meig, "Screen Captures to Support Switching Attention," in proc. of IEEE Transaction on Professional Communication, Vol. 45, No. 2, June 2002. [2] Thiriet, J.-M., Robert, M., Lappalainen, P., Hoffmann, M.H.W., Martins, M.J.M., and Seoane, A., "Toward a Pan-European virtual university in electrical and information engineering," in proc. of Education, IEEE Transactions on Education, Vol. 45 Issue: 2, May 2002. [3] Swamy, N., Kuljaca, O., and Lewis, F.L., "Internet-based educational control systems lab using NetMeeting," in proc. of IEEE Transactions on Education, Vol. 45 Issue: 2, May 2002. [4] Patel, G., Tabrizi, and M.H.N., "E-class - a multimedia and web based distance learning system," in proc. of International Conference on Information Technology: Coding and Computing, 2002. [5] Fong, A.C.M., and Hui, S.C., "Low-bandwidth Internet streaming of multimedia lectures," in proc. of Engineering Science and Education Journal, Vol.10 Issue: 6, Dec. 2001. [6] Deshpande, S.G. and Jenq Neng Hwang, "A real-time interactive virtual classroom multimedia distance learning system," in proc. of IEEE Transactions on Multimedia, Vol.3 Issue: 4, Dec. 2001. [7] Castro, M., Lopez-Rey, A., Perez-Molina, C.M., Colmenar, A., de Mora, C., Yeves, F., Carpio, J., Peire, J., and Daniel, J.S., "Examples of distance learning projects in the European Community," in proc. of IEEE Transactions on Education, Vol. 44 Issue: 4, Nov. 2001. [8] Te-Yi Liu, Yi-Chin Huang, and Wen-Chin Chen, "A Novel Algorithm for Real-time Full Screen Capture System," in proc. of IEEE Fourth Workshop on Multimedia Signal Processing, 2001. [9] http://www.flickerfree.com. [10] http://www.hyperionics.com. [11] http://www.microsoft.com/windows/window.htm. [12] http://www.screenwatch.com/screenwatch.html