REMOTE USER EXPERIENCE TESTING OF PEER-TO-PEER TELEVISION SYSTEMS: A PILOT STUDY OF TRIBLER Jenneke Fokker, Arnold Vermeeren, Huib de Ridder Faculty of Industrial Design Engineering Delft University of Technology Delft, The Netherlands {j.e.fokker, a.p.o.s.vermeeren, h.deridder}@tudelft.nl
Abstract
Peer-to-peer television (P2P-TV) systems like Tribler depend on the inducement of massive cooperation among users. In Tribler, knowledge from (social) psychology on altruistic behavior is used for developing cooperation inducing mechanisms. Aiming to verify the effectiveness of these mechanisms, this paper presents the first remote user experience test of the current version of Tribler. The test focused on four usage-related issues: downloading, seeding, moderation, and social reach. It was found that the current users are mainly interested in downloading. Users hardly kept files available for others to download (‘seeding’) and performed few actions in relation to maintaining their own network of friends (‘social reach’) or to performing quality control of content and metadata (‘moderation’).
Introduction With the increasingly sophisticated combination of Internet and Television, any application regarded as interactive Television (iTV) shows a growing complexity in technology, functionality, and interaction. Next to that, peer-to-peer (P2P) technology has brought clear advantages over traditional client-server architectures for broadcasters and viewers, as discussed in previous work (Fokker et al., 2007). Yet, the success of any P2P system fully depends on the level of cooperation among users. Technical enforcement of this cooperation is limited. In (Fokker et al., 2007) an alternative approach is proposed: Applying knowledge from (social) psychology on altruistic behavior for developing features that can induce cooperation. Currently this concept is being implemented in Tribler, a peer-to-peer television (P2P-TV) system for downloading, video-on-demand and live
streaming of television content (Pouwelse et al., 2007). The system gives users access to all discovered content and other users in the network, but also provides the means to browse personalized content with the distributed recommendation engine (Wang et al., 2007) and the advanced social network each user creates implicitly and explicitly. An advantage of having trustworthy friends in Tribler, is that they can speed up the downloading process by donating their own idle bandwidth. In order to evaluate if cooperation is successfully induced, data about the actual usage of Tribler and how users experience working with the software needs to be collected. However, existing measurements on Tribler usage focus on general information, like the ratio of new and active users as shown in Figure 1. This graph shows the effect of media-exposure on Tribler usage: the second big peak in the first week of May represents the sudden boost of new users after a news item about Tribler was broadcast on Dutch television. It also shows that a large part of these new users did not become active users, as the red line drops back almost completely. To better understand the usage and appreciation of Tribler, it is necessary to collect more detailed and timely information. This paper presents a pilot study on a new way for collecting such information. Section 2 reflects on the cooperative tasks users have to perform for Tribler’s success. Section 3 and 4 present the test method and results respectively.
Cooperation in Tribler: Research Questions Our research questions focus on the following topics: -
Downloading. If users download more, it will
help others find interesting content more easily
and it will improve their own recommendations as they are implicitly based on a user’s download history. The Buddycast algorithm implicitly clusters peers into social networks of
so-called ‘taste buddies’ according to their profiles (Wang et al., 2007).
Fig. 1. New users (New PermIDs) and active users (Active PermIDs) from the first public release of Tribler on March 17 2006 until May 15 2006. With credits to Steven Koolen. The questions involved for downloading are:
How much (recommended) files do people download? How often do people remove files from the download history? -
Seeding.
Users should keep completed downloads available for others. By doing so, they will improve the health and availability of content. The questions involved for seeding are: How often do people keep a (completed)
file available for others to download?
-
Moderation. As there is no central server to correct intentional pollution and user mistakes, it is essential that users moderate metadata and content themselves. The questions involved for moderation are: How often do people rate files or mark them as fake?
-
Social reach.
Tribler is a social p2p file sharing system; it is essential that users maintain a list of friends. They do so by inviting their (real-life) friends to using Tribler as well. The questions involved for social reach
How often do users invite, add, or delete friends and peers?
are:
Method Within the TUMCAT framework (TUMCAT, 2007; Vermeeren & Kort, 2006) software was developed for gathering the data on Tribler usage. Specified actions, like adding friends and rating a downloaded file, were logged. At a given action, contextual information like a user’s download history was sensed and questions to users were automatically triggered to collect subjective information (experience sampling). Next to this, users were provided with the possibility of giving feedback (user generated content) at any chosen moment by means of a feedback button in Tribler. For answering the research questions, the open source code of Tribler version 3.5.0 was instrumented with codes for 1) logging, 2) sensing, 3) user generated content, and 4) transferring the data to a remote server for further data analysis.
Downloading.
On average the users started 1.3 downloads per active day, of which 0.3 files were completed. Some indicated how they experienced the download process: “Nice and fast”, “Normal like downloading a Bittorrent file always proceeds. But finding legal downloads is troublesome and there is no good content available” and “(…). It seems that more help is needed for a user while downloading (what is ETA? %U/D, CX etc.?) (…).” Four out of 28 users never started downloading files. Two out of 28 users removed a file from their download history. One of them removed only one file and stated as the reason for that: “It did not download and (I) was not very interested in it. I think it will improve the recommendations closer to the remaining files.”
The 28 participants were academics from various research institutes and universities. One user participated from China. All had experience with file-sharing applications and had a normal broadband connection to the Internet. They were paid €25,- as a reward for participating. After five weeks of testing 20 participants took part in an online questionnaire on the pragmatic and hedonic qualities of Tribler using the survey method as advocated by Hassenzahl et al. (2003). To compare this with the general impression of first-time users, 14 participants in a regular onehour usability test of Tribler were asked to do the same questionnaire.
Results
Net change in download history, 1st week 30 25 20 15 10 5 0 1
5
9
Complete torrents seeded twice or more
13
17
21
Complete torrents seeded once
25
Incomplete torrents
40 35 30 25 20 15 10 5 0 1
5
9
13
17
21
25
User
Fig. 2. Total downloading (top) and seeding (bottom) during the 1st week of testing per user
The other removed all of his files (11 in total) from the download history, but did not answer any questions about this behavior. Ten out of 28 users started one or more downloads from their recommended downloads. Among them, user 27 started 40 downloads from his list of recommendations. He indicated what made him choose a specific file form the list: “The title. But I
do not know what it really is.” and “The singer, I like music.” User 22 replied to the same question “The number of seeders vs. leechers.” The top graphs in Fig. 2 and 3 show the development of the downloading task for all users during their first and second week of testing.
Net change in download history, 2nd week 30 25 20 15 10 5 0 1
5
9
13
17
21
25
Complete torrents seeded twice or more Complete torrents seeded once Incomplete torrents
40 35 30 25 20 15 10 5 0 1
5
9
13
17
21
25
User
Fig. 3. Total downloading (top) and seeding (bottom) during the 2nd week of testing per user
Seeding. Eight out of 28 users never had or kept a
completed file until the end of a session. Among these eight users are the four users who never started downloading at all. Ten out of 28 users kept a file until the next session and then removed it. Ten out of 28 users kept a file more than one consecutive session available for others to download. These are the only potential seeders. The bottom graphs in Fig. 2 and 3 show the
development of the seeding task for all users during their first and second week of testing.
Moderation.
Three out of 28 users rated one or more files with 1-5 stars, among whom user 15 rated 17 files. In the ES (experience samples) he explained what effect he thought this rating would have: “I am trying to find similar types of files/games or with the same name.” and four days later: “(To) change recommendations.” No user marked files as fake (-1).
Social reach.
Two out of 28 users added one friend. Seven out of 28 users added peers as friends, among whom one user added three peers as friends and one user added two. Some users indicated why they thought it was useful or beneficial to add a specific peer: “He turned out to have two files I was downloading. So the (…) files he shares are in common to mine.” or “Maybe, when using a friend, downloads may go faster in some way?” Three out of 28 users invited one friend. One user deleted two friends from his list, but did not answer the ES question.
Discussion and Conclusions The pragmatic and hedonic qualities of Tribler were judged moderately positive. The 14 first-time users judged Tribler more neutral on both scales. Downloading was done by most of the users. Some have stated that downloading as fast as possible is their only interest when using a file-sharing system. Some users explored the recommendations, but showed no continuous use of them. Most users did not keep files available for seeding longer than one session. Moderation was hardly performed. The social reach functions were hardly used either, apart from adding peers as friends. This was used relatively often, but users had no correct idea how it helps to have friends. The results of this pilot study will be used as a benchmark for testing future versions of Tribler. In those tests the following additional two topics will be included:
Injection.
It is vital that fresh content of good quality is available. That is why the injection of new content should be stimulated. Peer uptime. When a few peers stay online for a long period of time, the health and availability of content is improved. The method for user experience testing in this pilot study was regarded as useful, because it clarified what actions each individual performed. Also, it revealed their reasons for doing so, as well as how they valued interacting with P2P-TV systems. Moreover, participants appreciated the way of testing, and felt comfortable in sharing their information.
Acknowledgments
The authors would like to thank TUMCAT members, especially Joke Kort, Leon Roos van Raadshoven, Arjen van Reeven, and Marco Eggenkamp for setting up the testing software. They would also like to thank I-Share (2006) members, especially Arno Bakker, Johan Pouwelse,
and Jan David Mol, for helping instrument the Tribler source code, and bachelor students Elwin Schmitz and Daniel Hagmeijer for their contributions to researching the usability and user experience of Tribler 3.5.0. Finally, the authors are grateful to Isabel Krumholz Adler and Abbie Vanhoutte for their help and support throughout the process.
References Fokker, J., De Ridder, H., Westendorp, P., & Pouwelse, J. (2007). Psychological backgrounds for inducing cooperation in peer-to-peer television, EuroITV'07. Amsterdam, The Netherlands. Hassenzahl, M., Burmester, M., & Koller, F. (2003). Attrakdiff: Ein fragebogen zur messung wahrgenommener hedonischer und pragmatischer qualität. In J. Ziegler & G. Szwillus (Eds.), Mensch & computer 2003. Interaktion in bewegung (pp. 187-196). Stuttgart, Leipzig: B.G. Teubner. I-Share. (2006). Sharing resources in virtual communities for storage, communications, and processing of multimedia data. Retrieved www.freeband.nl/ December, 2006, from project.cfm?id=520 Pouwelse, J. A., Garbacki, P., Wang, J., Bakker, A., Yang, J., Iosup, A., et al. (2007). Tribler: A socialbased peer-to-peer system. Concurrency and computation: Practice and experience, 19, 1-11. TUMCAT. (2007). Testbed for user experience for mobile context-aware applications. Retrieved www.freeband.nl/ January, 2007, from project.cfm?id=1126 Vermeeren, A. P. O. S., & Kort, J. (2006).
Developing a testbed for automated user experience measurement of context aware mobile applications. Paper presented at the UX WS NordiChi'06, Oslo, Norway. Wang, J., Pouwelse, J., Fokker, J., De Vries, A. P., & Reinders, M. J. T. (2007). Personalization on a peer-to-peer television system. Multimedia Tools and Applications, 32(2).
