Graphics on Tablet Computers A Seminar Report Submitted in partial fulfillment of requirements for the degree of Master of Technology by Dhananjay Ambekar Roll No : 123059008 under the guidance of Prof. D. B. Phatak

Department of Computer Science and Engineering Indian Institute of Technology, Bombay April, 2013

Contents 1 Introduction

1

2 Augmented textual data visualizing in 3D visualizations 2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Implementation . . . . . . . . . . . . . . . . . . . . . . . . 2.3 Architectural characteristics of Interaction model: . . . . 2.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Fish-eye visualization 3.1 Introduction . . . . . . . . . . . . . . . . . 3.2 Need for Fish-eye on Tablets . . . . . . . . 3.3 Implementation Issues . . . . . . . . . . . 3.3.1 Alphabetical order . . . . . . . . . 3.3.2 Focus Lock mode by Ben. Bederson 3.4 Summary . . . . . . . . . . . . . . . . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

2 2 3 4 4

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

5 5 5 6 6 6 7

4 Data-Driven Documents : D3 Java-script Library 4.1 Design of the D3 library . . . . . . . . . . . . . . . 4.1.1 Selections . . . . . . . . . . . . . . . . . . . 4.1.2 Data : Enter and Exit . . . . . . . . . . . . 4.1.3 Animations and Transitions . . . . . . . . . 4.1.4 Modules . . . . . . . . . . . . . . . . . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

8 8 8 9 9 9

. . . .

11 11 11 12 12

5 Graphics processing on Aakash 5.1 NEONTM Technology . . . . . 5.1.1 Features of NEONTM . 5.1.2 How to use NEON . . 5.2 Android on ARM . . . . . . .

Tablet . . . . . . . . . . . . . . . . . . . .

6 Conclusion and Future Work

. . . .

. . . .

. . . . . . . . [1] . .

. . . .

. . . .

. . . . . .

. . . .

. . . . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

13

i

List of Figures 2.1 2.2

Occlusion created by textual Data [5] . . . . . . . . . . . . . . . . . . . . Interaction Model [5] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2 3

3.1 3.2

fish-eye list structure in Android Phone book . . . . . . . . . . . . . . . . Span of Focus (Degree of interest function)[1] . . . . . . . . . . . . . . . .

6 7

4.1

Tree-map using D3 [3] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

5.1

NEONTM logo[4] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

1

Abstract Tablet computers have grown enormous number of users in last few years. These slate styled devices use touch screen as main user interaction component. Tablets provide more mobility and comfort when compared to laptops. Tablets have become user’s premier choice device for information access. As a part of information visualization, Graphics on tablets and related technologies have become one of the important aspects in the field of tablet computing. This report discusses different visualization techniques along with method of implementation. Later part contains a discussion on visualization tool and Study of technologies in Aakash II tablet.

Chapter 1 Introduction Information visualization is visual representation of data put to human understanding. Using tablets as access device eventually lead developers to consider ’Information visualization on mobile devices’ as an important factor while designing applications.This study is performed with purposes such as understanding the visualization techniques which are related to use of tablets in the process of information projection. • The technique Augmented textual data visualizing in 3D visualizations[5] deals with visualizing large tree structured data with multiple displays and usage of tablets in it. • Fish-eye [1] visualization deals with managing space while displaying lists or menus. They can be used in applications as they are more intuitive than other traditional techniques. • D3 [2] is a JavaScript library used for manipulating content records/data. D3 helps you to visualize data to life using HTML, Scalable Vector Graphics (SVG) and CSS • To learn the hardware aspect of the topic ’graphics on tablets’, A brief introduction to Neon technology[4] was studied, which is implemented on Aakash II’s processor, which is Cortex A-8. More detailed introduction of each topic is given in respective chapter as report proceeds. In the chapters focusing on techniques, implementation issues and architectural characteristics are also included.

1

Chapter 2 Augmented textual data visualizing in 3D visualizations 2.1

Introduction

This visualization technique deals with Tablets as an essential component of the visualization process. Many data sets such as social Networking data or visualization of business data with many nodes require 3D viewing as the size and structure of data is complex, moreover, these kinds of data are made of massive structures such as graphs, tree-maps with fundamental units such as nodes (e.g. visualizing data of wikipedia with nodes as categories of information hosted). Traditionally large stereoscopic displays are used to visualize the above mentioned types of data. All the information which is to be displayed on node, is put in a text-box which is displayed in the same display as a part of image. This technique created major problems such as occlusion of graphical data due to large text-boxes.

Figure 2.1: Occlusion created by textual Data [5] One of the solutions proposed to this problem was, to use tablets along with large display so that information which previously was rendered on same stereoscopic display was now rendered on tablets, solving the occlusion problem and giving better visualization of 3D data as benefit. 2

2.2

Implementation

The implementing idea is, to project 3D or structural data on a shared display. In this visualization technique, as shared display is used to project 3D data only, a stereoscopic display with depth knowledge can be avoided and standard shared display can be used. users will use their tablets to view the textual data associated with the nodes of 3D data they are currently viewing. In addition, shared display also shows the history and the current activities of all the users who are browsing the information. we can take following measures to improve user interaction: 1. Using right tools for right job As a choice for ’Assistant viewing Device’ to display textual data, Tablets have proven to be a better interfaces. having dedicated reading devices improves text legibility and different nodes data can be compared without causing occlusion in the visualization. both the larger display and tablet with secondary reading utility can create a intuitive application which implements the visualization technique discussed throughout in this chapter. 2. Promote collaboration Consider a scenario where in a 3D data is used for data mining purposes by a group of users. since a user is only presented with a textual data associated with the nodes which he/she is browsing, automatically all the users will refer the common shared display for viewing structural data. in this way, collaborative browsing will increase which is helpful in data mining purposes. and information about all the user activity is presented on shared display which will help understand the users about interest in structural data.

Figure 2.2: Interaction Model [5]

3

2.3

Architectural characteristics of Interaction model:

Following are some of the important characteristics which, an implementation of the mentioned visualization technique is supposed to show[5]. 1. Each participating tablet should carry out a handshake with server in which a unique ID is shared for future usage. 2. Each tablet has a cursor control to a shared display, Providing control over structural data visualized. 3. If 3D structure is rolled over then each tablets receive messages of nodes currently visible and subsequently textual data displayed on tablets is updated. 4. As a part of interaction, user can select items appearing on his screen and after selection, tablets sends a notification on common display with updates performed by current user. 5. Model provides ample amount of user activity information on common display so as collaborative browsing is achieved.

2.4

Summary

Augmented textual data visualization technique removes the problem of occlusion caused by data over structural data using tablets as assisting displays to project textual data associated with nodes. If an utility is made considering the above mentioned architectural views, then it can be used to view the 3D data and perform collaborative browsing which is one of the important purposes of building 3D visualization systems.

4

Chapter 3 Fish-eye visualization 3.1

Introduction

Fish-eye visualization technique for mobile devices is straight forward derivation of traditional Fish-eye view strategy for graphs with hundreds of nodes, only difference in purpose of implementation of this technique on mobile devices is, here, it is used in visualizing information such as vertical menus with many number of menu items which are unable to fit on a normal mobile or tablet display. The strategy is also used in visualizing long indexed information content or graphical representations such as maps on small devices. The reason behind using this strategy is to present a large structure on a small display, so that no part of large structure is lost. The key concept is, part of the structure close to the focal point are clear and visible prominently to the user. number of items visible depends upon how much focus length is set [1]. In our case, i.e. on tablets part of the whole object will be zoomed-in so that it is distinguishably visible to the user. As an example, Android phone book browser can be considered where in, a faint colored index of characters A to Z is displayed on right side. User provides touch input to the character list and accordingly the zoomed character is displayed followed by contact list contents.

3.2

Need for Fish-eye on Tablets

Tablets have moderately small sized display. Particularly speaking for an example, Aakash II has a 7 inch screen. many applications in the same Tablet (Aakash) need to display a menu consisting of as many as 15 items. e.g. An application of video lectures has name of subjects in the menu, which currently is generated in drop down or a search-box. these many items can not be shown in a single menu on such small screen. many times developers categorizes these items so that a small segmented and divided menu is created where in, a sub-menu displays some items. The proposed idea is to implement a fish eye structure in possible scenarios so that we can accommodate more items in less space, simultaneously making user experience more friendly and intuitive. Depending on size of display and number of items to display, Fisheye visualization for list or menu can be created with setting a focus length dynamically to improve the effect [1]. as Tablets use touch interface for interaction, user will quickly adapt to the fish-eye structure while using the applications.

5

Figure 3.1: fish-eye list structure in Android Phone book

3.3

Implementation Issues

In fisheye menus or lists, the items consisting, are displayed completely in a single window, unlike the typical jMenu or drop-down lists. there are certain issues which a designer must consider before implementing Fish-eye structure.

3.3.1

Alphabetical order

The basic purpose behind usage of fish-eye structure is ’to put many items in less space’. so, It becomes obvious that their should be some indexing structure which is followed while placing these menu items. A common choice would be allowing user to use his alphabetical knowledge to navigate through the list in which all the consisting items are ordered alphabetically. As shown in Figure 3.1, on the rightmost end of the device screen, their an index of alphabets is shown which is used for browsing throughout. in this case, user touches on the index; eventually on the letter and the contents are displayed accordingly. Their might be some spacial purpose index items such as numbers or symbols like hash-tags (Figure 3.1) where in list items which are often accesses can be placed. more over, these Items can be chosen dynamically on the basis of history of user’s list access.

3.3.2

Focus Lock mode by Ben. Bederson [1]

As mentioned in his paper on Fish-eye Menu visualization technique, Benderson identifies an issue, which he describes as a difficulty in implementation of Fish-eye. the issue is as follows : While browsing using Fish-eye, Small cursor movement or Even small touch input results in large change of focus in a list. this may cause less smooth browsing of lists or menus. Same issue deals with Resolution selection for displaying data also. If resolution selected for few items of data is more than the others, then size of other items may go smaller. so Maintenance of size of items and resolution becomes an important factor in implementation 6

One of the solutions offered to the above described problem, which is proposed by Benderson himself is, to offer a Focus Lock Mode to the Fish-eye lists. As it can be seen in Figure 3.2, the items in ’Focus length’ part are shown in maximum font size decided and then remaining items are shown in a decreasing font size up to certain limits. all other items are shown in minimum font size. This attempts to solve the problem of ’small cursor/touch movement Figure 3.2: Span of Focus (Degree of interest causes large list displacement’. If de- function)[1] veloper implements the list with more focus area then user might face problem in browsing the whole list and same problem can arise with too much shorter focus also. So, the focus length should chosen such as to minimum browsing should be needed to traverse the list/menu and It should be visually friendly.

3.4

Summary

As this technique was studied with the perspective of its usage on tablets, following things can be drawn down as useful conclusions and observations: Tablets mostly use the list structure over the other options which are arrow bar, hierarchy etc. the reason behind usage of lists is small display size which can not accommodate hierarchical menus. There are many applications with number of data items in their list/menu more than 10-15. as an example, list of courses in academic application can be mentioned.

7

Chapter 4 Data-Driven Documents : D3 Java-script Library There are various methods to visualize information. one can use tree-maps, graph drawing or even heat maps in some specific cases. further to implement these visualization on specific device, there are various techniques. ex. we can use use c,c++ graphics libraries or modern photo-shop extensions. Tablets have been center of discussion devices throughout in this seminar. and many tablets run on Android operating systems. D3 is a JavaScript library used for manipulating content records/data. ‘The JavaScript runtime has access to hardware features of the devices Control runs on; this enables lowlatency UDP networking that would not normally be available to a web interface’. [2] D3 helps you to visualize data to life using HTML, Scalable Vector Graphics (SVG) and CSS. D3 can use data in dynamic ways. for example, you can use D3 to create an HTML table from an array of numbers. Or, using same data a bar chart can be created with transitions and graphical effects. Being a JavaScript library, it automatically becomes android and browser friendly. D3’s allows code reuse through a large collection of components and plug-ins.

4.1 4.1.1

Design of the D3 library Selections

D3 adopts the World wide web consortium (W3C) Selectors API to identify document elements for selection. This language consists many predicates. some of which are listed as follows. they can be used to filter elements from document. • tags • class • id of an element • attribute

8

D3 employs a declarative approach, operating on content sets/selections. For example, to change text color of body. d3.select("body").style("background-color", "black");

4.1.2

Data : Enter and Exit

Using D3’s enter and exit selections, you can create new nodes for incoming data and remove outgoing nodes that are no longer needed. d3.select("body").selectAll("p") .data([4, 8, 15, 16, 23, 42]) .enter().append("p") .text(function(d) return "I’m number " + d + "!"; ); D3 lets you manipulating documents based on data; this includes both creating and destroying elements.

4.1.3

Animations and Transitions

D3’s focus on transformation extends naturally to animated transitions. Easing functions such as “elastic”, “cubic-in-out” and “linear” are available to apply on selections. For example, To fade the background of the page to black: d3.select("body").transition() .style("background-color", "black"); Or, to resize circles in a symbol map with a staggered delay: d3.selectAll("circle").transition() .duration(750) .delay(function(d, i) return i * 10; ) .attr("r", function(d) return Math.sqrt(d * scale); );

4.1.4

Modules

D3 hav various reusable modules which can be easily included into an application to solve some common visualization problems. • SVG module provides various shapes which can be used for chars and graphs. For example, arc function enables user to draw various arcs and associate them with graphical elements such as circles,boxes etc. • Layouts module provides techniques to generate data structures. As an example, figure 4.1 shows tree-map layout implementation using D3 over sample data.

9

Figure 4.1: Tree-map using D3 [3] • For user interaction and browsing, Behavior module provides zoom, scan or even two dimensional translation and scaling

10

Chapter 5 Graphics processing on Aakash Tablet The Aakash 2 tablet or Ubislate 7ci uses Cortex-A8 as a primary processor. this processor is already used in phones, net-books, Digital TVs ant other devices. It has many computing features such as frequency ranging from 600MHz to 1GHz and Dedicated NEONTM technology for multi-media and graphics processing.

5.1

NEONTM Technology

NEON technology is a 128-bit SIMD (Single Instruction, Multiple Data) architecture extension for the ARM R NEONTM CortexTM -A series processors. “The ARM general-purpose SIMD engine efficiently processes current and future multimedia formats, enhancing the user experience. NEON technology can accelerate multimedia and signal processing algorithms such as video encode/decode, 2D/3D graphics, gaming, audio and speech processing, image processing, telephony, and sound synthesis”[4]. Figure 5.1: NEONTM logo[4]

5.1.1

Features of NEONTM

• Process multi-mega-pixel photos quickly. • Anti-aliased rendering and compositing. • Enhanced rendering of 2D and 3D, vector graphics. • NEON supports the widest range of multimedia codecs used for internet applications: Many soft codec standards: MPEG-4, H.264, On2 VP6/7/8, Real, AVS. • Game processing for more frames per second. • NEON development is supported by ARM connected community, one of the largest group of developers and contributers. [4]

11

5.1.2

How to use NEON

For Developers, OpenMAX DL (Development Layer) library is available through the manufacturers and It is recommended approach while developing code involving MPEG-4, JPEG or even audio data. It also provides functions for color space conversion, transformation on images etc. Through libraries, C function call interface to NEON operations supports all data types and operations supported by NEON. and It is available through ARM RealView Development Suite. NEON is uses in the Open Source Community, It is supported in the Open Source projects such as Android - NEON optimizations and NEON versions of critical shared libraries for Ubuntu 09.04

5.2

Android on ARM

Android development is provided with wide range of facilities with ARM architecture. By manipulating processor options, one can optimize performance of application. There are many libraries and customized extensions available for Android development with ARM processor embedded devices. The Android platform makes extensive use of the ARM instruction set and many of the architecture’s unique features to deliver maximum performance, power efficiency and user experience.

12

Chapter 6 Conclusion and Future Work The idea of using tablets as a supporting device for visualizing large graphics structure is innovative and gives a way to whole new style of representing data. new techniques can be built using the method discussed in the paper More intuitive systems can be built if the functionalities available in tablets such as multi-touch, gyroscopic meters are utilized to optimize the user experience. Fish-eye technique can turn out to be very useful If we use Fish-eye menus for browsing the list. If done so, then the task becomes a lot easier and friendly to user. dynamically populating indexes (Hash-tags for most used items) associated with menu items can also save users browsing time. It can be ideal for applications where users visit time to time and contains many items for user to browser. On a small sized displays of tablets it can be a good advantage to choose fish-eye over traditional menus. Creating applications using libraries such as D3 will be more suggestive in the process. Implementing visual techniques for future applications will result in more intuitive applications.

13

References [1] Benjamin B Bederson. Fisheye menus. In Proceedings of the 13th annual ACM symposium on User interface software and technology, pages 217–225. ACM, 2000. [2] Michael Bostock, Vadim Ogievetsky, and Jeffrey Heer. D3 data-driven documents. Visualization and Computer Graphics, IEEE Transactions on, 17(12):2301–2309, 2011. [3] Source for D3 Library. https://github.com/mbostock/d3. [4] Cortex-A8 Processor Resources. a/cortex-a8.php.

http://www.arm.com/products/processors/cortex-

[5] C. Roberts, B. Alper, J. Morin, and T. Hollerer. Augmented textual data viewing in 3d visualizations using tablets. In 3D User Interfaces (3DUI), 2012 IEEE Symposium on, pages 101–104, 2012.

14