Master’s Thesis

A Framework for Mobile Paper-based Computing by Tomas Sylverberg LITH-IDA-EX--07/004--SE 2007-02-14

Master’s thesis

A Framework for Mobile Paper-based Computing by Tomas Sylverberg LITH-IDA-EX--07/004--SE

Supervisor : Dr. Erik Berglund Dept. of Computer and Information Science at Link¨ oping University Examiner : Dr. Erik Berglund Dept. of Computer and Information Science at Link¨ oping University

Abstract Military work-practice is a difficult area of research where paper-based approaches are still extended. This thesis proposes a solution which permits the digitalization of information at the same time as work-practice remains unaltered for soldiers working with maps in the field. For this purpose, a mobile interactive paper-based platform has been developed which permits the users to maintain their current work-flow. The premise of the solution parts from a system consisting of a prepared paper-map, a cellular phone, a desktop computer, and a digital pen with bluetooth connection. The underlying idea is to permit soldiers to take advantage of the information a computerized system can offer, at the same time as the overhead it incurs is minimized. On one hand this implies that the solution must be light-weight, on the other it must retain current working procedures as far as possible. The desktop computer is used to develop new paper-driven applications through the application provided in the development framework, thus allowing the tailoring of applications to the changing needs of military operations. One major component in the application suite is a symbol recognizer which is capable of recognizing symbols parting from a template which can be created in one of the applications. This component permits the digitalization of information in the battlefield by drawing on the paper-map. The proposed solution has been found to be viable, but still there is a need for further development. Furthermore, there is a need to adapt the existing hardware to the requirements of the military to make it usable in a real-world situation. Keywords : pen-gesture recognition, paper-based user interfaces, ubiquitous computing, software engineering, digital pen

iii

iv

Acknowledgements I would like to thank my supervisor, Erik Berglund, for proposing such an interesting topic for this master’s thesis. Further, I would like to thank Olof Leifler for all the support and help with LATEX. It has been a pleasure to be able to discuss my problems with Anders Larsson and Per-Ola Kristensson who had many interesting ideas to share. Further, a big thank you to all the people who have tested the applications and have thereby given me the ambition to work even harder to improve the system. I would also like to thank my parents, my brother and my girlfriend who all have supported me during this thesis. Without your support this thesis would not have been what it is today.

v

vi

Contents 1 Introduction 1.1 Research Question . . . . . . . . . . . . . . . . . . . 1.2 Ubiquitous Computing . . . . . . . . . . . . . . . . . 1.3 Research Approach . . . . . . . . . . . . . . . . . . . 1.3.1 Supporting Development of Interactive Maps 1.3.2 Supporting Extensible Maps . . . . . . . . . 1.3.3 Supporting Different Coordinate Systems . . 1.3.4 Supporting Different Interaction Devices . . . 1.3.5 Supporting Symbol Recognition . . . . . . . . 1.3.6 Supporting Plug and Play of Components . . 1.4 Paper Summary . . . . . . . . . . . . . . . . . . . . 1.5 Glossary . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . .

1 3 3 4 5 5 6 6 6 7 7 8

. . . . .

9 9 11 11 13 14

3 Implementation 3.1 Development Platform . . . . . . . . . . . . . . . . . . . . . 3.1.1 Java 2 Micro Edition . . . . . . . . . . . . . . . . . . 3.1.2 CLDC and MIDP . . . . . . . . . . . . . . . . . . .

17 17 17 19

2 System Design 2.1 Component Based Development . . 2.2 The Development Framework . . . 2.2.1 Desktop Applications . . . 2.2.2 Cellular Phone Application 2.2.3 Digital Pen and Paper . . .

vii

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . . . . . . . .

. . . . .

. . . . . . . . . . .

. . . . .

. . . . . . . . . . .

. . . . .

viii

CONTENTS

3.2 3.3 3.4

Symbol Recognition . . . . . . . . . . . . . . . . . . . . . . Symbol Format . . . . . . . . . . . . . . . . . . . . . . . . . Use of XML in the Project . . . . . . . . . . . . . . . . . .

20 24 24

4 Conclusion

27

5 Article 5.1 Abstract . . . . . . . . . . . . . . . . . . 5.2 Introduction . . . . . . . . . . . . . . . . 5.2.1 Symbol Recognition in the Field 5.3 The Kartago Project . . . . . . . . . . . 5.4 Interactive Maps in the Field . . . . . . 5.5 System Overview . . . . . . . . . . . . . 5.6 Requirements . . . . . . . . . . . . . . . 5.6.1 Support Complex Pen-gestures . 5.6.2 Robust . . . . . . . . . . . . . . 5.6.3 Support Low-powered Devices . 5.6.4 Easy to learn . . . . . . . . . . . 5.6.5 Easy to maintain . . . . . . . . . 5.7 Recognition . . . . . . . . . . . . . . . . 5.7.1 Symbol Language . . . . . . . . . 5.7.2 Recognition Process . . . . . . . Related Work . . . . . . . . . . . Algorithms . . . . . . . . . . . . 5.8 Evaluation . . . . . . . . . . . . . . . . . 5.8.1 Participants . . . . . . . . . . . . 5.8.2 Material . . . . . . . . . . . . . . 5.8.3 Apparatus . . . . . . . . . . . . . 5.8.4 Procedure . . . . . . . . . . . . . 5.8.5 Results . . . . . . . . . . . . . . Error . . . . . . . . . . . . . . . Latency . . . . . . . . . . . . . . 5.9 Conclusion . . . . . . . . . . . . . . . . 5.10 Acknowledgements . . . . . . . . . . . .

29 29 30 31 31 33 34 35 35 35 35 35 35 36 36 36 36 38 41 41 41 41 41 42 42 42 42 43

Bibliography

. . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . .

44

Chapter 1

Introduction Ever since the advent of computers, engineers have continuously managed to reduce the size of the computer systems. The first computers occupied entire rooms, and it wasn’t until the desktop computers made their entrance, that computers became available to the general public. The third generation of computing is called ubiquitous computing, a term coined by Mark Weiser in the 1990’s [1]. The ubiquitous computer is supposed to become invisible at the same time as it provides the benefits of using a computerized system. An important implication of ubiquitous computing is that the computer system should allow people to perform their normal tasks in their normal environment, but with computer support. In this context a natural mean of interaction is paper and pen, which has formed part of people’s everyday life for a long time. Today, although computers are making their entrance in people’s life, work-practice still involves the use of paper and pen. For instance, military work-practice, relies heavily on working with maps. Soldiers draw on maps, and discuss strategies looking at maps. Current work-practice, however, implies that if the operators at headquarters want to have a complete overview of the situation, it is necessary for all units to report their findings manually. The arguments for having the information in a digital form is growing stronger as critiquing systems are being developed. Paper as a tool for interaction possesses various features which makes 1

2

it attractive. A system which is based on a paper application is easy to transport, and it has low weight. Further, paper has a high resolution which is demanded in, for example, military applications. In the future, a combination of paper and pen interaction can become reality in electronic paper. The technology at this stage, however, proposes the use of digital pens together with specialized paper in order to get a version of electronic paper. Whereas computer systems which use computer screens have problems with low resolution and low portability, this is not the case with paper which combines a higher resolution, and the portability of paper [2]. When electronic paper, in its true sense, will become available the ability to provide visual feedback directly from the computer system will be added. A further benefit from using paper is that it provides a natural mechanism for graceful fallback, which is a feature that can be important in, for example, military applications. If the computer system fails to function, the information is still available on the paper. For military purposes the weight of the soldiers equipment is crucial. Laptops and tablet PCs are highly portable, but their weight is considerably higher than a system consisting of, for example, a cellular phone, paper and pen. Currently there is no other solution than paper maps which allows the use of a large 1 , high-resolution image. A probable reason why computerized systems haven’t been accepted in the military is their weight, low resolution, and unnatural interaction models. This calls for the development of applications which better suits the specific needs of this user group. The goal with the project was to develop a tool for military use in which a digital pen could be used as an interaction device to work on maps. The main benefit from such a system was expected to be an augmented map representation together with digitalization of the information.

1 The

size of a paper map can typically be far bigger than A3-format. Even very big paper maps are wearable and can easily be folded.

1.2. Ubiquitous Computing

1.1

3

Research Question

The main research question was to investigate in which way it is possible to support the use of an online paper-driven computer system in military work-practice, where multiple users should be able to interact and share information. The requirements on the solution are implicitly decided by current work-practice, and limitations of the technology. To be usable, the solution must fulfill the following requirements: • The solution must be light-weight • The solution should be based on paper maps • The solution should facilitate the development of new interactive paper maps, thus enabling rapid prototyping • The application should support typical pen-based interaction, for instance symbol writing In this thesis a solution to the military’ needs can be found in the form of a framework for paper-driven applications. This thesis describes how a solution to the problem was developed. The thesis is multidisciplinary in the sense that it treats several disciplines within Computer Science. Ubiquitous Computing, Paper-Based Interfaces and Software Engineering are all fields that are related to this work.

1.2

Ubiquitous Computing

One of the ideas behind ubiquitous computing is the integration of computing into normal life in a way which requires less attention on the computer interaction while the benefits from using the system are substantial [3]. When developing ubiquitous computing solutions the need for explicit interaction should be kept to a minimum, which in turn means that the interaction with the computer virtually melts together with the environment [4]. The idea to hide computer interaction can be traced back to Mark Weiser who in 1991 wrote “The most profound technologies are those that disappear” and “They weave themselves into the fabric of everyday life

4

until they are indistinguishable from it” [1]. He also writes that the main problem of getting the computer out of the way is not the GUI, but rather the whole context of using the computer [5]. One way of taking the computer out of its current context is to make it mobile. Therefore, mobility is an important issue in ubiquitous computing. It permits the computer’s users to use the computer anywhere. Technology available nowadays allows users to use their cellular phones in much the same way as if they would use a desktop computer. However, there is a need for improvement when it comes to how interaction is carried out. Most systems rely heavily on a touch screen, which uses traditional desktop system based menu structures. As pen-based applications become more extended the need to develop pen-based user interfaces becomes more important [6]. Given that the use of pens to interface a computer system is relatively new, there is a need to investigate how a user interface which permits easy navigability through the system can be constructed, given that there is no possibility to hide inactive menu items. A further design consideration when working with ubiquitous computing is that the designer can not expect to keep the users attention during longer periods of time [4]. The implication of this is that the system must be very easy to understand, and require a small amount of time to learn. Existing experimental systems usually require heavy equipment, and alter work-practice, thus forcing the users to learn to work with the system. Such a system is the BARS system [7] which uses a head-up display and a mouse to interact with a laptop computer.

1.3

Research Approach

Within the context of this thesis the focus is set on the traditional paper map as the central work tool in an interactive system for the soldiers’ workflow. The hardware platform that the project work with is paper maps using an Anoto pen [8], paper, and a cell phone for communication and feedback. There are several reasons why this hardware platform has been chosen. First and foremost, it is highly wearable and uses to great extent technology which people is familiar with. Further, the development of new cellular phones with computer-like qualities is everyday more present which pro-

1.3. Research Approach

5

vides a platform where computing power will be improved at the same time as the platform is maturing. The choice to use an Anoto pen over any other device is natural since this technology is the only one which currently allows to build a system which, except for a computer2 , does not require any additional tools than the special paper3 and the pen to work. Altogether this provides us with a very light-weight system that also fits in a leg pocket or similar. However, there are also a series of software design issues to solve.

1.3.1

Supporting Development of Interactive Maps

Paper-maps typically cover limited regions, and are very detailed. This implies that in a typical setting, several paper-maps are needed to cover the battlefield. In a ubiquitous computing setting, with augmented maps it is necessary to individually prepare the map for each region. This calls for a method which permits rapid preparation of paper-maps. Furthermore, each paper-map can be seen as its own application which means that differentiated behavior could potentially be built-in to the maps. To be able to support both of the discussed ideas a component based approach toward development of new applications was adopted. The underlying idea was to permit an inexperienced user to build different paper-map applications by selecting building blocks from a menu.

1.3.2

Supporting Extensible Maps

One important aspect which was thought of was to allow new functionality to be added to an already working map. To make this possible a method to represent a map had to be developed. For this purpose a XML-format was developed which should support short development time, at the same time as it should be expandable. Further reasons to use a XML-format is that it provides a common platform for data-exchange between different platforms. The map representation was heavily influenced by the component based approach. The combination of a XML-driven application structure together 2 Computer

here is used in its broadest sense, including any kind of device which has computational power such as PDA,cellular phone, embedded system 3 The paper has a special pattern printed on it.

6

with a component based approach allows the creation of applications on the fly. Focusing on how the development time for new paper-maps could be accelerated an application which automatically generates files in the XMLformat was developed. This also allows to comply with the wish to allow inexperienced users to develop applications. Further, the application limits the need to have knowledge about programming on the J2ME platform to a minimum.

1.3.3

Supporting Different Coordinate Systems

One important aspect when working with maps are different coordinate systems, and how to handle them. For this thesis, the simplifying assumption that the coordinate system can be considered flat for small geographic regions was adopted. For more specialized uses it might be necessary to exchange the current representation with a more general one.

1.3.4

Supporting Different Interaction Devices

To be able to support different interaction devices the concept of an input device hook was introduced into the system. This design decision permits the user to allow any kind of interaction device to be connected to the system as long as appropriate drivers are available.

1.3.5

Supporting Symbol Recognition

One crucial component to make pen-based interactive maps a good solution is symbol recognition. It became obvious that to be able to support the soldiers’ current work-practice it was necessary to allow them to use their regular symbols when drawing on maps. As a result a literature study was started, and based on the information found about existing systems a partly new approach to symbol recognition was chosen. The choice to develop a different solution than the ones provided by various researches was based on the underlying platform, which has limited computational power. Further, symbol recognition is often based on glyphs rather than complex figures which automatically disqualified many of the methods, since they could not work with this kind of figures. Symbol recognition is a complex field in

1.4. Paper Summary

7

which entire studies focus on, for example, recognizing numbers, or other very specific fields [9]. For the purpose of this thesis a symbol recognizer which would allow the recognition of virtually any kind of symbol had to be developed. The choice to use a template-based approach toward symbol recognition, meant that it was necessary to represent symbols in a data format. There are very different ways how this can be carried out, sometimes bitmaps are used, other times database entries are used. In this case XML was found to be an appropriate format. To be able to quickly add new symbols to use with the symbol recognition a new desktop application was developed with this in mind. The symbol recognizer was first developed as a desktop application, since it would be more efficient to test the algorithm in a desktop setting. The first tests of the symbol recognition algorithm resulted in a desktop application which can be used to evaluate how well symbols are recognized, since the same algorithm was ported to the Java 2 Micro Edition platform.

1.3.6

Supporting Plug and Play of Components

The last step in the development process was to develop a set of plug-ins which would support the military in their work in the field. The result is an application which can recognize a set of military symbols, attach information to the map coordinates related to the symbol, and give auditory feedback of what the system has detected. Other plug-ins makes it possible to send SMS/MMS messages, download new maps through web-services and draw on the map. An important application is to show information when clicking on areas which have been triggered with information. The component based approach has permitted the development of new plug-ins knowing that a basic system is working and that any additional features can be implemented if time permits it.

1.4

Paper Summary

This thesis also contains a paper manuscript, focused on the symbol recognition part of the work performed within the context of the work. The paper is written with the intent of submitting it to a renowned interna-

8

tional conference. The following text is the abstract of the article, the article can be found in Chapter 5. We present a robust on-line handwritten recognition system that recognizes NATO symbols drawn on paper maps using a digital pen and a mobile phone. Our user evaluation shows that very high accuracy is achieved. Furthermore, our proposed system has low latency running on off-the-shelf mobile phones. The recognition system is part of a research effort in augmenting existing work-practices of soldiers in the field with very lightweight computer support.

1.5

Glossary

CDC The Connected Device Configuration is part of the Java runtime environment used by, for example, TV top-set boxes. CLDC The Connected Limited Device Configuration is one part of the Java runtime environment used by, for example, cellular phones. J2ME The Java 2 Micro Edition is a Java platform intended for devices with limited computational power J2SE The Java 2 Standard Edition is the Java platform which is mostly used for desktop computers JVM The Java Virtual Machine KVM The Kilobyte Virtual Machine is a virtual Java machine with limited memory footprint MIDP The Mobile Information Device Profile is one part of the Java runtime environment used by, for example, cellular phones MIDlet A Java program which requires MIDP to be able to execute. The concept is similar to an Applet in J2SE Pen-gesture In this context equivalent to a number of strokes which make up a symbol XML Extensible Markup Language

Chapter 2

System Design The specific constraints dictated by the way military work, together with the hardware constraints dictated how the system was developed. The following presentation of the system describes the key components to understand the system. A general overview is given in figure 2.1.

2.1

Component Based Development

The use of paper-based maps in military applications imply that each paper-map must individually be crafted as an application. For this purpose, three support programs were developed for a desktop computer. The application soldiers would use is executed on a cellular phone with support for MIDP/CLDC 1 . Military use of maps has traditionally been very extended, but there has hardly been any development of its uses. Ubiquitous computing permits new applications which can improve the use of maps in military practice. To support differentiated applications within the same support framework a component based approach was chosen. One benefit from a component based approach is that it supports incremental development. Further, components can be developed separately and then connected together [10]. This 1 Mobile

Information Device Profile/Connected Limited Device Configuration

9

10

Kartago System

Symbol Creator

XML

Cellular phone application

Symbol recognizer application

Figure 2.1: System overview

flexibility allows the adaption of the system’s functionality which gives shorter development time. Another possibility is to develop small components which are combined into more complex systems. This functionality has been used in various components to create complete sub-applications (plug-ins). Currently the cellular application is controlled using a stylus pen, but the system also supports the use of a digital pen. When a digital pen is connected, the system provides dual input possibilities, which provides a fallback to stylus if the digital pen would stop working. In a more general setting an application developer could provide different ways to interact with the system based on the hardware that is available. The framework supports any kind of input device to be used as long as it complies to an interface. How the device handles its input to comply with the interface is a problem the device driver programmer has to consider. Since the software can run on very different platforms it might be necessary to support some

2.2. The Development Framework

11

kind of fallback if certain functions are missing, for example, a plug-in which permits the user to compose a MMS message allows cellular phones with a compatible camera to attach a picture taken with the camera to the MMS message. The photo could potentially also be connected to an area on the screen. A user with a cellular phone without a camera will only be able to attach photos that are already stored in the phone.

2.2

The Development Framework

The implementation can be divided into two branches, one for desktop users, and the other for cellular phone users. In the desktop environment there are three applications 2 , which can be seen as a development framework.

2.2.1

Desktop Applications

The Kartago System Editor is the core of the desktop application development framework (see figure 2.2). The program permits a user to load an image and on top of the image draw boxes and polygons. The drawn figures can then be connected to functions (plug-ins). The actual connection of functions is achieved by choosing appropriate plug-ins from a drop-down menu whose items are loaded based on what plug-ins are available3 in the development area. It is also possible to submit arguments to the plug-ins which can modify the plug-in’s behavior. The result of the editing can be saved to a XML file which can then manually be installed on the cellular phone. Another option is to put the file in a database and retrieve it on-demand using web services.

2 Two

of the applications are developed in two different versions- supporting different interaction methods 3 The plugins are in a separate namespace to avoid name-clashes. When a new plugin has been developed it will appear in the drop-down box automatically.

12

Figure 2.2: Left: Main window. Right: Connecting a plug-in to an area

Figure 2.3: Left: Main window. Right: Creating a new symbol

2.2. The Development Framework

13

The Symbol Creator is an application which is used to create symbols and save them in a XML-format (see figure 2.3). For each symbol it is possible to specify if it should be able to rotate the symbol, how far it should be between different rotations etc. The most important function is, however, the creation of new symbols which can be used for the symbol recognition engine 4 . A symbol is created by drawing its representation on a canvas. One important issue is that the user must draw the symbol in the same order as the end-user should do it. As soon as the drawing is done the result is sampled to a specified number of equidistant sample points. The symbol is also translated and scaled. By creating different sets of symbol files it is possible to use different templates depending on different situations which makes recognition more reliable. The Symbol Recognizer is an application to test how well the recognition of the symbols work in a desktop setting (see figure 2.4). It uses the same symbol recognizer as the cellular phone application which assures its use as an indicator of how well symbols created with the framework will be correctly identified on the cellular phone platform. One important feature is that the application supports dual input, it can be controlled by any device that the system recognize as a pointer device, as well as with an Anoto digital pen 5 .

2.2.2

Cellular Phone Application

The Kartago System for Cellular Phones is an implementation which uses the framework and shows that the framework is a viable solution to use. Further, it demonstrates various of the plug-ins that have been developed. The application presents the user with a map which has hot-spots 6 . The functionality of the system depends entirely on what kind of functionality the current map has been equipped with. The functionality of a map is defined in a XML-file which is created using the development platform. In the most primitive application, a map has no additional functionality which 4 The

symbol recognition is template-based, and uses an algorithm which permits the detection of complex symbols. 5 For the Anoto digital pen a driver had to be written which in real-time processes the streaming data from the pen. 6 Hot-spots are areas on the map which are augmented with functionality.

14

Figure 2.4: Recognizing a symbol. The figure in the lower-right corner is the one drawn by the user. The template of the identified symbol is shown in the upper-left corner. In the lower-left corner the identifier of the identified symbol is shown. means that the application will only show the map with no possibility to interaction. When hot-spots are introduced the scenario changes considerably. The plug-in library provides various functions which can be connected to the map. The plug-ins usually extend the use of the application, and provide a method to tailor applications to different scenarios. There are, for example, plug-ins to recognize military symbols and attach information to them, draw on the map, send SMS/MMS- messages in various ways, play sounds etc.

2.2.3

Digital Pen and Paper

For this project a digital pen which uses Anoto technology [8] has been employed. There are several approaches to how the use of pen technology

2.2. The Development Framework

15

can be used together with computer systems. Tablet PCs have a touch screen, when doing gestures on the screen with a stylus the pointer on the screen follows the stylus’ movements. Taking a step toward the solution used in this thesis it is worth mentioning systems that use paper together with a digital pen, but where the paper has to be placed on top of a special device which detects what the user is doing with the pen. Anoto uses an approach which only requires a specialized digital pen together with a specially prepared paper [8]. This approach provides an extremely easy way to interface computer applications [11]. The digital pen communicates through bluetooth. This meant that a driver for a cellular phone 7 , as well as, a driver for desktop computers were developed. The optical sensors in the pen recognize where on the paper the user has made a gesture, with what pressure, and when the gesture was made. For this project a digital pen which uses streaming was used. Earlier versions used a tick-box to signal to the system to send the saved strokes to a device. Further, older systems had a very slow response time, which implied that the system could not be used for a setting where responsiveness is important. The digital pen used in this project does not have any of the two problems its predecessors had. Instead it demonstrates good responsiveness together with high-accuracy localization on the paper. It stands, however, clear that the digital pen, in its current form, is not suitable for military missions since it is a sensible piece of equipment. The digital pen used in this project needs a specially prepared paper to be able to function. The preparation of the paper consists in printing a pattern on the paper. The pattern consists of small dots with a nominal spacing of 0.3mm [8]. These dots form a unique pattern which is interpreted by the digital pen. There are various types of patterns, some are used for streaming data, whereas others are used for batch-processing of data. The pattern covers a region greater than Europe and Asia combined [8] which means that there are good possibilities to create applications with very big paper formats.

7A

bluetooth service had to be developed

16

Chapter 3

Implementation This chapter introduces the development platform and discusses some of the technologies used. Symbol recognition is a complicated area. Therefore, in this chapter a description of how the symbol recognition was implemented in this project is presented. XML has been used extensively in the project, and therefore a short explication to how it has been used is included.

3.1

Development Platform

The project is implemented using Java in two different flavours. The main system, which runs on a cellular phone, is programmed in Java 2 Micro Edition 1 whereas the rest of the programs are programmed using Java 2 Standard Edition 2 .

3.1.1

Java 2 Micro Edition

To better understand how development of Java applications for small devices is done a brief introduction is given in the following text. The J2ME context can be seen in figure 3.1. 1 From 2 From

know on J2ME know on J2SE

17

18

Figure 3.1: J2ME versions

J2ME is a Java version especially designed for small devices. It currently supports a broad range of devices such as PDAs, TV set-top boxes, cellular phones, printers and embedded systems. The differences between J2ME and J2SE are important since it is not possible to directly compile code written for J2SE on J2ME. Within J2ME there are differences between what can programmatically be done depending on the deployment platform. Different devices may use different parts of the framework (as can be seen in figure 3.1. These are called configurations. The basic functionality of the development platform is, as has been mentioned earlier, similar to J2SE. However, there are some differences given that J2ME runs on less powerful hardware and therefore has more limitations. One important aspect of programming in J2ME is that although many of the same classes exist in J2SE, they usually do not have the same methods. Important differences can be found in, for example, the String and Vector classes. Further, depending on deployment platform there may even be differences within J2ME when it comes to what functionality the classes provide.

3.1. Development Platform

3.1.2

19

CLDC and MIDP

Since applications that use J2ME potentially have limited I/O possibilities the platform uses a different approach to GUI-programming. The programming model is depicted in figure 3.2.

Figure 3.2: Programming model for cellular phones In a normal desktop environment users are used to work with systems that support multiple windows open at the same time. While J2SE provides the Swing and awt packages for GUI-programming, this is not true for the J2ME development environment. For the platform used in this project, a Java-enabled Sony Ericsson cellular phone, the lcdui package is central to GUI- programming. The Connected Limited Device Configuration or CLDC is a J2ME configuration which is designed for devices with limited memory, low processing power and primitive graphical capabilities [12]. More advanced devices (such as advanced PDAs and TV top-set boxes) use a different profile, the Connected Device Profile (CDC). Currently the CLDC and CDC are the only two J2ME configurations, and together they lay out the foundation of the J2ME framework [13]. Although applications developed for the two different platforms are both bundled as J2ME applications there are fundamental differences. The CLDC uses KVM 3 while the CDC uses JVM 4 , which means that the CLDC has a smaller memory footprint, at the expense of being more limited. The CLDC used for this project is version 1.1 which incorporated various enhancements compared to its predecessor CLDC 1.0. The most im3 Kilobyte 4 Java

Virtual Machine Virtual Machine

20

portant feature in the new version for the work with this project was the support for floating point, in earlier versions only integers were supported. Mobile Information Device Profile 2.0 5 is as of writing the last implemented version of the profile. MIDP is a supplement to the CLDC which provides the parts of the system libraries that are not provided by the CLDC profile. Additionally, it provides the possibility to create graphical applications, use persistent storage etc. The MIDP is targeted at cellular phones, and the corresponding profile for PDAs is called PDAP. [13] There are important differences between MIDP 1.0 and MIDP 2.0, in the later it is possible to make applications run in fullscreen mode [14]. This feature has extensively been used in the project since cellular phones have quite small screens. Other features that have been used in the project and are new for MIDP 2.0 are, for example, sound support and support for other media. The development of the profile facilitates development of advanced applications. When programming for a cellular phone the basic entity is a MIDlet which is a MIDP application. MIDlets provides a limited way of laying out GUI components and the navigation is based on switching between different displayable items [15]. In addition to the API provided by MIDP/CLDC there are optional packages which give further support for hardware, communication, cryptology etc.

3.2

Symbol Recognition

One important component for this project is the symbol recognition plugin. It uses a symbol recognition package, which was developed, and added to the framework in order to allow recognition of military symbols that are drawn with any kind of input device6 . The symbol recognition package is template-based which means that for each symbol a template is used to identify each symbol. This is an important feature since it is hard to find gestures that are easy to memorize [16]. Further, it is possible to claim that the use of symbol recognition should be done to maintain 5 From

know on MIDP. device can here be interpreted as any device who’s input can be transformed into an ordered set of points. 6 Input

3.2. Symbol Recognition

21

Figure 3.3: Elastic matching (left) versus proportional matching (right)

current work-flow, in this sense, having to learn how to draw totally new symbols to represent already known ones would be undesirable. Further, the representation on a paper map would no longer be maintained which would be an important drawback. When a user draws a symbol the strokes are saved as a sequence of positions which are sampled to be equidistant. A similar approach has been used in the SHARK2 system [17]. This approach stands in contrast to the one used by Tappert, who uses elastic matching [18]. Whereas proportional matching has a one-to-one relation between the sample points, the elastic matcher, depending on lookahead, is allowed to take the closest point from the template (see figure 3.3). The problem associated with Tappert’s approach for this thesis is that the patterns are relatively similar, which would cause the elastic matcher to do erroneous assumptions about how different the pattern actually is from the template. To be able to compare the reference pattern with the pattern drawn by the user it is necessary to scale the pattern so that the two patterns have the same size. The scaling is done by observing that the longest side of a

22

bounding box around the drawn pattern should always remain constant. To be able to draw patterns anywhere in a coordinate system it is necessary to translate the drawn pattern to the origin. When the drawn pattern has passed the before mentioned steps, the distance between the drawn pattern and the reference pattern in each point is compared and summed up. Relying on the belief that the start- and endpoints when drawing are drawn with greater care than in between is taken into account as a weight for the total distance. The symbol language supports rotation, thus permitting a symbol to be drawn rotated 7 . The plug-in for the cellular phone generates rotated templates which are added to the symbols library if a symbol should be allowed to be rotated. There are various approaches to symbol recognition, some approaches are easy to implement but can give poor results, other are very hard to implement, and the symbol recognition does not necessarily improve substantially. When developing a symbol recognizer for a cellular phone various aspects had to be accounted for. A cellular phone has limited computational power, and the support for floating point operations is generally based on emulation8 . Further, the J2ME maths library is limited compared to the J2SE library, for example, it does not provide support for the arcus functions. Therefore, for the symbol recognition in this project, it was necessary to implement a couple of arcus functions. Rubine [19] uses features of the collection of points to recognize patterns. The symbol recognizer has been used in many applications, for example the SATIN toolkit uses the gdt implementation of Rubine’s recognizer [20]. Rubine’s approach has several disadvantages, it is possible to analytically find cases when the algorithm will not be able to distinguish one symbol from another. Further, the algorithm requires several trigonometric operations which are slow on a small device. This is an issue on devices with limited computational power. Reducing calculation time by using look-up tables implies incremented memory requirements and reduced precision. A more pragmatic reason not to use Rubine involves that it is not template-based, which means that the system has to be trained. A template-based approach 7 There

are several reasons to not activate rotation. It decreases the recognition rate, and it may introduce ambiguity between different military symbols. 8 It is probable to see cellular phones with Floating Point Units (FPU) as these devices are getting more advanced.

3.2. Symbol Recognition

23

eliminates the need to train the system, instead each user must learn how a symbol is represented. For the recognition of military symbols some of the features would become meaningless since the military symbols can be considered quite similar. It is even possible to analytically construct symbols that are completely ambiguous to the Rubine recognizer’s feature set. The dependency of a feature such as initial stroke angle means that if the user uses an inappropriate angle of attack when starting the pen-gesture it could directly lead to an unintended response from the system. However, adding some of the features proposed by Rubine would probably improve the recognition rate even further, especially for rotated symbols. How input is represented in different systems can vary, for example, the SATIN system [20] saves the information as (x,y,t) tuples. A system which takes into account time might not be usable in certain settings. A soldier on the field wearing thick gloves will probably not draw as fast as he would do sitting at a desk. Instead of taking into account when a certain stroke was generated, the order of the strokes is considered. This approach has the additional benefit that order can be implicitly saved, which means that no extra data storage is necessary for the information. The downside of a system which takes into account the time-aspect or order is that how the symbol was drawn becomes important, as opposed to a system which only studies the final result of the drawing. Most military symbols are compound symbols, which means that they can be seen as various separate symbols. One approach to recognize compound symbols could be to incrementally recognize each part of the symbol. This could be achieved by on pen-up events triggering a timer which would force the user to continue drawing within a specified amount of time in order to recognize the following strokes as part of the initial symbol. The approach in the implemented symbol recognizer instead looks at the entire symbol when it is drawn and ignores pen up and pen down which results in a continuous symbol (see figure 3.4). The downside with this approach is that the representation of large, and detailed symbols require a higher sample rate than what would be necessary if the recognizer would use basic primitives to create the symbols. The benefit of the approach is that each symbol gets its unique representation, and there is no need to take into consideration where a detected symbol part is located compared to the rest of the symbol’s parts. Further, using a set of primitives to create complex

24

Figure 3.4: Continuous recognition (lower path) versus incremental recognition (upper path) symbols would require a greater effort from the user in order to be able to create new symbols.

3.3

Symbol Format

As has earlier been discussed a template based approach toward symbol recognition has been chosen. This implies that a prototype of the symbol must be represented in some form. The system provides support to load symbols from a XML-format. In an effort to use similar technologies where applicable the XML-format was chosen over alternative representations. A symbol is currently represented by various attributes, such as an image file, a sound file, and a ordered set of points defining the symbol. For other uses it might be necessary to adapt the format to support different attributes.

3.4

Use of XML in the Project

One important component in the system design is the use of a XML-format to represent functionality, as well as, basic entities. Most of the functionality of the cellular application is controlled by XML-files. The entire behavior of the system is determined by a XML-file

3.4. Use of XML in the Project

25

which defines which functions should be launched when specific areas have been clicked on. Further the XML-file also contains information on where to locate the background image that should be used. The structure of the basic XML-file is a general description of the application followed by a set of active elements which in turn specify functions. Active elements are the basis for the click-and-run approach. An active element can, for example, be a path which is located somewhere on the paper. Connected to the enclosing area of the path a set of functions, with their corresponding arguments, can be found. The functions are separate components that should be dispatched when clicking on the corresponding area. It is up to each plug-in developer to require the arguments needed to be able to implement the desired functionality. One example of when the use of an argument could be handy is when a predefined message (for example using SMS/MMS) should be sent to a predefined number. One important benefit of using XML is that it permits the separation of the functionality of the paper from the graphical representation of it. It is, for example, possible to restrict what operations different users are allowed to execute by providing different XML-files. This way it is possible to create applications where one user is able to draw and modify the functionality of the paper, while another only can see the changes. This is an example of how the use of separated user interfaces come into play, since two different views of the same information can be created [21]. It is also a powerful way to reduce the information the system provides to the amount needed to perform the required tasks. The Open Geospatial Consortium provides a standard for geospatial R [22]. The standard dictates a and location based services called OpenGIS R has specific format to represent geometrical data. Whereas the OpenGIS support for most of the situations where some representation of geospatial data should be used it was not optimal for this project. The main reason is that the standard is very extensive, and only a small percentage of the specification would be used. Further, this application provides additional functionality which requires its own representation. This specialization called for the development of the XML-format which is currently used.

26

Chapter 4

Conclusion This thesis was set to investigate in which way it is possible to support the use of an online paper-driven computer system in military work-practice. Several important aspects were detected and a system which to a highdegree complies with the needs’ of the military was implemented. Further, the implementation suggested that the same kind of technology could easily be applied to other areas within the field of paper-driven computer systems. Problems similar to the ones the military experience when trying to use computers in their daily work can also be applied to areas such as, firefighting, surveying, waiting on customers etc. The framework presented in this thesis was first intended for paper-driven map applications. During the development of the system it has, however, become clear that the same technology employed for this solution is also viable for other solutions. The efficiency when it comes to developing new applications with this framework makes it an interesting solution for all applications that may benefit from paper and pen as its interaction device. The support for gesture recognition through the symbol recognition package is another utility which can be used in many different settings and can make the work with the computer system more transparent to the user. The symbol recognizer has a very high accuracy rate, which makes it suitable for recognition of pen gestures even in difficult settings. The use of paper and pen as interaction device changes various concepts of how computers may be used. The possibility 27

28

to have a cellular phone in the pocket, and use its services without the need to consult its display is a powerful concept which integrates computers further into everyday life. Soldiers with no computer experience could use the cellular application to its full extent without having to know anything about the underlying technology. Whereas, it can be argued that this implies lower training costs it also implies a significant improvement in workpractice. It is known that in situations of great pressure it is common that a regression toward well-known response patterns occurs. This suggests that the methods soldiers use in the battlefield should be as well-known to them as possible. In this sense pen and paper are natural and powerful tools. Whereas the cellular application in this project uses some of the technologies currently available, there are many other that are interesting within the military field. It is, for example, possible to connect a GPSreceiver to the cellular phone and thereby be able to monitor the soldier’s progress, sending customized warning messages depending on the location of the soldier etc. The need for context-awareness in pervasive computing is disccused by, among others, Satayanarayanan [23]. Another possibility is to improve communication between the soldiers. Even more interesting is the possibility to connect cellular phones to a critiquing system which gets its input from a centralized server which merges the observations of all the connected cellular phones, and sends out information to support the decision-making in the battlefield. The development of digital pens has made them a viable choice for applications in which it is crucial to have a high-accuracy on the readings. For military applications it is however necessary to make the digital pens work in hard conditions where dust, and water are part of everyday use. The need for further studies of how the system would work in a real-life setting is essential to be able to further improve the applications, such a study could also give some clues on what to focus on to further develop the concept of paper-based computing.

Chapter 5

Drawing on Paper maps: Robust On-line Symbol Recognition of Handwritten NATO Symbols using a Digital Pen and a Mobile Phone 5.1

Abstract

We present a robust on-line handwritten recognition system that recognizes NATO symbols drawn on paper maps using a digital pen and a mobile phone. Our user evaluation shows that very high accuracy is achieved. Furthermore, our proposed system has low latency running on off-the-shelf mobile phones. The recognition system is part of a research effort in augmenting existing work-practices of soldiers in the field with very lightweight 29

30

computer support. Keywords: pen-gesture recognition, mobile phones, digital pen, paperbased user interfaces, ubiquitous computing.

5.2

Introduction

There is still extensive use of ordinary paper products within the military and advanced computer systems are sometimes unused [24]. The use of computer support is, naturally desirable, given satisfactory cost and mobility characteristics. For the soldier in the field, weight and cost can be imperative, design parameters that a computer system such as a communications, planning, or critiquing system needs to overcome. The current and ongoing computerization of physical paper provides us with a new technical platform for such systems. In this project, as part of the Kartago research framework, we explore the development of a papermap-based computer support tool providing very lightweight and low cost characteristics. The Kartago research framework focuses on the traditional paper map as the central hardware device and explore the augmentation of paper as a means of providing interactive computer support. Soldiers in the field use paper maps extensively. Current work-practice involves using paper maps for orientation, observation and collaboration. Observations are marked with a military symbol language on the paper map and are also reported to headquarters. The ability to recognize drawn symbols such as NATO symbols is essential in a paper-map-based computer system for soldiers in the field. Symbol recognition is a non-trivial area of research, further challenged by the military domain and by use in the field. In this paper we present our work in enabling symbol recognition for the Kartago System in its current form, a printed paper map used with a digital pen (Anoto) and a mobile phone as the system components. This system provides pen-based interactivity on a paper map to a computer system running on a mobile phone via wireless communication. The system fits in a soldier’s leg-pocket and it retains all positive aspects of traditional paper maps. Our user evaluation results show that very high accuracy (98% before

5.3. The Kartago Project

31

training, 100% after minimal training) is achieved in the Kartago symbol recognition running on off-the-shelf mobile phones. Consequentially, we are confident that the symbol recognition part of the Kartago system will provide sufficiently ease-off-use in the field.

5.2.1

Symbol Recognition in the Field

Armies invest enormous sums in technology to keep up with the technological development. Yet, in the field, it may still not be to bold a statement to say that paper still rules. Whereas paper maps have several advantages over ordinary computerized systems, they do not provide automated connectivity to other military systems. One important leap toward the next generation of soldier equipment is the digitalization of the soldiers’ work. For instance, when a soldier wants to report an observation it could be done by drawing with a pen on a paper map. One of the issues with such a system is how to support the recognition of symbols. Symbol recognition for military applications requires a different approach. One important aspect is that the input data can vary considerably in quality due to the special conditions in which military operations take place. For example, how well a symbol is drawn depends to great extent on what input device is used, if the symbol is drawn by a person who stands or sits, if the canvas is supported by a hard surface underneath or not. This suggests that a solution must be robust and permissive to bad data. Although symbol recognition is not a new phenomena, it unveils the full potential of a mobile military computerized system. Annotations on a map can be captured and used locally, or by remote systems, for instance, by command and control systems.

5.3

The Kartago Project

Kartago is a research project aimed at a mobile paper-based or paper-like interactive map platform. The aim is to provide a platform for military operations which is usable in the field for tasks such as planning, simulation, communication, and documentation. It is an alternative approach to digi-

32

Figure 5.1: Drawing NATO symbols on a paper map with a digital Anoto pen.

talization of map-related tasks which focus on augmenting the traditional paper map as means of digitizing map-related work practice. The research project envisions that paper or a paper-like material (such as a flexible screen) can be the central artifact in an interactive-map system where networking, communication and user interactivity can be added on a level as advanced as that of an everyday computer. The goal is to achieve a printable leg-pocket sized system with a mapsized interface (folded and carried in hand or in a leg-pocket) with symbolbased interaction, and information/communication task management (see Figure 5.2). A highly relevant part of the Kartago platform is symbol management and symbol recognition for the use of hand-written symbols in interactive systems such as planning and critiquing systems. Being able to use the symbol language already in place is an important part of a mobility platform for the military.

5.4. Interactive Maps in the Field

33

Figure 5.2: The system in its natural workplace.

5.4

Interactive Maps in the Field

In a regular computer-enabled staff environment, a multitude of systems can provide decision support for staff members. Mixed-initiative planning systems can help staff officers analyze a course of action with respect to physical limitations, rules of engagement or other known constraints (e.g., [25] [26][27][28] [29]). Collaboration systems can analyze the interactions between multiple commanders [30] and thereby help them synchronize their actions. The usefulness of these systems could be extended greatly if they allowed for field officers to access the same information that is available in a regular staff environment through the use of pen and paper . Further, the availability of observations from the field can increase the usefulness of these systems even further.

34

5.5

System Overview

The outlined solution uses paper maps together with a cellular phone, and an Anoto pen. The pen deduces where the user is drawing by a special pattern, which is printed on the paper map. This technology allows the soldier to have a full-size paper map which can be folded and put in a pocket. The cellular phone used is a normal bluetooth cellular phone which supports J2ME. Altogether this hardware configuration assures that the hardware is easily wearable. This paper focuses on the pen-gesture recognition component of a larger system intended for military use. Figure 5.3 illustrates the high-level relationship between the paper maps, the digital pen, the cell phone, and an optional server.

Figure 5.3: Overview of the pen-gesture recognition component. The user draws pen-gestures with digital pen on a paper map. The pen is connected wirelessly to a cell phone, which in turn is connected to a server over the mobile network (e.g. GSM, EDGE, 3G, etc.). Within the framework for Kartago System for Cellular Phones, the symbol recognition is one important component. In a case implementation the symbol recognition allows the soldier to draw a symbol, and get it recognized. When the symbol is recognized the soldier is allowed to position the symbol and can, optionally, add information which can be used by a critiquing system. The information collected during this work is attached

5.6. Requirements

35

to a XML-representation of the map which can be sent through SMS/MMS, or web services to other soldiers, or to headquarters.

5.6 5.6.1

Requirements Support Complex Pen-gestures

Military symbols are typically complex compound symbols consisting of a combination of various pen-strokes, see Figure 5.4 for examples. Therefore the pen-gesture recognizer must be able to handle multiple pen-strokes.

5.6.2

Robust

Since the pen-gesture recognition system is intended to be used in the field, it must be robust to distortions and maintain high-accuracy.

5.6.3

Support Low-powered Devices

Yet another difficulty is the limited processing power of cell phones which limits the amount of calculations that can be performed. Also, mobile phones typically lack a floating point unit (FPU).

5.6.4

Easy to learn

The recognition system should work with a minimal amount of training required by the users.

5.6.5

Easy to maintain

Many recognition systems are based on training-data where several writing samples for each symbol are obtained into a database. A side-effect of the data-driven approach is that it is harder to introduce new symbols since it is not enough to submit one ideal template symbol (e.g. from a handbook of symbols). Instead several symbols need to be written using a specialized software. As such, a requirement is also that it is easy to maintain the

36

symbol set and add new symbols as needed with minimum effort required from the users.

5.7 5.7.1

Recognition Symbol Language

Cooperation between different countries’ armies creates new challenges to the world of symbol representation. Whereas NATO has one symbol language, individual countries may use a different system. In order to support different military symbol languages a special symbol format was developed, which permits the symbol recognition engine to be loaded with different symbol languages. Figure 5.4 shows a selection of the symbols the system recognizes. The symbol format is based on XML, which is a natural choice given that the system, in which the symbol recognizer is one part, is XML-driven. The system provides support to load symbols expressed in the XML-format. A symbol is represented by various attributes, such as an image file, a sound file, and a ordered set of points defining the symbol. The symbol recognizer only require the ordered set of points, together with information regarding whether the symbol could be rotated, and how far it should be between each rotation of the symbol. The rest of the specifiers are optional and are used by other parts of the application to enhance the user’s experience. The implication of requiring an ordered set of points is that drawing order becomes important. It is therefore necessary for the user to learn in which order a symbol should be constructed.

5.7.2

Recognition Process

Related Work A common pen-gesture recognizer is the Rubine recognizer [19]. The Rubine recognizer is the “classic” statistical linear machine that extracts a feature vector from the input signal and attempts to find the most similar class by statistical inference. The features in the Rubine recognizer

5.7. Recognition

Figure 5.4: Selection of primitives and NATO symbols.

37

38

are, for example, initial angle, and the length of the stroke [19]. The Rubine recognizer has been integrated into many toolkits, for instance SATIN [20]. The Rubine recognizer approach is practical for a few pen-gestures but is increasingly less robust as the number of symbols increase. For instance, the dependency of a feature such as initial stroke angle means that if the users initiated an inappropriate angle of attack when starting the pen-gesture it could directly lead to an unintended response. Analytically another downside with the Rubine recognizer is the possibility to create two geometrically distinct symbols that are completely ambiguous to the Rubine recognizer’s feature set. Algorithms The symbol recognition is template-based and the symbol templates are used by the recognizer to identify the user’s intended symbol. User input is saved as ordered strokes and is equidistantly sampled and normalized in scale and translation to permit comparison with the reference pattern. The actual symbol recognition is performed when the unknown symbol has passed the preprocessing steps for recognition. For each template symbol, the distance between the pattern drawn with the digital pen and the reference symbol pattern is calculated by the scoring function q (Equation 5.1). To improve recognition accuracy we also give extra weight to the start and end point comparisons between the input signal (drawing trace) and the template pattern. In Equation 5.1 α is the weight of the start and end points. In the summation the distance between each ordered pair is summed up: m X q(x, y) = (1 − α)( kxi − yi k2 ) + αγ(x, y)

(5.1)

i=0

where x and y are two patterns, the subscripted xi and yi components refer to individual sample points, k2 · 2k2 is the L2 (Euclidean) norm and γ is a function defined as: γ(x, y) = kx0 − y0 k2 + kxm − ym k2

(5.2)

5.7. Recognition

39

The recognition engine is based on the concept of proportional matching presented by Kristensson and Zhai [17]. Points are matched on a one-toone basis as opposed to the elastic matching approach which was used in some early handwriting recognition systems [18]. There are three primary advantages to proportional matching in relation to elastic matching [17]: 1. Time complexity of the proportional matching algorithm is linear in relation to the number of sample points. By contrast, elastic matching has quadratic time complexity. 2. Proportional matching is easier to implement. 3. Proportional matching is easier to work with analytically. For instance, to compare elastic matching scores against each other, the scores need to be normalized. However the normalization procedure for elastic matching is non-trivial [31]. Other approaches to recognizing on-line writing (mainly data-driven handwriting recognition) have been proposed (e.g. [32, 33], also see Tappert et al. [34] for an extensive review). Since our system is template-based rather than data driven new symbols can be added by simply inserting an ideal template of a symbol (e.g. from a handbook) into the recognizer’s set of reference symbols. To make proportional matching work well in a military setting we found it necessary to introduce a couple of modifications of the concept. For military applications in the field, special care has to be taken to how a stroke should be represented. Several systems save strokes as (x,y,t) triplets. In a military setting a symbol recognition that depends on movement dynamics is probably not reliable. A soldier in the field wearing thick gloves will probably not draw as fast as he would do sitting at a desk. Furthermore, expert users have dramatically different movement patterns than novice users or users under considerable stress who may suffer considerable hand tremor. As a result, instead of taking into account when a certain stroke was generated, the order of the strokes is considered. This increases the tolerance toward disruptions in the input procedure. Furthermore, given the risk of unwanted pen-up events the representation of the symbols discards any pen-up and pen-down data. Figure 5.5 shows the result of how a symbol is represented internally, when pen-up and pen-down is discarded. As has earlier been mentioned, most military symbols are compound symbols, which means that they can be seen as various separate symbols. One approach to recognize compound symbols could be to incrementally recognize each part of the symbol. This could be achieved by on pen-up

40

Figure 5.5: Graphical representation (left) and internal representation (right) of two symbols.

events triggering a timer which would force the user to continue drawing within a specified amount of time in order to recognize the following strokes as part of the initial symbol. The approach in the implemented symbol recognizer instead looks at the entire symbol when it is drawn and ignores pen-up and pen-down which results in a continuous symbol. The downside with this approach is that the representation of large, and detailed symbols require a higher sample rate than what would be necessary if the recognizer would use basic primitives to create the symbols. The benefit of the approach is that each symbol has a unique representation, and there is no need to take into consideration where a detected symbol part is located compared to the rest of the symbol’s parts. Further, using a set of primitives to create complex symbols would require a greater effort from the user in order to be able to create new symbols. As is always the case when developing recognition-based systems there are multiple conflicting dimensions of goodness: high accuracy, ease-of-use, fast recognition response time, etc. In this application we prioritized high accuracy (robust recognition) at the cost of initial ease-of-use.

5.8. Evaluation

5.8

41

Evaluation

There are many research questions when developing a new recognition system for soldiers in the field. In this paper we have focused on the most fundamental questions: 1. Is recognition robust enough to be practical? 2. Can the system recognize symbols fast enough to run on off-the-shelf mobile phones?

5.8.1

Participants

Fifteen participants were recruited. They were told to draw the twenty compound pen-gestures depicted in figure 5.4. The ages ranged from 27 to 50 (average 40). 11 were female and 4 were male.

5.8.2

Material

The participants were asked to reproduce 20 pen-gestures (Figure 5.4). Each pen-gesture was re-sampled into 100 equidistant sample points.

5.8.3

Apparatus

An Anoto pen model Maxell DP-201 was used as the digital pen. The Anoto pen was used on paper sheets with Anoto’s background pattern imprinted. Using the background paper the Anoto pen transmitted absolute and relative positional coordinates to a desktop computer executing the recognizer software. The performance tests were conducted on a Sony Ericsson P990i mobile phone with a 250 MHz 32-bit ARM9 CPU.

5.8.4

Procedure

The experiment was run in two sessions. In the first session participants were shown how to draw the symbols in Figure 5.4 and afterwards instructed to reproduce them. The second session was performed after users had been allowed to practice drawing the symbols that were difficult to, or had failed to draw, in the first attempt.

42

5.8.5

Results

Error As can be seen in Table 5.1 the error rate at the first test was well over the acceptable level (2%). After minor training the error rate reduces to zero. The results show that after minimal training very high accuracy is achievable with our proposed NATO symbol recognition system, and it is clearly practical for prototype testing in the field. Table 5.1: Error rates before and after training First Test Second Test Error Rate 2% 0%

Latency Latency was calculated by recognizing five randomly chosen symbols from the symbol set in Figure 5.4. The average latency was 589 ms. Latency varied from 492 ms up to 703 ms. The variation can be attributed to how complex the input data was to process (re-sample and normalize). We want to emphasize that the latency data points given above are of a preliminary nature since they were obtained without using any pruning or indexing mechanisms. We are in the process of developing a filter that prunes out unlikely patterns early in the recognition process to reduce latency.

5.9

Conclusion

A symbol recognizer capable of recognizing for instance hand-drawn NATO symbols is an essential component in a paper-map-based computer system supporting information and communication tasks for soldiers in the field. This paper show that the presented Kartago symbol recognizer provides a very robust solution. High accuracy is achievable using the symbol recognizer after minimal training. The Kartago system illustrates that interactive and mobile computer systems with lightweight and low-cost characteristics can be provided by

5.10. Acknowledgements

43

augmenting paper maps. Combined with technologies included in off-theshelf mobile phones such as GPS, Bluetooth and GPRS/3G we can support advanced information tasks in the field. The fact that the system is wearable while still providing the use of fullsize maps suggests that Kartago is a usable tool in military work practice. The work presented here is a step towards real-time critiquing and computer support in the field. With the positive results of our symbol recognizer we are planing to implement and evaluate the use of the Kartago with a critiquing system, which would allow us to study the use of interactive paper maps in the field using traditional paper maps.

5.10

Acknowledgements

This work was in part sponsored by the Swedish National Defence College, Anoto AB and Santa Anna IT Research Institute AB (SICS Link¨ oping).

Bibliography [1] Mark Weiser. The computer for the 21st century. SIGMOBILE Mob. Comput. Commun. Rev., 3(3):3–11, 1999. [2] Simon Downs. Is it a book, is it a screen, no it’s... graphics and the interface in electronic paper. Digital Creativity, 16(1):31–42, 2005. [3] Xerox. Avilable at http://sandbox.xerox.com/parctab/csl9501/paper.html. accessed 5 december. [4] Anind K. Dey, Peter Ljungstrand, and Albrecht Schmidt. Distributed and disappearing user interfaces in ubiquitous computing. In CHI ’01: CHI ’01 extended abstracts on Human factors in computing systems, pages 487–488, New York, NY, USA, 2001. ACM Press. [5] Mark Weiser. Some computer science issues in ubiquitous computing. Commun. ACM, 36(7):75–84, 1993. [6] Jr. Allan Christian Long, James A. Landay, and Lawrence A. Rowe. Implications for a gesture design tool. In CHI ’99: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 40–47, New York, NY, USA, 1999. ACM Press. [7] Mark A. Livingston and et al. An augmented reality system for military operations in urban terrain. [8] Anoto AB. Available at: www.anoto.com. accessed december 5, 2006. 44

BIBLIOGRAPHY

45

[9] Joseph LaViola, Randall Davis, and Takeo Igarashi. An introduction to sketch-based interfaces. In SIGGRAPH ’06: Material presented at the ACM SIGGRAPH 2006 conference, page 18, New York, NY, USA, 2006. ACM Press. [10] Yu David Liu and Scott F. Smith. A formal framework for component deployment. SIGPLAN Not., 41(10):325–344, 2006. [11] Mario Kolberg. Programming a pvr with pen and paper. Consumer Communications and Networking Conference, 2006, 2:1332– 1333, 2006. [12] Sun. Available at: java.sun.com. accessed november 28, 2006. [13] James White. An introduction to java 2 micro edition (j2me); java in small things. In ICSE ’01: Proceedings of the 23rd International Conference on Software Engineering, pages 724–725, Washington, DC, USA, 2001. IEEE Computer Society. [14] Christopher Williams and Mark Burge. Midp 2.0 changing the face of j2me gaming. In ACM-SE 42: Proceedings of the 42nd annual Southeast regional conference, pages 37–41, New York, NY, USA, 2004. ACM Press. [15] Ian Utting. Problems in the initial teaching of programming using java: the case for replacing j2se with j2me. SIGCSE Bull., 38(3):193–196, 2006. [16] Jr. A. Chris Long, James A. Landay, Lawrence A. Rowe, and Joseph Michiels. Visual similarity of pen gestures. In CHI ’00: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 360–367, New York, NY, USA, 2000. ACM Press. [17] Per-Ola Kristensson and Shumin Zhai. Shark2: a large vocabulary shorthand writing system for pen-based computers. In UIST ’04: Proceedings of the 17th annual ACM symposium on User interface software and technology, pages 43–52, New York, NY, USA, 2004. ACM Press.

46

BIBLIOGRAPHY

[18] Charles C. Tappert. Cursive script recognition by elastic matching. IBM Journal of Research and Development, 26(6):765–771, 1982. [19] Dean Rubine. Specifying gestures by example. In SIGGRAPH ’91: Proceedings of the 18th annual conference on Computer graphics and interactive techniques, pages 329–337, New York, NY, USA, 1991. ACM Press. [20] Jason I. Hong and James A. Landay. Satin: a toolkit for informal inkbased applications. In UIST ’00: Proceedings of the 13th annual ACM symposium on User interface software and technology, pages 63–72, New York, NY, USA, 2000. ACM Press. [21] Chris Vandervelpen and Karin Coninx. Towards model-based design support for distributed user interfaces. In NordiCHI ’04: Proceedings of the third Nordic conference on Human-computer interaction, pages 61–70, New York, NY, USA, 2004. ACM Press. [22] Open Geospatial Consortium Inc. Available www.opengeospatial.com. accessed november 30, 2006.

at:

[23] M. Satyanarayanan. Pervasive computing: Vision and challenges. IEEE Personal Communications, pages 10–17, aug 2001. [24] David R. McGee, Philip R. Cohen, and Lizhong Wu. Something from nothing: augmenting a paper-based work practice via multimodal interaction. In DARE ’00: Proceedings of DARE 2000 on Designing augmented reality environments, pages 71–80, New York, NY, USA, 2000. ACM Press. ´ [25] Juan Fdez-Olivares, Luis Castillo, Oscar Garc´ıa-P´erez, and Francisco Palao. Bringing users and planning technology together. experiences in SIADEX. In 16th International Conference on Automated Planning and Scheduling (ICAPS 2006), 2006. [26] Marcel A. Becker and Stephen F. Smith. Mixed-initiative resource management: The AMC Barrel Allocator. In Proceedings 5th International Conference on Artificial Intelligence Planning and Scheduling (AIPS-2000), 2000.

[27] Karen L. Myers, Peter A. Jarvis, W. Mabry Tyson, and Michael J. Wolverton. A mixed-initiative framework for robust plan sketching. In Thirteenth International Conference on Automated Planning and Scheduling (ICAPS-03), 2003. [28] John L. Bresina, Ari K. J´ onsson, Paul H. Morris, and Kanna Rajan. Mixed-initiative planning in MAPGEN: Capabilities and shortcomings. In Proceedings of the IJCAI-2005 Workshop in Mixed-Initiative Planning and Scheduling, 2005. [29] Stephen F. Smith, David W. Hildum, and David R. Crimm. Comirem: An intelligent form for resource management. IEEE Intelligent Systems, 20(2):16–24, 2005. [30] Karen Myers, Peter A. Jarvis, and Thomas Lee. CODA: Coordinating human planners. In Proceedings of the 6th European Conference on Planning (ECP-01), 2001. [31] A. Marzal and E. Vidal. Computation of normalized edit distance and applications. IEEE Trans. Pattern Anal. Mach. Intell., 15(9):926–932, 1993. [32] E.J. Bellegarda, J.R. Bellegarda, D. Nahamoo, and K.S. Nathan. A fast statistical mixture algorithm for on-line handwriting recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 16(12):1227–1233, 1994. [33] Daiki Okumur, Seiichi Uchida, and Hiroaki Sakoe. An hmm implementation for on-line handwriting recognition - based on pen-coordinate feature and pen-direction feature. icdar, 0:26–30, 2005. [34] C. C. Tappert, C. Y. Suen, and T. Wakahara. The state of the art in online handwriting recognition. IEEE Trans. Pattern Anal. Mach. Intell., 12(8):787–808, 1990.

Avdelning, Institution Division, Department

Datum Date

HCS, Dept. of Computer and Information Science ¨ 581 83 LINKOPING Spr˚ ak Language

Rapporttyp Report category

2 Svenska/Swedish

4 Engelska/English

2 Licentiatavhandling

4 Examensarbete 2 C-uppsats

2 D-uppsats ¨ 2 Ovrig rapport

2

2007-02-14

ISBN

— ISRN

LITH-IDA-EX--07/004--SE Serietitel och serienummer ISSN Title of series, numbering —

2

URL f¨ or elektronisk version http://www.ep.liu.se/exjobb/ida/2007/dd-d/004/

Titel

Ett ramverk f¨ or mobilt pappersbaserat datoranv¨ andande

Title

A Framework for Mobile Paper-based Computing

F¨ orfattare Author

Tomas Sylverberg

Sammanfattning Abstract

Military work-practice is a difficult area of research where paper-based approaches are still extended. This thesis proposes a solution which permits the digitalization of information at the same time as workpractice remains unaltered for soldiers working with maps in the field. For this purpose, a mobile interactive paper-based platform has been developed which permits the users to maintain their current work-flow. The premise of the solution parts from a system consisting of a prepared paper-map, a cellular phone, a desktop computer, and a digital pen with bluetooth connection. The underlying idea is to permit soldiers to take advantage of the information a computerized system can offer, at the same time as the overhead it incurs is minimized. On one hand this implies that the solution must be light-weight, on the other it must retain current working procedures as far as possible. The desktop computer is used to develop new paper-driven applications through the application provided in the development framework, thus allowing the tailoring of applications to the changing needs of military operations. One major component in the application suite is a symbol recognizer which is capable of recognizing symbols parting from a template which can be created in one of the applications. This component permits the digitalization of information in the battlefield by drawing on the paper-map. The proposed solution has been found to be viable, but still there is a need for further development. Furthermore, there is a need to adapt the existing hardware to the requirements of the military to make it usable in a real-world situation. Nyckelord Keywords

pen-gesture recognition, paper-based user interfaces, ubiquitous computing, software engineering, digital pen

50

Copyright Svenska Detta dokument h˚ alls tillg¨ angligt p˚ a Internet - eller dess framtida ers¨ attare - under 25 ˚ ar fr˚ an publiceringsdatum under f¨ oruts¨ attning att inga extraordin¨ ara omst¨ andigheter uppst˚ ar. Tillg˚ ang till dokumentet inneb¨ ar tillst˚ and f¨ or var och en att l¨ asa, ladda ner, skriva ut enstaka kopior f¨ or enskilt bruk och att anv¨ anda det of¨ or¨ andrat f¨ or ickekommersiell forskning ¨ och f¨ or undervisning. Overf¨ oring av upphovsr¨ atten vid en senare tidpunkt kan inte upph¨ ava detta tillst˚ and. All annan anv¨ andning av dokumentet kr¨ aver upphovsmannens medgivande. F¨ or att garantera ¨ aktheten, s¨ akerheten och tillg¨ angligheten finns det l¨ osningar av teknisk och administrativ art. Upphovsmannens ideella r¨ att innefattar r¨ att att bli n¨ amnd som upphovsman i den omfattning som god sed kr¨ aver vid anv¨ andning av dokumentet p˚ a ovan beskrivna s¨ att samt skydd mot att dokumentet a adan form eller i s˚ adant sammanhang som a ¨ndras eller presenteras i s˚ ¨r kr¨ ankande f¨ or upphovsmannens litter¨ ara eller konstn¨ arliga anseende eller egenart. F¨ or ytterligare information om Link¨ oping University Electronic Press se f¨ orlagets hemsida http://www.ep.liu.se/

English The publishers will keep this document online on the Internet - or its possible replacement for a period of 25 years from the date of publication barring exceptional circumstances. The online availability of the document implies a permanent permission for anyone to read, to download, to print out single copies for your own use and to use it unchanged for any non-commercial research and educational purpose. Subsequent transfers of copyright cannot revoke this permission. All other uses of the document are conditional on the consent of the copyright owner. The publisher has taken technical and administrative measures to assure authenticity, security and accessibility. According to intellectual property law the author has the right to be mentioned when his/her work is accessed as described above and to be protected against infringement. For additional information about the Link¨ oping University Electronic Press and its procedures for publication and for assurance of document integrity, please refer to its WWW home page: http://www.ep.liu.se/ c Tomas Sylverberg

Link¨ oping, February 22, 2007