Tracking On-body Location of a Mobile Phone Kaori Fujinami, Chunshan Jin, and Satoshi Kouchi Department of Computer and Information Sciences, Tokyo University of Agriculture and Technology, Japan [email protected]

Abstract. A mobile phone is getting ready for sensors and aware of various contexts about a user and the terminal itself. The contextual information is utilized to provide an appropriate information/service to a user. In this paper, we deal with six locations of a mobile phone on our body as a context: 1) hung on the neck, 2) jacket pocket (side), 3) chest pocket, 4) front pocket of trousers, 5) back pocket of trousers, and 6) hand. We propose a method for identification with 84 features that characterize specific patterns in changing location in a nonstationary motion, e.g. standing, as well as in moving in a stationary manner, e.g. walking. The results of offline/online performance evaluations showed that about 70% of cases in a nonstationary motion were correctly identified, while in a stationary motion, the accuracy was almost 100%.

1

Introduction

According to a recent study of phone carrying [1], 17% of people determine the position of storing a mobile phone based on contextual restrictions, e.g. no pocket in the T-shirt, too large phone size for a pants pocket, comfort for an ongoing activity. These factors are variable throughout the day, and thus mobile phone users would change the locations in a day. This suggests that the on-body location context is utilized to provide services that are useful in communication, play important roles in sensor-dependent services and in improving usability of the terminal, and contribute to reduce unnecessary energy consumption. In this paper, we propose a method to track the location of a mobile phone. Here, the location is not an exact 3D coordinate, but parts of our body or clothes such as “hand” and “front trousers pocket”. Tracking allows a system to identify current location even in the change while a person is standing still. The work done by Kunze et al. partially shares the goal with us; however, they focus on the period that a target person is in a stationary motion [7], or daily activities with a certain motion [6], e.g. kitchen work. Their method would misidentify a new location until the carrier restarts walking or kitchen work, for example. The rest of the paper is organized as follows: possible applications are presented in Section 2 to clarify the concept and the benefit of sensing location on the body. Then, algorithm design is presented in Section 3. The performance of the algorithm is evaluated in Section 4 in both offline and online manner. Finally, Section 5 concludes the paper with future directions.

2

Possible Applications

2.1 Location-aware Functionality Control A mobile phone will be able to control its functionality based on the placement. According to a recent study of phone carrying [1], generally 30% of men and 40% of women do not always notice the incoming calls and messages, where the rate varies from 15 (0) % (belt case/clip) to 50 (50) %(bag) for men (women). This suggests the notification should be adaptive to the location where a mobile phone is stored. For example, strong vibration would be effective for a mobile phone in a trousers pocket while weak one would be enough for a chest pocket. Also, as Harrison et al. suggested [2], a screen component would be switched off when a mobile phone is not visible, and keypads would be locked to avoid accidental input, e.g. in a pocket with keys. Furthermore, due to the diverse characteristics of electromagnetic propagation across human body. The work done by Shah et al. suggests a need for a placement-aware transmission power control for short range communication, e.g. IEEE 802.15.4.[9]. One may claim that a user has only to specify appropriate notification modality once as people usually store a mobile phone in the same location; however, 17% of the previous survey participants were influenced by contextual restrictions, e.g. no pocket in the T-shirt, too large phone size for today’s chest pants pocket, comfort for an ongoing activity. These factors are variable throughout the day, and thus mobile phone users would change the locations in a day. 2.2 Meta-data for Primary Sensor Readings A sensor-augmanted mobile phone is regarded as a wearable sensor device. A wearable sensor is suitable for recognizing person’s activity and physiological states in an implicit and continuous manner [5, 10]. Also, specific context is obtained from a specific location on our body [5]. However, current research on context recognition by wearable sensors is basically based on an assumption that the sensors are at an intended position, which indicates an application would not perform as designed if the prerequisite is not upheld. In the vision of Human-Probe [10], a sensor-augmented mobile phone would be utilized to capture environmental information throughout daily lives. A dense heat map of a city can be created using a GPS and an embedded thermometer. In a preliminary study, we found the thermometer readings from the neck had showed generally higher than that from a trousers pocket at an air temperature above a body temperature, e.g. 39 ◦ C, while it occurs in reverse at a relatively low air temperature, e.g. 27 ◦ C, due to the effect of body heat propagation. This suggests that the positional information of a mobile phone (thermometer) should be utilized to calibrate the ambient temperature reading. We further found that there was a case where an alert for heat stroke had been underestimated. We measured a Wet Bulb Globe Temperature (WBGT) value, that is often utilized as an index to judge the risk of heat stress, using a dedicated measuring device and found that there had been a period of time in which the value from a neck had been considerably higher than any other location, e.g. a trousers front pocket. A WBGT sensing device is basically designed to measure

the air condition outside our body. These facts imply that a person who happens to put it into a trousers pocket may underestimate an alert-worthy measurement. We strongly argue the need for assuring the sensor output, mobile phone placement in this case, for reliable application’s behavior. Here, the term assurance indicates that an application, e.g. heat stroker alerter, is notified of a change to another situation by an underlying system to take an appropriate action: suspending to avoid undesired behavior, asking a user to check, or adapting to the change by itself. For this purpose, the storing position of a mobile phone is utilized as meta-data for primary sensor data.

3

On-body Localization

3.1 Target Locations and Sensor The location identification in this work is performed on a person’s body: 1) a jacket side pocket, 2) a chest pocket, 3) a front pocket of trousers, 4) a back pocket of trousers, and 5) hung on the neck. These targets were determined based on a survey report [4] and our observation. Although a bag is one of popular locations for storing a mobile phone as shown in [1, 4], we have removed it to simplify the problem in the first trial because a bag has relatively a wide variety of shapes and storing motions that complicate the movement of a terminal. We have adopted an accelerometer to obtain signals that can characterize movement patterns generated by dedicated storing locations. The utilization of an accelerometer also allows the location tracking capability to be easily implemented on a today’s mobile phone. A state of utilization of a mobile phone terminal, e.g. calling, can be supplementally applied to guarantee the terminal is not inside any pocket, which is a subject of future investigation. 3.2 Localization Process Fig. 1 shows a typical waveform of acceleration signals in carrying a mobile phone. A data processing (classification) window (a solid rectangle) is generated with 600 samples of raw acceleration signals sampled at 40Hz. The interval for a window, i.e. 15 sec, has been heuristically determined so that it could be longer than any other interval of storing that was observed during pilot study. Then, the first classification is to identify the mode of a motion of a mobile phone (Fig. 2 C1). Here, the stationary motion mode represents the movement of a mobile phone is generated by the continuous movement of a body, e.g. walking, while the nonstationary motion mode indicates an action of storing/removing a mobile phone into/from a certain storage position, e.g. a chest pocket. C2 classifies five classes because the classification is accurate enough to be done at a time. On the other hands, the storing motion classification for the nonstationary motion mode is complicated. So, we adopted cascaded binary classifiers (C3– C7). The more discriminative class is classified at an earlier stage to maximize overall accuracy (True-Positive Ratio: TPR). A binary classifier C3 is applied to separate a situation where “a mobile phone is in hand” from others. Then, if not in hand, the situation where “a mobile phone is hung on the neck” is the next subject to separate (C4). The rest of the classifiers (C5-C7) are also utilized

in this way. The datasets given to C3-C7 are segmented ones that contain the region of interest in a window. Note that the data are normalized to mean 0.0 and variance 1.0 in a window. Here, a window is slid by four samples (100 msec). To construct the classifiers, a multi-layer perceptron (MLP) has been adopted. The classification in the nonstationary motion mode is regarded as a gesture recognition, where Hidden Markov Model (HMM) is often utilized. We also tested with HMM; however, because of terribly low accuracy (less than 30%), we have determined to classify the storing gestures by the features that characterize the waveform for each gesture.

Fig. 1. Acceleration signals from nonstationary and stationary motions.

Fig. 2. Hierarchy of Binary Classifiers and True-Positive Ratios for Offline Experiments

3.3

Storage Motion Segmentation

As described in section 3.2, each classification is done against a classifiacation window (15 sec); however, a window given into C3 may contain unrelated state of a mobile phone. It is necessary to remove unrelated parts as much as possible to obtain accurate result. We have set an assumption that there are at least three

seconds of stable periods on both sides, which means a mobile phone should be stable at wherever it is stored, e.g. on the desk, in hand, in a chest pocket, before and after storing. The dotted rectangle in Fig. 1 indicates a segmented window. Although this might not always be true, we have utilized this assumption to make extraction of actual storage motion easier and to focus on the classification at this moment. A segment is identified when the period of “storing” is longer than a certain period of time. Here, the state “storing” is determined when the value of a moving standard deviation of one of three axes of acceleration signals gets larger than a threshold value. Extraction of a storing motion from complex movements of a mobile phone is listed as future work. 3.4 Smoothing Classification is done window-by-window basis, which indicates that a consecutive result could be different due to erroneous classification. So, we introduced a smoothing mechanism that takes into account the possibility of transition from one location to another. The “hand” location is considered as a special one that relays the other locations. If a sudden jump from “neck” to “chest pocket”, for example, is observed, it is considered as an error, and the previous location, i.e. neck, is taken over as a final result. 3.5 Classification Features To specify the classification features, we observed various motions that occur when a person store away a mobile phone into a certain place on the body. Then, we have identified five key factors: 1) the posture of a mobile phone, 2) the direction of a motion, 3) the complexity of a motion, 4) the velocity of a motion, and 5) the periodicity of a motion. Classification features were selected to capture the characteristics of these factors. They are not only calculated against raw acceleration data, but also obtained from transformed time series data. The transformation process could be repeated multiple times. Totally, we have adopted 84 features to all the classifiers. High level categories with the number of the features and examples are as follows: – – – – – –

4

First-order value of data (25 features; max of x(z)-axis data, etc.) Point of change in a waveform (18; time to the max of x(y,z)-axis data, etc.) Unevenness (13; max of “moving standard deviation-ed” data of x(y,z)-axis, etc.) Relationship of the 3-axes (16; correlation coefficient of x and y axes, etc.) Linearity of a waveform (4; maximum gradient of z-axis data, etc.) Frequency domain features (8; max of “moving mean crossing 1 ” of x(y,z)-axis, etc.)

Offline and Online Experiments

4.1 Methodology We have performed both offline and online basis experiments. A Bluetooth-based wireless 3-axes accelerometer (WAA-001) [11] was utilized in both cases. The offline experiment was done against five subjects. The data transmitted at 40Hz 1

The number of crossing the mean value in a calculation (sub)window, that approximates frequency component .

were collected on a laptop PC, where LNKNet [8] was utilized to train the seven MLP classifiers and to test with 4-fold cross validation. The subjects repeated every nonstationary motion ten times standing still. For stationary motion, they walked for 30 seconds putting a mobile phone in each storing position. By contrast, in the online experiment, the identification was performed on a PC (CPU: Core Duo 1.66GHz, Memory: 1GB) in a real-time manner. The system’s decision was marked on a picture of a person in a viewer application when a subject performed a storing motion (Fig. 3). An experimenter confronted the system’s decision with Fig. 3. Experimental Environment and the location where a subject ac- Viewer Application tually stores a mobile phone. The viewer application called a function for MLP-based classification that had been generated by LNKnet. In addition to the five subjects who had participated to the offline experiment, four subjects whose data were not included in the training dataset were newly joined to the online experiment. In this experiment, only nonstationary motion was tested 10 times each time. 4.2

Results and Analysis

The successful recognition rate (True-Positve Ratio; TPR) for each binary classifier in the offline experiment is shown in Fig. 2, where the numbers indicate the one successfully classified into a particular class and the total number (N) in the class. Note that the results of the upper-stage classifiers should be accumulated to obtain the actual performance since the result presented in Fig. 2 merely indicates the performance of each classifier. For example, the TPR for “hand” (classifier C3) was 99.2%, which should take into account upper classifier C1; finally, the actual TPR was 98.6% (=99.4%×99.2%). As for stationary motions shown in the right branch, the average TPR was 99.9%. Table 1 (middle column) shows the actual TPRs for six locations Table 1. Actual True-Positive Ratios [%] in a nonstationary motion in of- for Cascaded and Single Classifiers in Offline experiments. Here, the aver- fline Experiment (nonstationary motion) age TPR is 76.7%, where the highLocation Cascaded Single est (96.9%) and the lowest (58.1%) hung on the neck 96.7 100.0 came from the neck and the back 82.0 80.3 jacket pocket pocket, respectively. Table 1 also chest pocket 64.5 58.3 shows the TPRs obtained from a 60.3 6.1 trousers pocket (front) single classifier, where five classes trousers pocket (back) 58.1 43.4 were classified at a time. The comhand 98.6 – parison shows the cascaded clasaverage 76.7 50.4 sifiers performed well as designed.

Although the hierarchy was manually determined to maximize the average TPR, there would be another strategy for optimization, e.g. the False-Positive Ratio for a neck is subject to minimize to avoid underestimation of an alert-worthy heat stroke indicator. By contrast, the TPRs for the online evaluation is an average of 73.2% in a nonstationary motion. Fig. 4 shows the detail of five of nine subjects’ results. The row and column indicate the answers and the results, respectively. The performance for the front pocket (4) was the lowest of all. We consider this was because the features were Fig. 4. The classification result of nonstill not enough “strong” to dis- stationary motion in online experiment criminate the gesture of a back trousers pocket from that of a front trousers pocket. We further investigated to see the robustness of classification in the shape of a pocket, where every subject tried two jackets with different shape of pockets; however, we have not confirmed significant difference in the TPRs. An outlier in a storing motion might be contained when a subject was asked to store an unusual location for storing a mobile phone. We confirmed this with a subject who had never hung a mobile phone on the neck; she struggled with the action because of her long hair. We need to consider to remove such a case when we collect training dataset as well as conducting an online test.

5

Conclusions and Future Directions

We have proposed a method that keeps track of the location of a mobile phone on the body. The results of offline and online experiments showed that an overall accuracy of identifying six locations was about 70%, while in a stationary motion, the accuracy was almost 100%. We will improve the capability of the tracking system as a whole. Here, capability indicates not only quantitative one, i.e. TPR, but also qualitative ones, i.e. 1) real-time processing on a mobile phone terminal and 2) supporting a bag as a storing position. The TPR depends not only on the classifiers, but also on the segmentation of the region that a subject actually performs a storing motion. Currently, a user is required to be still for 3 seconds before and after storing a mobile phone, otherwise a segment contains two actions (actual storing and the other unrelated movement). The requirements obviously limits a natural action of a user and thus impractical. We are improving the segmentation algorithm so that it can extract only an actual storing motion. Our final goal is to realize a self-contained localization on a resource constrained mobile phone terminal, although the implementation utilized in the online experiment runs on a PC. We are porting the current prototype system into Android-based commercial mobile phone terminal (HTC HT-03A [3]), which will clarify the feasibility of the localization process and the features described in Section 3.

We utilized specific patterns of storing into current six locations; however it would not be so easy as to be applicable to a bag since the size of a storage container, e.g. a side pocket and the main bag, is more diverse than clothes’ pocket. A hypothesis to address the issue is that the trajectory of storing into a bag and the forward acceleration to a bag are different from those of other locations. Additionally, the temperature data collection in Section 2.2 has dropped a hint to us. We consider pockets on clothes are susceptible to the effect of the body heat, while the temperature inside a bag might not be so. We will collect storing motions for a wide variety of bags and temperature data to find a solution. Moreover, to validate the concept, we will develop applications presented in Section 2, such as an on-body location-sensitive call/e-mail notification and assurance of a mobile phone placement for reliable environmental sensing. Also, an application-specific classifier hierarchy and patterns of user interaction are studied through the applications. Finally, a location tracking service is provided on an Android-based terminal as a common component for applications.

Acknowledgments This research has been supported by MEXT Funds for Promoting Research on Symbiotic Information Technology and for Division of Young Researchers, and by the Kayamori Foundation of Informational Science Advancement.

References 1. Y. Cui, et al. A Cross Culture Study on Phone Carrying and Physical Personalization. In Proc. of HCI International 2007, pp. 483–492, 2007. 2. C. Harrison and S. E. Hudson. Lightweight material detection for placement-aware mobile computing. In Proc. of the 21st annual ACM symposium on User interface software and technology (UIST’08), pp. 279–282, 2008. 3. HTC Co. HT-03A. URL: 〈http://www.htc.com/jp/product/ht03a/ specification.html〉. (in Japanese) 4. iShare Inc. Survey on mobile phone usage, November 2005. URL: 〈http://blog. ishare1.com/press/archives/2005/11/post 19.html〉. (in Japanese) 5. Y. Kawahara, et al. Recognizing user context using mobile handsets with acceleration sensors. In IEEE Int’nl Conf. on Portable Information Devices (Portable 2007), pp. 1–5, 2007. 6. K. Kunze and P. Lukowicz. Using acceleration signatures from everyday activities for on-body device location. In Proc. of the 11th Int’nl Symposium on Wearable Computers (ISWC2007), pp. 115–116, 2007. 7. K. Kunze, et al. Where am I: Recognizing On-body Positions of Wearable Sensors. In Proc. of Int’nl Workshop on Location- and Context-Awareness (LoCA 2005), pp. 264–275, 2005. 8. MIT Media Laboratory. LNKnet. URL: 〈http://www.ll.mit.edu/mission/ communications/ist/lnknet/〉. 9. R. C. Shah, et al. On the Performance of Bluetooth and IEEE 802.15.4 Radios in a Body Area Network. In Proc. of the 3rd Int’nl Conf. on Body Area Networks (BodyNets ’08), pp. 1–9, 2008. 10. N. Thepvilojanapong, et al. A Simulation Study of Human-Probe Approach for Urban Sensing. Tech. Report USN2009-9, IEICE, May 2009. 11. Wireless Technologies, Inc. WAA-001. URL: 〈http://www.wireless-t.jp/〉.