ABSTRACT REAL-TIME POINT CLOUD DATA TRANSMISSION IN KINECT POINT CLOUD DATA BASED TELEREHABILITATION SYSTEM

ABSTRACT REAL-TIME POINT CLOUD DATA TRANSMISSION IN KINECT POINT CLOUD DATA BASED TELEREHABILITATION SYSTEM Real-time monitoring is a key aspect of Te...
Author: Bethany Bailey
1 downloads 0 Views 2MB Size
ABSTRACT REAL-TIME POINT CLOUD DATA TRANSMISSION IN KINECT POINT CLOUD DATA BASED TELEREHABILITATION SYSTEM Real-time monitoring is a key aspect of Telemedicine. Using Kinect sensor for remote video based rehabilitation diagnosis has been a low cost, convenient, and effective technology. However, point cloud data from Kinect sensors are usually quite large (90MB/s). Therefore, how to ensure the quality of service (QoS) of data transmission under limited bandwidth is a challenging issue. In this thesis, we propose a selfadaptive and hybrid mode with both patient self-management and doctor remote diagnosis based telemedicine framework. The core of this framework is a Kinect differential octree based point cloud data dynamic algorithm, point cloud detail compression, frame skipping, dynamic compression and etc. We performed numerical analysis and simulation. Results show the system has good effective and feasibility under limited bandwidth. In addition, a Speed-based Telerehabilitation System was proposed as test system for dynamic algorithm. Yilin Chen December 2015

REAL-TIME POINT CLOUD DATA TRANSMISSION IN KINECT POINT CLOUD DATA BASED TELEREHABILITATION SYSTEM

by Yilin Chen

A thesis submitted in partial fulfillment of the requirements for the degree of Master of Science in Computer Science College of Science and Mathematics California State University, Fresno December 2015

APPROVED For the Department of Computer Science: We, the undersigned, certify that the thesis of the following student meets the required standards of scholarship, format, and style of the university and the student's graduate degree program for the awarding of the master's degree.

Yilin Chen Thesis Author

Ming Li (Chair)

Computer Science

Alex Liu

Computer Science

Jin Park

Computer Science

For the University Graduate Committee:

Dean, Division of Graduate Studies

AUTHORIZATION FOR REPRODUCTION OF MASTER’S THESIS X

I grant permission for the reproduction of this thesis in part or in its entirety without further authorization from me, on the condition that the person or agency requesting reproduction absorbs the cost and provides proper acknowledgment of authorship. Permission to reproduce this thesis in part or in its entirety must be obtained from me.

Signature of thesis author:

ACKNOWLEDGMENTS Thanks Dr. Ming Li helped me to modify this thesis.

TABLE OF CONTENTS Page LIST OF TABLES .................................................................................. vi LIST OF FIGURES ................................................................................. 1 INTRODUCTION .................................................................................... 1 LITERATURE REVIEW ........................................................................... 4 Related Works ................................................................................. 4 Related Material and Knowledge ...................................................... 9 METHODS AND MATERIALS ............................................................... 17 Kinect PCD based Telerehabilitation Scheme ................................. 17 Dynamic Transmission Scheme in Telerehabilitation Environment ........................................................................ 20 Speed-based Telerehabilitation System .......................................... 25 RESULTS AND DISCUSSION ............................................................... 35 Phase 1 of Dynamic Compression Scheme’s Feasibility .................. 35 Speed-based Telerehabilitation System .......................................... 37 CONCLUSION ...................................................................................... 48 REFERENCES ..................................................................................... 49

LIST OF TABLES Page Table 1. Compression ratio output under different situation ................ 36 Table 2. Condition of system running’s output under different situation ................................................................................. 36 Table 3. PCD output for case of success background checking ............. 38 Table 4. PCD output for case of “good” in IDM...................................... 39 Table 5. PCD output for case of success background checking in HM... 41 Table 6. PCD output for case of people detecting in HM........................ 42 Table 7. PCD output for case of success location adjustment in HM ..... 43 Table 8. PCD Output for Case of System Training in HM ...................... 44 Table 9. PCD Output for Case of System Recording in HM ................... 45 Table 10. Digit statistic output for table 7 ............................................ 45 Table 11. PCD Output for Case of Success Background Checking in RDM ....................................................................................... 46 Table 12. PCD Output for Case of Real-time Diagnosis in RDM ............ 47

LIST OF FIGURES Page Figure 1. Structure of Kinect (http://www.amazon.com/KinectSensor-Adventures-Xbox-360/dp/B002BSA298) .................. 11 Figure 2. Structure of octree ................................................................ 13 Figure 3. Structure of differential octree ............................................... 13 Figure 4. Kinect PCD based Telerehabilitation Scheme ......................... 17 Figure 5. Dynamic Compression Scheme for Kinect PCD ...................... 21 Figure 6. PCD image of suitable background in IDM ........................... 38 Figure 7. PCD image with DS=0 in IDM................................................ 39 Figure 8. PCD image with DS ≠0 in IDM ............................................... 40 Figure 9. PCD image output example for case of success background checking in HM .................................................................... 41 Figure 10. PCD image output example for case of success location adjustment in HI&HRM ........................................................ 43 Figure 11. PCD output example for case of real-time diagnosis in RDM..................................................................................... 47

INTRODUCTION Kinect is a peripheral sensor device for Xbox games. Main character of Kinect is its powerful ability on depth data grabbing for target object. Appearance of Kinect made an important impact on the game software industry. At the same time, depending on Kinect’s three functions[1]: 3D image detecting, human skeleton tracing, audio processing, Kinect also aids research in other fields like robotic, medical care, manufacturing, retailing, and etc. In the telerehabilitation field, Kinect can be applied in a telerehabilitation system. Kinect-based telerehabilitation system has the advantages of being convenient, fast, and cheap. On the other hand, because Kinect can grab 3D image and convert it to PCD (Point Cloud Data), the high-speed development of PCL (Point Cloud Library) provides a new way to improve Kinect-based telerehabilitation system. The way of improvement is applying PCL to PCD’s processing. Relying on PCL’s powerful PCD processing ability, this will provide the system new functions and make the system more powerful. The architecture of Kinect PCD based Telerehabilitation System proposed in this thesis includes two parts: intelligent diagnosis and realtime diagnosis. Generally, a patient can use intelligent diagnosis to take a self-service diagnosis first, if the patient satisfies intelligent analyzing’s output, real-time diagnosis is avoided. Otherwise, the patient can use real-time diagnosis to take a manual diagnosis. The combination of intelligent way and real-time way reduces doctor’s work amount in diagnosis on contrast with traditional telerehabilitation’s single real-time

2 diagnosis. On the other hand, intelligent diagnosis provides the patient a wearable-device-free and a funny way in diagnosising. To some degree, patients free themselves from repetitive, dull, and boring postures in traditional rehabilitation. As the Kinect PCD based Telerehabilitation System’s design, the main challenge is real-time PCD transmission. In this thesis, Dynamic Compression Scheme is proposed for this challenge. Dynamic compression should keep telerehabilitation quality and reduce real-time PCD transfer to save bandwidth in the same time. Based on this idea, in the Dynamic Compression Scheme’s design, a frame skipping technique is proposed. Frame skipping means the system will skip the frame which contains a small PCD change compared to the last frame. After processing frame skipping, based on the frames’ transfer in present second, the system can predict the frames’ amount need to be transferred in the next second. The predictive frames’ amount is also based on the thought of patient’s postures usually similar in neighbor seconds. According to this predicted frames’ amount for next second, lowest bandwidth of transmission’s requirement also can be predicted. For keeping quality of real-time diagnosis, the system will compare a present network’s transmission ability and the lowest bandwidth requirement to calculate the lowest compression ratio. If compression ratio is suitable, the system will take a differential octree-based compression to compress PCD frames and transfer them to the server. Otherwise, if the compression ratio is too big, the system will abandon the frames’ transmission. In addition, detail compression is also applied in this differential octree-based compression; the compression’s encoding

3 includes differential octree encoding, position detail encoding, and feature detail encoding. The feature detail includes color, normal, luminance, curve, and etc. of point. In the end, the feasibility and operability of this Dynamic Compression Scheme is demonstrated by C++ programming based simulation. For testing the Dynamic Compression Scheme in a real PCD environment, a simple Kinect PCD-based Telerehabilitation System is developed. However, it is worth noting that this system is only a speedbased system. Aiming at purpose of testing, a speed-based system is enough. Since the author’s time is limited, Dynamic Compression Scheme’s testing in real PCD environment is the future work. So the author only demonstrates the feasibility and operability of the Dynamic Compression Scheme in this thesis.

LITERATURE REVIEW Related Works KiReS is Kinect based Telerehabilitation System which can customize rehabilitation therapies and help users improve their performance in an intelligent manner[2]. Prototype of KiReS is already developed and which is already in operation now. KiReS provides a wearable device free and funny solution for the telerehabilitation. It can be divided into modules and follows a client-server approach. The modules of KiReS include an exercise evaluation module, data processing module, and an interface module. The exercise evaluation module compares performed exercises with expected exercises, and then provides results of the intelligent diagnosis for patient. The data processing module handles received data from Kinect. After Kinect receives the data, a descriptor is created based on the patient’s performance. The descriptor has a skeleton structure. In this skeleton structure, there are 20 joints described by points with 3 coordinates including width, height, and depth. As an interface module, there is a difference between the patient’s interface and the doctor’s interface. The system of KiReS has two avatars. One avatar shows the exercise a patient must take, and the other avatar shows the patient’s motion. In this way, a patient can improve their performance according to the standard. In addition, at the top of the interface, there is an indicator to show how much exercise remains. From the view of the doctor, KeRiS can define customized exercises for the patient. Pre-defined exercises were stored in the system. If the doctor wants to add new exercises for a patient, he/she only needs

5 to perform the new exercises in front of Kinect and the system will record it automatically. In 2013, the “Intelligent Sport Analyzing and Training System” developed by the Chong Jun Company in Hangzhou (China) earned a championship in the “1st Competition of Kinect for Windows Innovative Application Development” in China. From 2015, this system already had been utilized in some main hospitals in the Zhejiang Province of China[3]. For example, in the People’s Hospital of Zhejiang Province, before introducing the“Intelligent Sport Analyzing and Training System,” the patients needed to perform a lot of repetitive postures for rehabilitation. Such a manner of repetitive postures is hard and boring for the patients. But at the same time, unfortunately, this is the key of rehabilitation for patient. The“Intelligent Sport Analyzing and Training System” is constituted by host computer, touch monitor, Kinect for Windows, and special software for rehabilitation. When the system is operating, the patient stands before the monitor and performs the postures according to indications based on the doctor’s customized exercises for the patient. The monitor also provides evaluations and posture-adjustment suggestions to the patient according to the intelligent analysis based on patient’s performance. Besides that, the patient also can take interactivity with the content in the monitor just like playing Xbox games. For example, the patient needs to perform postures of waving their hands 50 times before the system is introduced. But now, the repetitive postures of waving hands can be avoided. Because the new system can make rehabilitation processes just like the Xbox games. The patient can make postures of throwing balls to the monitor and Kinect can record

6 these postures to take an intelligent analysis. After the analysis, the system will grade the patient’s performance. This manner can provide the patient with a much more enjoyable experience compared to traditional rehabilitation’s repetitive postures. Comparing with the traditional rehabilitation system, the“Intelligent Sport Analyzing and Training System” is a wearable device free approach for the patient’s physical body, and it is also a much cheaper, low complexity, and less cross infection approach for rehabilitation. At the real-time compression of PCD, Julius, Rusu et al. proposed a differential octree based compression scheme[4], which has a twobuffering octree architecture. So it is also an octree based approach. At the same time, it is a lossy approach. It can handle unorganized point cloud streams with varying size, point order, and point density. It allows controlling coding complexity and coding precision. Specifically, the encoding is not only aimed at the point position, but also aimed at point details like colors, intensives, or geometric features. In the compression system, point detail encoding, point component encoding, and differential octree’s binary serialization encoding are separate encoding processes. In addition, the point component encoding is aimed at the points’ features like color, normal, and etc. For improving the transfer speed of compressed PCD, an integer-arithmetic variant of an arithmetic coder for entropy encoding is applied after these three encoding processes. After entropy encoding, compressed PCD can be transferred in the network. Processes of decoding are just the reverse processes of encoding. A compression system like this can efficiently detect and encode spatially and temporally redundant signal information from a stream of point

7 cloud. In addition, position detail encoding can reduce complexity. Experiment output shows the system can keep computational requirements minimal for reducing the processing latency, and enable real-time or closed to real-time encoding-decoding performances. In present, this compression approach has been implemented and released as a part of the open-source PCL. Except the differential octree based PCD compression scheme, other kind of compression scheme is also proposed. Fan et al. proposed a hierarchical clustering based PCD compression scheme [5]. This scheme performs clustering to the points to generate a coarser level of detail (LOD). After clustering, a sequence of LODs is generated and LOD hierarchy is formed. And then, width-first traverse is applied to this LOD hierarchy from top to down. For each node encountered in the traverse, corresponding geometric updates associated with its children are encoded. This geometric updates led a dynamic encoding. In addition, in point clustering, GLA (Generalized Lloyd Algorithm) is applied to maximize the approximation quality of the intermediate LODs and distance metric proposed in [5] can drive the clustering process which takes both the position and the normal attributes into account for best preserved quality of the approximating LODs. Processes of hierarchical clustering based PCD compression scheme involve three parts: LOD hierarchy construction process, LOD hierarchy encoding process, and LOD hierarchy encoding process. Specifically, LOD hierarchy encoding and decoding processes are reverse to each other. Advantages of this compression scheme include generic applicability and outstanding low bitrate coding performance.

8 As metioned above, KiReS and “Intelligent Sport Analyzing and Training System” are both Kinect based system. They cannot handle PCD. Up to now, Kinect PCD based telerehabilitation is stills a blank. On contrast with Kinect-based telerehabilitation system, Kinect PCD based system introduce PCD into system’s design. Rely on PCL’s powerful PCD processing ability, Kinect PCD based system can provide more functions in diagnosis, especially in intelligent diagnosis. But if apply PCD to system’s design, there has two challenges. Firstly, bottleneck of bandwidth led by PCD’s big size; secondly, Kinect PCD based system’s design. As solutions for these challenges, real-time PCD compression scheme’s design is the key. This compression scheme should have characters of dynamic and generic applicability for most features of point in PCD. Differential octree based PCD compression scheme can support real-time PCD transmission, but it cannot realize dynamic compression. Hierarchical clustering based PCD compression scheme only focuses on compressing point’s positions and normal. For satisfying requirements of Kinect PCD based telerehabilitation, new system architecture and new PCD compression scheme are needed. The main contributions of the thesis are:  It proposed a Kinect PCD based Telerehabilitation System;  It proposed a dynamic real-time PCD compression scheme;  It developed a speed-based telerehabilitation system for testing realtime PCD compression scheme.

9 Related Material and Knowledge Kinect In the past few decades, various kinds of equipment were developed for grabbing 3D image from physical world, including TOF (time-of-light) camera, stereo camera, laser scanner, and structured light camera[6]. They use light wave’s properties to grab depth data of target object. But their concrete principle of depth data grabbing are distinct from each other. Since high cost and bulky size, these 3D grabbers usually applied in industrious field. On contrast with them, Kinect provide a low-cost and efficient way to realize 3D image grabbing. As the core hardware in telerehabilitation system in this thesis, Kinect is presented to market in November 4, 2010 as sensor device for Xbox 360 games. Appearance of Kinect is a landmark in game software industry. It makes players abandon traditional game controllers like keyboard, mouse and etc. to play games but completely based on gesture and voice[7]. In the same time of the Kinect’s success in game software field, Kinect also draws attention from researchers in other fields like robotics, medical care, theater performers, natural interfaces and etc. For satisfying requirements of researching, in June 2011, Microsoft released a SDK for Windows 7. In February 2012, it was launched the Kinect for Windows [8]. The combination of Kinect hardware and SDK generates Kinect for Windows, which can support applications built with C++, C#, and Visual Basic. On contrast with Kinect for Xbox 360, in June 2014, Microsoft release Kinect for Xbox One. Xbox 360’s Kinect is known as Kinect 1.0 and Xbox One’s Kinect is known as Kinect 2.0. At the same time, there is also has a different between SDK 1.0 and SDK 2.0 for

10 Kinect 1.0 and Kinect 2.0 respectively. As Kinect 2.0 with SDK 2.0, on contrast with Kinect 1.0 with SDK 1.0, researchers can get a better precision, responsiveness, and intuitive capabilities when it respond to movement, gesture, or voice [9]. Specifically, in author’s research, Kinect 1.0 and SDK 1.0 are applied. As a future work, author will replace them to Kinect 2.0 and SDK 2.0. In this thesis, where Kinect is mentioned, that is Kinect 1.0. Structure of Kinect is shown in Figure 1. It mainly includes RGB camera, IR emitter, IR camera [10]. RGB camera is a traditional camera with ability to take 2D color image. RGB means three basic colors: red, green, and blue. These color channels can using 8 bits to represent matrix of integers. So RGB image is capable to use 256 colors. As depth data, the combination of IR emitter and camera can take depth image. Depth data uses 16 bits to represent matrix of integer. In addition, though 16 bits is applied in representation, only 11 bits are applied in data transmission for saving space. As mentioned above, format of the image Kinect grabbed can be expressed as RGBD image [8]. Speed of Kinect’s RGBD image grabbing is about 30 fps (frame per second) [11]. This RGBD data can be applied in tasks that difficult before. The principle of depth image grabbing is utilizing relationship between disparity map and depth map, and then, mathematical calculation based on triangle is applied to get the depth data. PCL As the main tool of PCD processing, PCL (Point Cloud Library) is a comprehensive free, BSD licensed, fully template C++ library for 3D point

11

Figure 1. Structure of Kinect (http://www.amazon.com/Kinect-SensorAdventures-Xbox-360/dp/B002BSA298) cloud processing. The library has algorithms including filtering, feature estimation, surface reconstruction, registration, model fitting, and segmentation [12]. In PCL, most mathematical operations are based on Eigen, which is an open source template library for linear algebra. Fast k-nearest neighbor search operations is provided by FLANN (Fast Library for Approximate Nearest Neighbors). In addition, all the modules and algorithms in PCL using boost to share pointers. This method can avoid data re-copying for the data that already exist. In present, PCL can be ported to Windows, Mac OS, and Linux. Specifically, Android ports are in the works [12]. As algorithms in PCL, each set of algorithms is defined by basic classes. This method can integrate all the common functionality used throughout the entire pipeline, which can keep the implementations of the actual algorithms compact and clean [12]. The basic processing pipeline in PCL is:

12 1.

Object Creating: create object like segmentation,

registration, filter, and etc. 2.

Input Point Cloud: using “setInputCloud” to input point

cloud for processing. 3.

Parameters Setting: set parameters for the object. These

parameters will affect PCD processing. 4.

Outputting: call filter, compute, segment, or others can get

data processing result. Nowadays, an image grabbed by Kinect can be directly converted to a PCD, which can be processed by PCL. The combination of PCL and Kinect is the foundation of research in this thesis. Differential Octree based PCD Compression Before introduce differential octree, simple review of octree is needed. Octree divide a cube into 8 equal small cubes. Every cube has a same time and spatial complexity [13]. By recursive dividing, octree can indicate location of point in point cloud by serialized octree (see Figure 2). Times of cube dividing is more, resolution of octree is higher. By adjusting resolution of octree, or adjusting size of leaf cube, octree based compression algorithm can be designed. In PCD video, every frame’s PCD image can be compressed based on octree algorithm and transferred in network. But in most situations, neighbor PCD image’s part of changes is much smaller than part of unchanges. If apply octree algorithm to compression scheme directly, whole image includes part of changes and unchanges will be transported without any difference in every frame.

13

Figure 2. Structure of octree This manner will lead a bandwidth cost increasing. For saving bandwidth, only transferring part of changes in present frame’s PCD image, and keeping part of unchanges without any transfer on contrast with last frame, differenctial octree algorithm is proposed in previous work. As the differential octree technology, it can describe spatial info about part of changes in PCD image and transport binary code for this info. As part of unchanges in PCD image, differential octree technology can avoid them as redundant data. The architecture and work principle of differential octree is shown in Figure 3.

Figure 3. Structure of differential octree The concrete thought of describing spatial info about part of changes in PCD image by differential octree is like this: Every node of differential octree has two buffers: buffer X and buffer Y. they can be treat as mutual independent octree and save real-time octree data respectively. After system initiated, octree data go to buffer X first, in the

14 same time, buffer Y set 0. When next frame arrived, the new octree data go to buffer Y, at the same time, buffer X still keep octree data of last frame. Once frame arrived, a bit-XOR operation is taken for octree data’s processing between buffer X and buffer Y. If the calculation output is 00000000, that means the corresponding spatial area’s PCD in octree of buffer X and buffer Y has no changes. Otherwise, changes are exist. As the new adding nodes or new coming nodes, initialization method in buffer X and buffer Y is like these: If assume buffer Y need adding new nodes, the new added nodes in buffer Y (corresponding spatial area where is set 1) will take a contrast with the same spatial area in buffer X. If there has no node in this area, buffer X of new node will set 0. Otherwise, just copy buffer X’s info of buffer X to new node’s buffer X of buffer Y [4]. As PCD, it is a big size data set. Points’ info in PCD includes 3D coordinates, RGB, strength, distance, normal, and etc. So Kinect PCD’s features inherited features of RGB data, depth data, and other related data. In application based on Kinect and PCL, PCD can represent depth data of Kinect. Nowadays, PCD can be grabbed by Kinect directly [8], so potential of Kinect PCD based Telerehabilitation System’s real-time feature and dynamic feature are good. But at the same time, Kinect PCD’s big size will inflict both real-time transmission and Kinect PCD’s creating speed. A concrete example of PCD is shown in Figure 6 in other chapter. In this PCD image, there are 307200 points and size of this image is 10.5MB. So there has two difficult points in Kinect PCD based Telerehabilitation System’s design: Kinect PCD creating speed and its real-time transmission. In this thesis, author only discuss the second

15 difficult point. The solutions for real-time transmission need compressing PCD maximally and keeping quality of telerehabilitation diagnosis in the same time. The octree compression algorithm in PCL can compress all kinds of PCD include organized-PCD and unorganized-PCD [8]. Organized and unorganized PCD is mainly differentiated by reference points exist or not. They are also differentiated by changes of points’ size, pixel, distribution density, points’ orders exist or not. In reality, application based on octree compression algorithm cannot satisfy requirements of bandwidth saving and image quality’s improving in real-time transmission. Appearance of differential octree compression algorithm compensates the shortages of octree compression algorithm. But specifically, differential octree itself can not realize dynamic compression. Dynamic compression need to decide minimum compression ratio according to real bandwidth and speed of patient’s posture. Maximal quality keeping can be realized by dynamic compression. Aiming at important characters of telerehabilitation environment include patient’s slow motion and requirement of frequently monitoring patient by doctor, in the design of compression algorithm, redundant info should be deleted for complexity reducing and bandwidth saving. Based on the consideration mentioned above, Kinect PCD based dynamic transmission scheme is proposed in this thesis. Main characters of this scheme include:  Dynamic optimal compression is realized base on differential octree algorithm;  Frame skipping technology is proposed for reducing requirement of bandwidth’s cost.

16  System can calculate parameters automatically and take a selfadjustment. PCD Position Detail Compression As the differential octree based PCD compression, for detecting PCD’s change in detail, unlimited depth adding of differential octree can be applied in theory. This depth adding means using more bits to describe points’ position. But in reality, this measure will make calculation amount increased exponentially. For reducing calculation complexity, leaf node of octree can be limited in a certain degree of depth. In cube which leaf node represented, central position will represent points’ position in it. On the other hand, points’ position in leaf cube also can be described by discrete coordination [4]. First method and second method mentioned above are called PCD merging compression for location and PCD detail compression for location respectively. Concrete method of the PCD position detail compression is like these: Firstly, PCD in leaf cube will be down sampled and only maintain feature points in leaf node. The feature points are much less than original points in leaf cube. One vertex of leaf cube can be treated as original point in coordination. Discrete coordinates and distance to origin point for every feature nodes will be saved. This info will get a separately encoding in transmission as serialized binary encoding for position detail of PCD on contrast with serialized binary encoding for differential octree of PCD. Except position of PCD, PCD’s other info like: color, size of root cube, depth of differential octree, and etc. also need separately encoding [6]. This encoding can be called serialized binary encoding for PCD’s feature.

METHODS AND MATERIALS Kinect PCD based Telerehabilitation Scheme Design of Kinect PCD based Telerehabilitation Scheme Kinect PCD based telerehabilitation scheme has two diagnosis modes: the intelligent diagnosis and the real-time diagnosis. In the intelligent diagnosis, patient can take a self-service diagnosis under the instructions of system. In the intelligent diagnosis, a Kinect PCD video will be saved in client(patient). When network is free, these video files will be sent to the server(doctor) for a back-up. Back-up files could be reviewed by doctor in future. After intelligent analyzing, patient can get result of the intelligent diagnosis. If patient satisfy to this intelligent result, real-time diagnosis can be avoided. Otherwise, patient can take the real-time diagnosis for a further diagnosing. Specifically, Figure 4 shows the architecture of this Kinect PCD based Telerehabilitation Scheme.

Figure 4. Kinect PCD based Telerehabilitation Scheme

18 Kinect PCD based Dynamic Compression Scheme Aiming at the challenges mentioned in “LITERATURE REVIEW”, in this thesis, the Kinect PCD based Dynamic Compression Scheme focuses on the real-time diagnosis. The PCD compression and the frame skipping technology is the core technology in this scheme’s design. These technologies can reduce the bandwidth requirement in data transmission maximally. At the same time, data transmission quality is kept to satisfy the telerehabilitation’s requirement. In the real-time diagnosis, since real-time Kinect PCD’s big size, when some patients log in the real-time diagnosis in the same time, the problem of bandwidth’s bottleneck should be concerned. This problem can be solved by applying the dynamic compression technology. The dynamic compression’s application has an idea like this: when doctor take a check concern about patient’s rehabilitation. He/she only need take knowledge about the body’s motion of patient. At the same time, the information of detail characters like facial character, height, dress, and etc. can be ignored. Based on the consideration of the predictive minimum bandwidth’s requirement in PCD video transmission and the real bandwidth at that moment, the application of dynamic compression can keep video quality for the main characters of patient and ignore video quality for the less important characters of patient. So the dynamic compression can aid bandwidth saving and keep quality of diagnosis in the same time. The telerehabilitation scheme proposed in this thesis is shown in Figure 4. Specifically, the Kinect PCD based Dynamic Compression Scheme in real-time diagnosis will be discussed as an important point in

19 this thesis. According to the C++ based simulation experiment of feasibility checking for this scheme, a good feasibility and operability of this scheme can be demonstrated. Cmpression Algorithm Suitable for Telerehabilitation Environment In general, the Kinect PCD based compression schemes include these items: 1. The octree based compression with the PCD position merging compression; 2. The differential octree based compression with the PCD position merging compression; 3. The octree based compression with the PCD position detail compression; 4. The differential octree based compression with the PCD position detail compression; Since author’s research aims at the real-time monitoring in a telerehabilitation environment, the original PCD after being processed by scheme 1 or 3 based compression still has a big size not suitable for the real-time compression. Comparing with octree, the differential octree based compression can effectively reduce bpp (bits per pixel). More accurate of the compression or more small of the leaf cube’s size, the effects of bpp’s reducing are more obviously. As scheme 2, the PCD position merging compression can reduce the calculation complexity in compression. But at the same time, the accurate of compression still be reduced. Comparing with schemes mentioned above, the differential

20 octree based compression with the PCD position detail compression is a suitable choice for the real-time monitoring in a telerehabilitation environment. Dynamic Transmission Scheme in Telerehabilitation Environment Dynamic Compressing Scheme for Kinect PCD Video In the real-time diagnosis, keep the quality of Kinect PCD video is a key. The factors can inflict this quality including: 1. The PCD frames’ amount need to transfer is unstable; 2. the data transmission ability of network is unstable. Based on these considerations, the Dynamic Compression Scheme proposed in thesis can be divided to 2 phases: 1. checking transmission’s possibility; 2. transmission. The concrete work processes of Dynamic Compression Scheme are like these: 1. Grabbing a Kinect PCD video of patient’s posture; 2. Decide the predictive amount of frames Ptransport (t ) will be transferred by frame skipping technology; 3. Based on Ptransport (t ) calculate the minimum bandwidth requirement BWrequire . At the same time, calculate the real bandwidth BWreal (t ) ; 4. Based on BWrequire and BWreal (t ) calculate the minimum compression ratio N (t ) ; 5. Comparing N (t ) with the maximum compression ratio N h which is decided by doctor to check the system terminal condition is satisfied or not. If N (t )  Nh , step 6 is carried out. Otherwise, stop the transmission.

21 6. The differential octree based compression with the PCD position detail compression is applied for the Kinect PCD compression; 7. Transferring the compressed Kinect PCD to a server or a monitor on side of doctor. Based on these steps, essence of the Dynamic Compression Scheme can be treated as a self-adjustment scheme for Kinect PCD transmission. These steps also can be described as form of a flow chart shown in Figure 5.

Figure 5. Dynamic Compression Scheme for Kinect PCD Frame Skipping In the real-time data transmission, author proposed an inconstant fps as an improvement of the differential octree based PCD compression. The concrete method of this improvement is like these: Comparing the

22 present frame’s image and the last frame’s image. If the changes are tiny, the present frame will be skipped. This method can reduce the fps for saving bandwidth. Author called this method as the frame skipping. Based on the frame skipping, follow definitions are given: Def 1: the amount of skipped frames in t seconds is the frame-skipamount in t seconds Fabandon (t ) , t 



;

Def 2: the amount of real-time transferred frames in t seconds is the frame-transport-amount in t seconds Ftransport (t ) , t 





Def 3: based on Fabandon (t )  0 , Ftransport (t ) is the full frame-transport-amount

Ffull (t ) , t 



;

Def 4: the frame-skip-ratio in t seconds is Rabandon (t )  Fabandon (t ) / Ffull (t ) , t 



Def 5: the frame-transport-ratio in t seconds is Rtransport (t )  Ftransport (t ) / Ffull (t ) ,

t



;

Def 6: the predictive frame-skip-number at t second for [t, t+1] is Ptransport (t )  Ftransport (t )  Ftransport (t  1) , t 



;

Def 7: the compression ratio at t second is N (t ) , t 



;

Def 8: N h is the maximum compression ratio pre-established by doctor. On the issue of comparing present frame with last frame, two schemes are proposed: 1. On condition that compression ratio has been decided, or size of leaf cube has been decided, if all outputs of the XOR operation for X levels in differential octree are 00000000, the present frame will be skipped. That means the present differential octree’s XOR outputs will be treated as 0 at all. In addition, the value of X can be decided by the present compression ratio;

;

23 2. Set a weight for every level in the differential octree. The concrete weight setting method is like this: the root level can be treated as level 0 and weight for the level 0 is 0. And then, the level 1 has a weight 1, the level 2 has a weight a, the level 3 has a weight b and etc. In the end, in each level, a multiply operation is applied for the weight and the XOR output. Output of this multiply operation is a weight sum for a level. Applying an add operation for the weight sum for all levels, a weight sum for the differential octree is attained. For the frames with a low weight sum for the differential octree, the frame skipping is applied. In the end, according to Ptransport (t ) and N (t  1) , the minimum bandwidth requirement BWrequire (t  1) in [t, t+1] can be calculated. Realizable Network Bandwidth The calculation method of realizable network bandwidth BWreal (t ) at t second is like these: a client send 3 test packs to a server at intervals (eg. 30s) and require a respond. After receiving the respond from the server, BWreal (t ) can be calculated based on the size and the latency of test packs in transmission. BWreal (t )  ( BWreal1 (t )  BWreal 2 (t )  BWreal 3 (t )) / 3  [( P1 (t ) / 1 (t ))  ( P2 (t ) /  2 (t ))  ( P3 (t ) /  3 (t ))] / 3

In this formula, P1 (t ) , P2 (t ) , P3 (t ) are the sizes of test pack 1, test pack 2, test pack 3 at t second respectively.  1 (t ) ,  2 (t ) ,  3 (t ) are the latencies of test pack 1, test pack 2, test pack 3 sent to server at t second respectively. BWreal1 (t ) , BWreal 2 (t ) , BWreal 3 (t ) are the realizable bandwidths tested by test pack 1, test pack 2, test pack 3 at t second. In addition,

24 this BWreal (t ) is different from a realizable bandwidth provided by network service provider. Condition of System Running In the telerehabilitation, for keeping quality of a diagnosis, data transmission should satisfy the condition of system running. The concrete work process of this condition is like these: 1. Doctor pre-establish N h ; 2. According to BWrequire (t  1) and BWreal (t ) , calculating N (t  1) ; 3. Comparing N h and N (t  1) . If N (t  1)  N h , system keeps running. Otherwise, stop running. Dynamic Compression Ratio Adjustment In the telerehabilitation’s data transmission, an optimal quality for video is needed according to the present bandwidth condition. That means system should transfer data as more as possible to improve the video quality. For this target, the dynamic data compression is applied. Dynamic compression means system will not compress a PCD in a certain ratio, but adjust this ratio according to the bandwidth condition. As the differential octree based compression with the PCD position detail compression, compression ratio is decided by the size of leaf cube. As factor for deciding the compression ratio, the size of leaf cube is equal to the depth of leaf node or bpp. In this thesis, the side length of leaf cube will be treated as the standard for deciding N (t ) at t second. Specifically,

p  N (t ) . The dynamic compression proposed in this thesis should

25 combine with the condition of system running. Formula for this combination is like this: N (t  1)  BWrequire (t  1) / BWreal (t )

Specifically, dynamic compression has no conflict to frame skipping. The compression ratio only connected with leaf node’s depth in differential octree. But scheme 1 of frame skipping calculates X levels from root level. As scheme 2, weight of the first level is the biggest. Specifically, the first level is the level next to root level. Along with the increasing of levels, the weight is also become smaller. In case of increasing N (t ) , Rabandon (t ) is also increasing for a further bandwidth saving. In case of reducing N (t ) , as scheme 1, when X levels are not including the level where new added cube is exist, Rabandon (t ) will not be inflicted and vice versa. As scheme 2, the depth of differential octree is bigger, the smaller inflict will be

Rabandon (t ) and vice versa. Speed-based Telerehabilitation System Introduction of Speed-based Telerehablitation System For testing the Dynamic Compression Scheme for Kinect PCD video, a Kinect PCD based Telerehabilitation System is designed. But this system checking patient’s rehabilitation only based on a motion speed. That is because the Dynamic Compression Scheme is only highlight motion speed. In this thesis, the system designed for testing is called a speed-based telerehabilitation system. Specifically, except testing the dynamic compression, the potential of Kinect PCD based

26 Telerehabilitation System also can be demonstrated by this speed-based telerehabilitation system. Main Object of PCL Utilized in Speed-based Telerehabilitation System Main objects in PCL that applied in the Speed-based Telerehabilitation System include: “OctreePointCloudChangeDetector”, “VoxelGrid”, “PassThrough”. A simple introduction for these objects is given below:  pcl::octree::OctreePointCloudChangeDetector For detecting a spatial changes between neighbors frames of PCD, system applied “OctreePointCloudChangeDetector” to work on this. The “OctreePointCloudChangeDetector” related code pattern is shown below [14]: // create OctreePointCloudChangeDetector instance pcl::octree::OctreePointCloudChangeDetector *octree(new pcl::octree::OctreePointCloudChangeDetector( resolution)); // assign point cloud to octree octree.setInputCloud (cloud); // add points from cloud to octree octree.addPointsFromInputCloud (); boost::shared_ptr newPointIdxVector (new std::vector); // get a vector of new points, which did not exist in previous buffer

27 octree.getPointIndicesFromNewVoxels (*newPointIdxVector); //switch octree buffers octree.switchBuffers ();  pcl::VoxelGrid “VoxelGrid” is applied for down sampling a point cloud to save bandwidth in the real time data transmission. The code pattern of “VoxelGrid” is shown below [15]: // create VoxelGrid filter instance pcl::VoxelGrid voxgrid; //set voxel size to x meters in each dimension voxgrid.setLeafSize(x,x,x); // set input cloud voxgrid.setInputCloud(cloud); // filtering and save down sampled point cloud in “new_cloud” voxgrid.filter(*new_cloud);  pcl::PassThrough “PassThrough” can filter something out from an image by set a range for x, y, z coordinate. The code pattern of “PassThrough” is shown below [15]: // create passthrough filter instance pcl::PassThrough pass; // set input cloud pass.setInputCloud (cloud); // set fieldname we want to filter over pass.setFilterFieldName ("z"); // set range for selected field to m-n meters

28 pass.setFilterLimits (m, n); //filtering and save filtered point cloud in “new_cloud” pass.filter(*new_cloud); Diagnosis Modes for Speed-based Telerehabilitation System The speed-based Telerehabilitation System in this thesis can work in three modes: Intelligent Diagnosis Mode (IDM), Hybrid Mode (HM), Real-time Diagnosis Mode (RDM). Concrete work principle for them is shown below:  Intelligent Diagnosis Mode (IDM) The IDM provide a convenient and fast way to make patients get rehabilitation condition by a self-service manner. Patient can use IDM to get a self-service diagnosising for most kinds of athletic rehabilitation at home without monitored by doctor online. IDM using pcl::octree::OctreePointCloudChangeDetector to detect the changes in a frame on contrast with last frame. According to the points size of different part in point cloud on contrast with last frame. System can get information of motion intensity of patient between neighbor frames. By processing this intensity value in every frame, system can check the rehabilitation condition of patient. In particular, the points size of different part in a frame on contrast with last frame is a basic indicator of motion intensity, in this thesis, author call this indicator as Different Size(DS), in the experiment, DS take absolute value, so DS≥0. In addition, define the total point size in a frame as Total Size (TS), if there exist TS’s differences between neighbor frames, author

29 define this differences as Different Total Size (DTS). The work steps of IDM are like these: Step 1: Background Checking Although patient has option to stand before the Kinect immediately and start moving to start an intelligent diagnosis, for keeping quality of medical care, patient should not stand before Kinect immediately after system worked in IDM and make system working for some rounds toward standstill background. If most frames have DS=0, that means background is stable enough and suitable for the intelligent diagnosis. Otherwise, patient should change a standstill background. Step 2: Intelligent Diagnosing Every round of diagnosis will grab 10 frames and then get a result of diagnosis as "good", "middle" or "not good" based on patient's performance. The principle of grading is like this: in every frame, system will record the digit number of DS. When a round is accomplished, system will statistic the sum of frames with a DS has 1 digit, 2 digits, and 3 digits respectively. And then, Base on this statistic result, system can do grading in the end of every round. The relationship between statistic and grade is like this: If 3 digit DS > 2 digit DS and 3 digit DS > 1 digit DS: Grade is "good" If 2 digit DS ≥ 3 digit DS and 2 digit DS ≥ 1 digit DS: Grade is "middle" If 1 digit DS ≥ 2 digit DS and 1 digit DS ≥ 3 digit DS: Grade is "not good"

30 In this pseudo code, if DS has a digit bigger than 3, system also count it as 3 digits. As the digit statistic method used in IDM, in theory, using a sum of DS of all frames in a round as the sum statistic method is simpler than the digit statistic method. But in practical, since a high sensitive and unstable character of PCD, sum statistic method cannot accurately evaluating condition of patient. Even when patient has a similar motion in two different frames before Kinect, sometimes, DS in these two frames also perform very different. As another deficiency of IDM, because of the character of PCD mentioned above, when system takes a standstill background checking, every frame should satisfy DS=0 in theory. But in practical, though most of frames have DS=0. But there also exist some exceptional case as DS equal to arbitrary value in scope of (0, 307200]. In this scope, 307200 is TS in every frame. In the IDM, TS is the same in every frame, in author’s research, a frame with 307200 points based on pcl::PointXYZRGBA can seize 10.5M space. If transporting such a big size frame online, the requirement of bandwidth will be very high. But the IDM only emphasize frame creation speed and has no requirement of real time transmission, so the TS in frame can be ignored. As the deficiencies mentioned above, IDM cannot provide a high accuracy diagnosis service in a single round. For keeping quality of the intelligent diagnosis, patient can take motion for some rounds and comparing outputs of different rounds. In the end, the most frequently outputted result can represent patient’s condition more closely. As a supplement of IDM for improving diagnosis’s accuracy, the system also can work in Half Intelligent and Half Real-time Mode (HM).

31  Hybrid Mode (HM) The HM can provide patient a more accurate diagnosis by combining intelligent manner and real-time manner together. And according to concrete athletic rehabilitation's character, HM has sub modes to respectively satisfy rehabilitation requirements for arm, leg, waist, and etc. In author’s research, sub mode of waist rehabilitation is provided as an example. HM 's design mainly using pcl::VoxelGrid and pcl::octree::OctreePointCloudChangeDetector. On contrast with IDM, HM calculate both DS and TS in every frame. In addition, every round of HM has 30 frames. Before detecting changes in a frame, system uses pcl::VoxelGrid to down sample original frame’s point cloud with 307200 points. This down sampling process is for saving bandwidth in real-time transport. So the DTS in HM is not always equal to 0 like DTS in IDM. The amount of sample points depends on two factors, one is the distance between Kinect and target object, the other is the surface complexity of target object and background. Bigger distance between target and Kinect is, or more complex the surface is, more sample points system can grabbing and vice versa. After a frame is created, system will transfer this frame with its TS, DS to server(doctor). In the end of every round, transcript will be created and it will also be sent to server. Because of down sampling process, the bandwidth is saved, that makes doctor can monitor more than 1 patient in the same time. In particular, since every frame will be down sampled when it was created, the frame creation speed becomes slower than it is in IDM. For reducing this adverse effect, patient should follow the instructions of system to take postures for

32 grabbing DS in every frame more accurately. The work steps of system in HM are like these: Step 1: Background Checking Different from background checking in IDM, this background checking is a forcible step since system should keep the amount of points suitably after down sampling to make sure the quality of doctor's monitor and saving bandwidth as more as possible. At the same time, it also requires a standstill background. In the research, the suitable number of points in a standstill background is in the range of [18000, 26000]; Step 2: Location Adjustment After the system detect background is suitable, patient can stand before Kinect to adjust his/her location. Since the distance between patient and Kinect is smaller than distance between background and Kinect. After patient standing before Kinect, the sample points on patient are sparser than they are on the background. So the TS in a frame have a grand change at this time. By utilizing this character, system can detect if patient already stand before Kinect or not. When system detect a frame's TS in scope of [12000, 13000], the system will note patient his/her location is good and keeping a standstill. After 10 frames, if all of TS in these 10 frames have a value in the range. System will take next step. Otherwise, the system will note patient come closer or go more distance to Kinect according to the TS. The pseudo code of location adjustment is like this: If TS≥12000 and TS≤13000: location is good; If TS13000: come closer to Kinect; Step 3: System Training After system detect suitable location of patient. The system will note patient start moving on instructions. But a question is like this, if system starts round now, under most conditions, first two or three frames usually get a DS=0 or other value which cannot express motion in reality. That is mainly because the system is not stable from a standstill situation to a changes exist situation in such a short time. So between the location adjustment and the system recording, the system training is added for making system experience a transitional period. This measure can improving accuracy of transcript created in the end of every round; Step 4: System Recording This process makes the round started. A round in HM has 30 frames in calculation. Every frame with its TS, DS will be transferred to a server in real-time when they are created. Doctor can monitor the frame and its TS, DS to check if the transcript created in the end of a round precisely enough to descript rehabilitation condition of the patient. After doctor takes a checking and gets a result, he/she can make this result with some suggestions responding to patient. In addition, the intelligent evaluation for patient’s performance in the end of system recording also uses the digit statistic method like in IDM. The pseudo code for digit statistic method in HM is shown below: If 1 digit DS>20: Grade is “system cannot detect your action” If 4 digit DS>3 digit DS and 4 digit DS>2 digit DS and 4 digit DS>1 digit DS:

34 Grade is “good” If 3 digit DS≥4 digit DS and 3 digit DS≥2 digit DS and 3 digit DS≥1 digit DS: Grade is “middle” If 2 digit DS>4 digit DS and 2 digit DS>3 digit DS and 2 digit DS≥1 digit DS: Grade is “not good” Some deficiencies because of character of PCD also have an adverse effect to HM's accuracy of intelligent diagnosing like in IDM. In addition, HM requires patient follow instruction of system strictly. It produces some inconvenient to patient. In the end, different sub mode of HM needs a different standard to take intelligent checking and the system complexity will be improved by adding sub modes. This mode is added for the situation that doctor might require patient take motion which HM have no sub mode matching it or patient need a real-time diagnosis. In these conditions, RDM can be applied for a further diagnosing. The RDM part is designed mainly by pcl::PassThrough. There has no need to calculate DS for every frame because RDM contains no intelligent diagnosis. RDM only concerns about TS and frame creation speed. As the TS in RDM, it should big enough to provide a high quality image in frame. At the same time, it should not surpass network transport ability to affect video quality in the side of doctor. Based on these concerns, RDM using pcl::PassThrough to extract patient from background, and then transfer frames to server after the background filtering.

RESULTS AND DISCUSSION Phase 1 of Dynamic Compression Scheme’s Feasibility In this part, a C++ programming is applied for experiment of simulating the work processes in Dynamic Compression Scheme’s phase 1. Phase 1 is “checking transmission’s possibility”. In this experiment, BWrequire (t  1) and BWreal (t ) is given. Based on BWrequire (t  1) and BWreal (t ) ,

N (t  1) is calculated. Since this experiment is at t second. So N (t  1) is a constant value. That means BWrequire ( Ptransport (t ), N (t  1)) can be replaced as

BWrequire ( Ptransport (t )) . It will not inflict accuracy of the experiment. In this experiment, assuming system transmission ability is 30fps and every frame has a size of 3MB. Based on this assumption, the predictive minimum bandwidth requirement is 720Mbps. In the telerehabilitation, a pre-established N h is needed. In this experiment, assuming Nh  100times . After take a check in “condition of system running”, if there has a N (t  1)  N h , system will go to the phase of transmission. Table 1 shows N (t  1) outputted base on different Ptransport (t ) and BWreal (t ) . Since in a diagnosis, the patient’s motion speed usually is slow. The speed of transmission generally is in scope of [5, 10] fps. If bandwidth is 10Mbps, and compression ratio is in scope of [12, 24], according to [7], this compression ratio can keep the quality of video. Based on Nh  100times , the condition of system running’s work is shown in Table 2. In Table 2, ‘0’ means system can go into the phase of transmission, ‘1’ means system cannot go into the phase of transmission.

36 Experiment output above shows phase 1 of the Dynamic Compression Scheme has a good feasibility and operability. Table 1. Compression ratio output under different situation BWreal (t )

1 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30

Ptransport (t )

5 120 60 30 20 15 12 10 8 7 6 6 5 5 4 4 4

10 240 120 60 40 30 24 20 17 15 13 12 10 10 9 8 8

15 360 180 90 60 45 36 30 25 22 20 18 16 15 13 12 12

20 480 240 120 80 60 48 40 34 30 26 24 21 20 18 17 16

25 600 300 150 100 75 60 50 42 37 33 30 27 25 23 21 20

30 720 360 180 120 90 72 60 51 45 40 36 32 30 27 25 24

Table 2. Condition of system running’s output under different situation BWreal (t )

1 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30

Ptransport (t )

5 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0

10 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0

15 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0

20 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0

25 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0

30 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0

37 Speed-based Telerehabilitation System Intelligent Diagnosis Mode (IDM) Step 1: Background checking. A success background checking has data outputted similar to Table 3, the corresponding point cloud image is shown in Figure 6. In IDM, TS has a certain value 307200. In the background checking, if patient find most frames in a round has a DS=0, he/she can assume there has a standstill background. Because the system will not take an intelligent background checking, so patient should take a check by himself/herself. It is worth to note that, in frame 6, there has a DS=55. This is an exceptional case because of point cloud data’s unstable and sensitive character. And because there has no patient to take any motion in the background, so the intelligent diagnosis result is “not good”. In IDM, the background checking is an optional process for patient. But author recommends patient to take this process to make sure a standstill background. In the end, specifically, every round of IDM has 10 frames in calculation. Step 2: Intelligent diagnosing. PCD in intelligent diagnosing is shown in Table 4, the corresponding PCD image examples are shown in Figure 7 and Figure 8. Table 4 shows a “good” outputted intelligent diagnosis result based on a digit statistic. As an exceptional case like frame 9, system detect no motion and take a DS=0. The corresponding PCD image for this case is shown in Figure 7. And other frames’ corresponding PCD images are similar to PCD image shown in Figure 8. In particular, the red points represent DS part in Figure 8. The red points are more in amount, the changes are bigger.

38 Table 3. PCD output for case of success background checking Frame

TS

DS

1

307200

0

2

307200

0

3

307200

0

4

307200

0

5

307200

0

6

307200

55

7

307200

0

8

307200

0

9

307200

0

10

307200

0

Figure 6. PCD image of suitable background in IDM

39 Table 4. PCD output for case of “good” in IDM Frame

TS

DS

1

307200

2119

2

307200

198322

3

307200

33541

4

307200

107296

5

307200

20958

6

307200

14320

7

307200

22686

8

307200

124318

9

307200

0

10

307200

20865

Figure 7. PCD image with DS=0 in IDM

40

Figure 8. PCD image with DS ≠0 in IDM Half Intelligent and Half Real-time Mode (HM) Step1: Background checking. The background checking allow TS in a scope of [18000, 26000], Table 5 shows PCD output in a success background checking and corresponding PCD image example is shown in Figure 9.In addition, DS>600 also will lead an unsuccessful checking because that means background is not stable. The background checking in HM is an intelligent and forcible process for keep the diagnosis quality. Every round of background checking in HM has 10 frames. On contrast with IDM’s situation that most DS has a DS=0 in background checking, in the HM, there has very few DS=0. That is mainly because IDM’s TS is a certain value (307200), but HM’s TS is not certain after down sampling process. The changes of TS in neighbor frames can be indicated by DTS. DTS also take an absolute value like DS. Specifically, in background checking’s data outputted in Table 5. The compression ratio is between

41 [14.33, 14.58] times. In addition, Figure 9 shows the down sampled image on this compression ratio. Table 5. PCD output for case of success background checking in HM Frame

TS

DS

DTS

1

21149

14

82

2

21360

26

211

3

21291

64

69

4

21071

21

220

5

21440

4

369

6

21423

48

17

7

21393

4

30

8

21232

12

161

9

21384

48

152

10

21230

15

154

Figure 9. PCD image output example for case of success background checking in HM

42 Step 2: Location adjustment. In the location adjustment, patient can stand behind Kinect to adjust location. After patient stand behind Kinect, TS will reduce massively between neighbor frames like frame 6 and frame 7 in Table 6, value of TS reduced from 21114 to 13916. By utilizing this character, system can take a check if patient already stand behind Kinect by DTS>2000 satisfied or not. After patient is detected, suitable location will make TS in a scope of [12000, 13000]. If TS13000, system will note patient to come closer to Kinect. A success location adjustment’s PCD is shown in Table 7 and corresponding PCD image is shown in Figure 10. In this step, as showed in Table 7, compression ratio becomes bigger which is in scope of [24.01, 25.23] since patient is closer to Kinect than background and sample points are sparser on the patient’s body than on the background. In addition, note frame 2 and frame 3 in Table 6. , though DTS≠0, DS=0. That means changes of DTS have no connection with changes of DS. And since character of PCD, in Table 6, there has no neighbor frames has the same TS. Actually, most DTS in HM has a DTS≠0. Table 6. PCD output for case of people detecting in HM Frame TS DS 1 21152 3 2 21275 77 3 21366 0 4 21292 2 5 21404 0 6 21114 47 7 13916 308

43 Table 7. PCD output for case of success location adjustment in HM Frame

TS

DS

1

12797

6

2

12565

2

3

12291

3

4

12184

3

5

12178

2

6

12358

30

7

12364

5

8

12276

4

9

12396

30

10

12355

1

Figure 10. PCD image output example for case of success location adjustment in HI&HRM

44 Step 3: System training. The system training’s PCD output is shown in Table 8. After patient starts to take a motion, frame 1 to 4 did not detect motion in time. But from frame 5, the motion was detected in experiment since there has a grand change between neighbor DSs(from 97 to 2077). From step 3, patient should follow system’s instructions. That is because the frame creation speed becomes slower after the down sampling process. As the sub mode for waist, “left” means move waist to the left direction and “right” means move waist to the right direction. Because of system’s reaction and patient’s posture cannot be kept in a same step. So the system training process is very needed for keeping system’s intelligent output accurately. Table 8. PCD Output for Case of System Training in HM Frame

Instruction

TS

DS

1

Left

12655

2

2

Right

12549

0

3

Left

12759

9

4

Right

12530

97

5

Left

12941

2077

Step 4: System recording. The system recording’s PCD output is shown in Table 9. In the end of recording, the corresponding digit statistic output is shown in Table 10. In this statistic case, diagnosis output is “good”. The process of system recording takes 30 frames in count. In theory, DS can indicate the motion speed, but the digit statistic is applied for evaluating motion speed, because this manner can reducing adverse effects of PCD’s unstable and sensitive character.

45 Table 9. PCD Output for Case of System Recording in HM Frame Instruction TS DS 1 Right 12005 3908 2 Left 13037 2136 3 Right 12355 4146 4 Left 12752 1878 5 Right 12610 24 6 Left 12066 3449 7 Right 12929 1937 8 Left 13054 2 9 Right 12291 4288 10 Left 12811 1909 11 Right 12197 3920 12 Left 12664 1860 13 Right 12173 3713 14 Left 13145 2130 15 Right 12619 4358 16 Left 12834 1893 17 Right 12419 4150 18 Left 12150 64 19 Right 12115 850 20 Left 12063 3151 21 Right 12757 1913 22 Left 12267 4140 23 Right 12990 2038 24 Left 12119 3726 25 Right 12962 2175 26 Left 12343 4241 27 Right 12922 2147 28 Left 12370 4485 29 Right 12278 0 30 Left 13230 2410 Table 10. Digit statistic output for table 7 DS with digit

Amount

1

2

2

2

3

1

4

25

46 Real-time Diagnosis Mode (RDM) Step 1: Background checking. A success background checking is shown in Table 11. The character of success checking is TS=0 for all of 10 frames in a round. In RDM, because there has no down sampling process, so the frame creation speed is faster than it is in HM. RDM grabbing PCD in a limited scope for x,y,z coordinates. So the successful background checking in Table 11 has a TS=0 for all frames. Table 11. PCD Output for Case of Success Background Checking in RDM Frame 1

TS 0

2

0

3

0

4

0

5

0

6

0

7

0

8

0

9

0

10

0 Step 2: Real-time diagnosis. PCD output and corresponding

example of PCD image in real-time diagnosis is shown in Table 12 and Figure 11 respectively. Real-time diagnosis has no intelligent diagnosis’s round calculation. It only highlights TS in every frame because that will affect real-time transmission. TS mentioned here is only include the points in describing the patient’s body but the background.

47 Table 12. PCD Output for Case of Real-time Diagnosis in RDM Frame 1

TS 67390

2

71152

3

74480

4

76582

5

74262

6

74262

7

74721

8

77124

9

71382

10

71372

11

72431

12

77598

13

72059





Figure 11. PCD output example for case of real-time diagnosis in RDM

CONCLUSION The Dynamic Compression Scheme for Kinect PCD video designed in this thesis has been evaluated in its realizable possibility in the phase of Checking Transmission’s Possibility by a C++ simulation. So the potential of applying this scheme to Kinect PCD based Telerehabilitation System also can be demonstrated. But because of the technology limitation nowadays, the combination possibility of these two phases (“Checking Transmission’s Possibility” and “Transmission”) in dynamic compression has not been demonstrated. On the other hand, author’s time is also limited, so the further research and development is future work. In addition, Speed-based Telerehabilitaion System designed in this thesis has demonstrated the potential of developing Kinect PCD based Telerehabilitation System. In the future work, author will add a new feature as a posture evalutation to this system to accomplish prototype of Kinect PCD based Telerehabilitation System. On the other hand, this speed-based telerehabilitation system also can be treated as a test system for Kinect PCD’s dynamic compression and transmission in the future research.

REFERENCES [1]DING Chen, WANG Jun-ze, QU Chang, et al., “Kinect Somatosensory Interaction Technology and Its Application in Medical Rehabilitation,” Chinese Journal of Rehabilitation Theory and Practice, vol. 19, no. 2, pp. 136-138, 2013. [2]David Antón, Alfredo Goñi, Arantza Illarramendi, Juan José TorresUnda, and Jesús Seco, “KiReS: A Kinect-based telerehabilitation system,” in IEEE 15th International Conference on e-Health Networking, Applications and Services, Lisbon, Portugal, October 912 2013. [3]“Hospital of Zhejiang Province take Kinect for Windows for Rehabilitation Medical Care,” http://soft.zdnet.com.cn/software_zone/2014/0110/3008488.shtm l, 2014. [4]Nico Blodow, Radu Bogdan Rusu, Suat Gedikli, Michael Beetz, Eckehard Steinbach, “Real-time compression of point cloud streams,” in IEEE International Conference on Robotics and Automation (ICRA), Saint Paul, U.S. , May 14-18 2012. [5]Yuxue Fan, Yan Huang, Jingliang Peng, “Point cloud compression based on hierarchical point clustering,” in Signal and Information Processing Association Annual Summit and Conference (APSIPA), Kaohsiung, Taiwan, October 29–November 1 2013. [6]Jingjing Fu, Dan Miao, Weiren Yu, Shiqi Wang, Yan Lu, Shipeng Li, “Kinect-Like Depth Data Compression,” IEEE Transactions on Multimedia, vol. 15, no. 2, pp. 1340-1351, 2013. [7]Neumann D, Lugauer F, Bauer S, Wasza J, Hornegger J, “Real-time RGB-D mapping and 3-D modeling on the GPU using the random ball cover data structure,” in IEEE International Conference on Computer Vision Workshops (ICCV Workshops), Barcelona, Spain, November 6-13 2011. [8]Cruz L, Rio de Janeiro, Lucio D, Velho L, “Kinect and RGBD Images: Challenges and Applications,” in 25th SIBGRAPI Conference on Graphics, Patterns and Images Tutorials (SIBGRAPI-T), Ouro Preto, Brazil, Aug 22-25 2012. [9] “Kinect for Windows,” https://www.microsoft.com/enus/kinectforwindows/develop/, 2015.

50 [10]Leandro Cruz, Djalma Lucio, Luiz Velho, “Kinect and RGBD Images: Challenges and Applications,” in XXV SIBGRAPI Conference on Graphics, Patterns and Images Tutorials, Ouro Preto, Brazil, August 22-25 2012. [11]ZHANG Yi, ZHANG Shuo, LUO Yuan, XU Xiao-don, “Gesture track recognition based on Kinect depth image information and its applications,” Application Research of Computers, vol. 29, no. 9, pp. 3547-3550, September 2012. [12]Radu Bogdan Rusu, Steve Cousins, “3d is here: Point cloud library (pcl),” in IEEE International Conference on Robotics and Automation (ICRA), Shanghai, China, May 9-13 2011. [13]QUAN Yu-shu, HE Ming-yi, “Encoding Compression Algorithm of Linear Octree Based on Three-Dimensional Point Cloud Data,” Application Research of Computers, pp. 70-71, 2005. [14]Julius Kammerl, “Search, change detection, and compression using octrees - Point Cloud Library,” http://www.pointclouds.org/assets/icra2012/search.pdf, 2011. [15]Suat Gedikli, “PCL :: Filtering - Point Cloud Library,” http://www.pointclouds.org/assets/iros2011/filtering.pdf, 2011.

Fresno State Non-Exclusive Distribution License (to archive your thesis/dissertation electronically via the library’s eCollections database) By submitting this license, you (the author or copyright holder) grant to Fresno State Digital Scholar the non-exclusive right to reproduce, translate (as defined in the next paragraph), and/or distribute your submission (including the abstract) worldwide in print and electronic format and in any medium, including but not limited to audio or video. You agree that Fresno State may, without changing the content, translate the submission to any medium or format for the purpose of preservation. You also agree that the submission is your original work, and that you have the right to grant the rights contained in this license. You also represent that your submission does not, to the best of your knowledge, infringe upon anyone’s copyright. If the submission reproduces material for which you do not hold copyright and that would not be considered fair use outside the copyright law, you represent that you have obtained the unrestricted permission of the copyright owner to grant Fresno State the rights required by this license, and that such third-party material is clearly identified and acknowledged within the text or content of the submission. If the submission is based upon work that has been sponsored or supported by an agency or organization other than Fresno State, you represent that you have fulfilled any right of review or other obligations required by such contract or agreement. Fresno State will clearly identify your name as the author or owner of the submission and will not make any alteration, other than as allowed by this license, to your submission. By typing your name and date in the fields below, you indicate your agreement to the terms of this distribution license. Embargo options (fill box with an X). x

Make my thesis or dissertation available to eCollections immediately upon submission. Embargo my thesis or dissertation for a period of 2 years from date of graduation. Embargo my thesis or dissertation for a period of 5 years from date of graduation.

Yilin Chen Type full name as it appears on submission

November 23, 2015 Date