Man-Computer Interaction System Using Kinect Geovane Griesang (UNISC ) [email protected] Rafael Peiter (UNISC ) [email protected] Rolf Fredi Molz (UNISC ) [email protected]

With the advancement of technology, more and more innovative ideas emerge to improve the interaction between men and computer systems. In this context arose devices capable of capturing the movements of a person and, through it, manipulate these gestures. One of the devices that came with this purpose was Kinect. This new technology motivated this project. The objective is to develop a communication interface between a person and a computer using Kinect as an input device. The system will provide an electronic whiteboard functionality that will be directed to public and private schools. Another feature that has been developed is a slide showing using specific movements. Furthermore, the implemented system aims to recognize hand gestures simulating the behavior of the mouse. Palavras-chaves: Kinect, interaction between men and computer systems, capturing the movements, electronic whiteboard.

1

ICIEOM – CIO 2013 – Valladolid, Spain

1

Introduction

According to Carroll (1991) the human-computer interaction is an interdisciplinary area of applied research to design practice. Its main purpose is to understand and facilitate the creation of "user interfaces", i.e. computers being manipulated by human users. With this, we intend to use this concept in many areas such as psychology, computer science, anthropology, management science, industrial design. One of the first companies that invested in human-computer interaction was Xerox. In 1970 this company founded a new research center in the city of Palo Alto in the state of California in the United States for the purpose of exploring digital electronic technologies. Since that time, the research center in Palo Alto has become known for its developments in interactive computing, based on personal computers. Since its inception there were discussions about the possibility of an active role for psychological research on the interaction between humans and computers (Moran et al., 1983). It was then that emerged the most known devices and used recently: Monitor, keyboard and mouse. Even with all the technological developments that occurred in the area of information technology, these devices remain virtually the same, changing only some specific features. The Kinect has emerged as a new alternative in using a camera to recognize and interpret what is happening in the environment and information obtained through work on them. Considering the amount of resources, which are discussed later, this study was conducted using this device.

2

Kinect

In November 2010 Microsoft launched its new product called Kinect. It appeared as an accessory for the Xbox 360 console offering an innovative interaction without using traditional controls. The first time it was mentioned was in June 2009, until then known as "Project Natal." Its launch was highly awaited to be dealing with a futuristic technology. In its first two weeks, the equivalent of eight million units was sold (MICROSOFT NEWS CENTER, 2012). In the same month of its release, there were already people interested in using the device for other purposes. Many have tried to launch drivers to be used mainly in computers running Windows or Linux. The highlight was the company PrimeSense that in December 2010 provided the drivers with the source code along with middleware called NITE, this package became known as OpenNI, shortly after Microsoft launched its own development drivers, known as Microsoft Kinect SDK.

2.1 Technologies The Kinect was originally developed for games and similar scenarios. With interest emerging market, Microsoft launched in February 2012 a Kinect exclusive to the PC with a standard USB. The device for Xbox has a USB input a little different, must be connected to an adapter to be connected to the PC. It has a sensor that is capable of capturing the motions depending on distance. This equipment developed by Microsoft offers a new means of interactivity. The accessory is capable of mapping the movements of all body parts. It was developed by combining multiple technologies such as the use of video camera, depth sensor and multi-array microphone.

2

ICIEOM – CIO 2013 – Valladolid, Spain

2.2 Development Originally the only way to develop applications for the Kinect was using a development kit (XDK) supplied by Microsoft to its partners so they could produce unique applications for the Xbox console. This changed dramatically a few days after its launch, software drivers were able to interpret the signal from the USB port of Kinect, which was then written and released as open source software on the Internet later known as OpenKinect or Libfreekinect. With the availability of these drivers any application using the Kinect could be produced (KEAN et al., 2011). According to Kean and colleagues (2011), although developers can use OpenKinect/libfreenect, it still left much to be desired. The complexity of producing an application motivated the company PrimeSense, the same company that developed the depth sensing technology in the Kinect, to launch OpenNI framework. This software came with tools that increased the development productivity; this system was made available in open source along with a closed source system to the middleware, called NITE. This package was similar to the technology used in games developed for the Xbox development kit. After the release of OpenNI, Microsoft announced its official development kit for Windows that became available in 2011 for non-commercial use. To develop the proposed system these two frameworks have been studied, each one has its specific features as well as advantages and disadvantages. The OpenNI has a longer maturation and development, it is clear that several improvements have already been made on this technology. There is also a large community of people involved updating and helping in the evolution of the framework. One advantage of OpenNI over Microsoft Kinect SDK is the cross-platform, which can produce an application that runs on Windows operating systems, Linux or Mac OS, while Microsoft is still exclusive to devices with the Windows platform. The objective is to produce the idea of a system limiting it to only one platform, making its development easier and faster. Despite these differences, the two frameworks are very similar in many aspects, especially because they improve as new needs arise. Problems that used to occur with the OpenNI, such as the need to calibrate it before using the device, are no longer recurring in the new versions. The framework chosen to be used in this project was the Microsoft Kinect SDK for various reasons. Firstly, the manufacturer invests a lot of money on their technology, even though there are a lot of people offering help in OpenNI, the fact of financially investing on their technology ensures commitment and the objective of reaching prominence. Moreover, the fact of being a technology produced by the same company that makes the hardware makes its software more optimized because the manufacturer knows better all the components used in Kinect. A major problem was the inability to publish a product in beta versions, but this problem was solved in its latest release. The ease of installation and development are also major factors in this decision, because an application using this framework can be developed in a short time. The choice of the framework to be used is essential for the system modeling. Upon choosing Microsoft Kinect SDK, the way to develop the layers that make up this system was based on the characteristics of this framework. The next chapter will address the modeling system and the use of the technologies described in this chapter.

3

Modeling system

A detailed study was conducted on how teachers of university UNISC and the school São Luís in Santa Cruz do Sul use the electronic whiteboard (in Portuguese, lousa eletrônica). In this research, it was found that teachers develop a slideshow typically in programs of PowerPoint or Impress, putting the description of the content to be worked in class. By

3

ICIEOM – CIO 2013 – Valladolid, Spain submitting content to students, the main feature observed is the possibility to draw illustrations and markings on the slides using an electronic pen. Thus, the teacher has more freedom in the way he/she presents the content and the students have a more refined attention on how it is presented. In the project, the user can freely move his/her hand, simulating the same movement directions of the mouse. By moving the hand up, the screen cursor will be moved up, and so for the other directions. To make a better use of this feature, a specific application to interpret the movements has been developed. In order to provide an interactive interface to the host and at the same time interesting to the audience, a specific application for the presentation of a document has been developed. The user can import a document created on PowerPoint or Impress to make the presentation in this application, the presenter will be able to make a specific gesture to move the slides forwards or backwards. The screen cursor is represented by a hand aiming a better visibility to the audience and permitting the presenter to make better use of it. This makes it easier and more intuitive to highlight words or phrases. The proposed application has the function of free drawing. This function provides, at the time of making the presentation, the ability to draw anything on the screen using hand movements. Therefore, a specific movement has been defined to indicate that upon moving the cursor on the screen, a line will be drawn according to the movement made. All these features enabled the development of the modules of electronic whiteboard and slideshow in a single application. In the next section it will be explained how this project was implemented and what tools were used.

3.1 Technical specification For the use of the proposed system some requirements are necessary. The first one refers to the platform – Windows operating system version 7.0 should be used. This system has mainly been chosen because Microsoft Kinect SDK framework is exclusive to the platform. Any operating system that runs Windows 7.0 without bugs will be able to use this system. This work was done in a way so there is communication among applications that use different languages and concepts. Figure 1 shows the structure of the developed system, all its components will be discussed in the following sections.

Figure 1: Developed system structure. Source: (author, 2013)

3.2 Capturing movements For the development of the motion capture layer of Kinect, IDE Visual Studio 2010 with C# language has been used. Using this IDE with C#, the development is high quality and fast, since most manuals and examples are available in this language.

4

ICIEOM – CIO 2013 – Valladolid, Spain The motion capture layer is responsible for identifying the movements made by a person and performing this action after its detection. The software is called "Kinect Controller"; it is responsible for the communication with the Kinect device. When the application is started, it checks whether the device is connected or disconnected. Once it is initialized and verificated that the device is connected, the program can be minimized to the taskbar to be running like an operating system service. From this moment on, the program waits until it receives the instruction of what needs to be recognized and the response that should be returned, this mapping will be explained in detail in section 3.4. Any application to be developed will be able to communicate with this service using the communication layer. In the case of application of electronic whiteboard, some information are sent to be interpreted and responded. The electronic whiteboard uses hand movement to simulate the behavior of the mouse cursor. The Kinect controller interprets the current position of the right hand and applies the position of the mouse cursor on the screen. To do this mapping it is necessary to know the resolution used in the operational system. Through the resolution, a scale is applied to determine in which position the mouse cursor should move. The ease of developing specific gestures for this service is due to the number of resources provided by Microsoft Kinect SDK framework. By making the mapping of the person's body, it is possible to capture any portion thereof. Each body part mapped is returned to its position (x, y, z) in relation to the current plan. Due to this functionality it was possible to develop specific gestures to be used on the whiteboard. Two types of gestures were developed to pass slides forwards and backwards, this function is called Swype. In the development of a new movement, one should always make the comparison among the body parts, according to the movements described in this section. The body in an environment does not stay in an invariable position, so the best way to detect if a particular gesture is being performed is to make this kind of comparison. The next section will detail the application layer that will use the movements that have been described.

3.3 Application T The implementation of electronic whiteboard was developed in C# using Microsoft Visual Studio 2010 IDE. The choice is due to some facilities found to manage the implementation in an interactive way. However, nothing prevents new modules to be developed using other languages such as Java, C#. As demonstrated in Figure 1, the applications may interact with the layer of motion capture independently of the programming language, therefore, it was developed a form of communication between these two layers, which will be explained in the next section. Upon opening the electronic whiteboard, it is possible to choose among four options: start, upload, settings and exit, as shows Figure 2.

5

ICIEOM – CIO 2013 – Valladolid, Spain

Figure 2: Initial screen of electronic whiteboard. Source: (author, 2013)

Before starting the presentation it is necessary to select the document that will be open by using the option “upload” (Carregar). All presentations contained in the Documents folder will be shown in this option. Through the option “Settings” (Configurar), it is possible to choose which gesture will be used to pass the slides. This setting does not need to be performed whenever entering the application; it gets saved in the Settings file. The option “Exit” (Sair) simply closes the implementation of electronic whiteboard. After loading the document to be submitted, the next step is to select the option “Start” (Iniciar) from the main screen. Once presentation has been started, some features are provided as shown in Figure 3:

Figure 3: Presentation screen of a document. Source: (author, 2013)

  

1 - Exit presentation (Sair da apresentação): finalizes the presentation and returns to the home screen of the program. 2 - Mouse cursor (Cursor do mouse): the moving of the right hand moves the cursor that is represented by the picture of a hand in White color. 3 - Return to the previous slide (Retornar para o slide anterior): to return to the previous slide, one can use the button itself or the gesture defined in the application configuration.

6

ICIEOM – CIO 2013 – Valladolid, Spain 

4 - Go to the next slide (Avançar para o próximo slide): to go to the next slide, one can use the button itself or the gesture defined in the application configuration. The full presentation of a document can be made through these features. However, a very important resource for the whiteboard is the drawing function. A concept of drawing on the presentation was developed with the main objective of marking or highlighting something important in the presentation. All the elements described in this section made it possible to develop the specific application of electronic whiteboard. It should be used together with the capture layer. Importantly, each application that uses this methodology should be developed based on some relevant concepts to build an interface to work with different types of movements. The larger the icons are the easier and faster the user will select them. Upon opening the application developed, a message will be sent to the layer of motion capture, warning that it was started and informing the gestures to be recognized. This communication will be discussed in the next section.

3.4 Integration among the layers To integrate the motion capture with the applications, it was developed a concept of communication between them. First must be initialized motion capture developed, the controller Kinect. As explained earlier, it will run as a service by making the recognition of movements. Upon start the application developed in the case of this work the electronic whiteboard, a file is created in a specific folder. When it detects that a file is new in the area of monitoring, motion capture makes loading the parameters contained in this file. Communication is done via an xml file. Xml’s files can describe any type of data. One of its advantages is the ease of sharing information, in addition to its simplicity readability (W3Schools, 2012). So to start the application xml file is sent to the folder monitoring controller Kinect, this folder is called "controls". It is contained in the same folder of application controller installation. One of the main advantages of this form of communication is the ability of each application have their own settings of movement. The user can choose whether to use the A or B gesture to advance the slides, such as used in the type of slide movement. In the settings file can be created your own gestures, i.e. informing some values the capture layer interprets the movements and recognizes them. This type of movement is specifically for implementing a position in a period of time. For example, stick to the left and right hands raised for five seconds. With all that has been said, it is possible to produce an application that makes complete integration with the controller Kinect. Insofar as new needs arise, new parameters can be added to complete the xml file.

3.5 Budget solutions market related to the proposed system In Brazil there isn't still much publicity on the internet about market solutions with the Kinect, although it could have various applications in different areas, companies have not released any big news about the device. There are some small independent projects, but none with commercial focus was found in this research. Out of the country there are companies trying to become a market leader using the new technology. The German company Evoluce is one pioneer, it developed a software called Win&I. According to analyses of Danilo Amoroso (2011), this software aims to simulate the mouse using hand gestures, but this is done with some problems and inaccuracy. It is hard to simulate a Click through hand movements, sometimes the user just ends up dragging icons or folders unintentionally. Moreover, the software is not free, to use this benefit it is necessary to pay an amount equivalent to R$ 91,00 (R$ 1,00 costs about USD 0,50) (EVOLUCE, 2012).

7

ICIEOM – CIO 2013 – Valladolid, Spain The StagePresence solution developed by the company Nuvixa is mainly based on the idea of producing video lessons with the help of the Kinect. The program has some interesting features such as importing a presentation developed in Power Point. The software developed by the company permits the transition of slides backwards and forwards with a specific movement, while the video is being recorded. The program has the feature of uploading the video directly to the company’s website. Although, the software still has its limitations, it offers no further options besides those cited above. Another disadvantage is the need to be connected to the internet to access the program (NUVIXA, 2012). The company Luidia has developed a product that is differentiated from others, known as eBeam. Unlike previous solutions, it uses its own receiver and an electronic pen to communicate with the computer. The pen works like a mouse, by touching the tip of it on some sort of solid surface its movements are transmitted to the computer that is connected to the receiver. In spite of having several features, the installation of all its equipment is complicated, since it is very difficult to fix the receiver on the wall. Another downside is its cost, the sensors themselves cost around R$ 1.400,00. Comparing this value to the Kinect sensor the difference would be at least R$ 1.000,00 (LUIDIA, 2012). Table 1: Comparison among the technologies described Present project Win&I Project cost Low Low Specific applications Yes No Communication systems Kinect Kinect Technologic Microsoft Kinect SDK Microsoft Kinect SDK Source: (author, 2013)

eBeam High Yes Sensor and electronic pen Electronic Whiteboard

StagePresence Variable Yes Kinect OpenNI

Analyzing the Table 1, among the solutions available in the market, there is not now a lowcost technology to aid teachers in schools and professors in universities. Among the projects mentioned, only one has a low cost while others have a high cost or variable that has low cost has not developed specific applications, limiting the use of the Kinect. The best proposal for the type of project proposed in this work was the Luidia's company that provides interesting tools for teachers. Checking the budget of each solution, one can reach the conclusion that the values are a little high for what each project proposes to do. One goal of this work is to provide a project with a low cost to schools and universities. Analyzing table 2 can be proven that statement. Table 2: Budget of present project Material Netbook CCE N23S com Intel Atom N435, 2GB, 320GB, na loja PontoFrio. Kinect, mercado livre Projetor Epson S10 2600 Lumens - 350 Polegadas 800 X 600, na loja Panamericanas PARTIAL AMOUNT Markup of 30% Total (excluding state and federal taxes) Source: (author, 2013)

4

Price(R$) 699,00 340,00 1049,56 2088,56 626,56 2715,12

Conclusion

Hoping to develop innovative projects to be used in a near future, many companies try to introduce new technological trends for the current market, and so does this project, while considering users’ new needs. Technological advances and the current interest in launching new tendencies in the field of computer-man interaction enabled the access to the technologies described in this work. The Kinect and its various possibilities of interaction motivated the present work. The form of integration among the layers allows new applications to be developed using the same concept. The electronic whiteboard technology itself can be improved as new needs appear.

8

ICIEOM – CIO 2013 – Valladolid, Spain The cost of market of similar systems is high considering current standards. This statement is proved in market research presented in section 3.5. Thus, if compared with the values of the projects developed by many companies, the proposed work is effective, generating a competitive market system, with its cost around R$ 2.715,12. Therefore, through the topics presented throughout this work, the arguments and conclusions taken, it is believed that all objectives were achieved. It was developed an interface of complete integration, which may be reused in the future. The application developed can be used both in schools and in universities, though some tests are still necessary, the whiteboard should be tested in study environments to verify the need of any adjusts. It's believed that new ideas for more features in electronic whiteboard will come with its use and exposure. The Kinect controller may also be improved as new types of movements are created to be used by the application of the whiteboard itself or other applications that may be developed. Thus, this work is finalized, intending the development of the product as a trial version.

References AMOROSO, Daniel. (2012). O Win&I, software para usar o Kinect no Windows 7, é um desastre. Accessed in February, 25, 2012. Available at: http://www.tecmundo.com.br/kinect/9640-o-win-i-software-para-usar-o-kinect-no-windows-7e-um-desastre.htm. CRAWFORD, Stephanie. (2012). “Kinect Sensor for Xbox 360 – Formerly Project Natal”. Accessed in January, 10, 2012. Available at: http://eletronicos.hsw.uol.com.br/microsoft-kinect2.htm. EVOLUCE. (2012). “Win&I Gesture Control Software?”. Accessed in February, 25, 2012. Available at: http://www.evoluce.com/_win-and-i/en/software/overview/index.php. KEAN, Sean. et al. (2011). Meet the Kinect: Programming and Scripting Natural User Interfaces (1st ed.) Springer Science+Business Media New York. LUIDIA, eBeam. (2012). Accessed in March, 5, 2012. Avaliable at: http://www.e-beam.com/home.html. MICROSOFT KINECT SDK. (2012). Kinect for Windows SDK. Accessed in April, 1, 2012. Avaliable at: http://www.microsoft.com/en-us/kinectforwindows/develop/overview.aspx. MOES, V. (2012). Kinect: Como funciona?. Accessed in February, 1, 2012. Avaliable at: http://www.minasmorgul.com.br/blog/kinect-como-funciona. MORAN, Thomas, P. et al. (1983). The Psychology of Human-Computer Interaction. Lawrence Erlbaum Associates. NUVIXA, StagePresence. (2012). Accessed in April, 5, 2012. Avaliable at: https://www.vix.tv/. OPENNI. (2012). Programmer Guide. Accessed in April, 10, 2012. Avaliable at: http://openni.org/Documentation/ProgrammerGuide.html. ROBINSON, Simon. et al. (2004). Professional C# (3rd ed.). Wiley Publishing.

9