Features

The main focus of the DirActNI framework is to realize and evaluate contactless 3D interaction techniques. To do this as flexible and adaptable as possible it uses a lightweight installation and is based on only free and standardized software components. The following two sections will illustrate the main features by introducing the installation and the realized interaction techniques.

Installation

As mentioned before, the installation was chosen to be as lightweight as possible. It requires no instrumentalization on the user and no complicated calibration or time-consuming setup. On the one hand it does not use any sensors on the user's body (1) because it captures his motions contactless with a depth camera, like the Microsoft Kinect or the ASUS Xtion (2). On the other hand it uses conventional hardware like a low performance computer (3) to process the motions and visualize the virtual environment and also a conventional projector (4) and screen (5) to display the environment. Thus the installation provides a very low instrumentalization of the user, the possibility of an aseptic interaction and is although very lightweight and inexpensive.

On the software side the DirActNI framework uses the free OpenNI framework for the motion processing to achieve a flexibility in terms of different depth sensors or operating systems. On the other hand the DirActNI framework uses with Unity 3D a powerful authoring tool for the creation of platform independent interactive 3D environments.

Furthermore the framework uses an intelligent algorithm to detect the opening state of the hands to allow an even more natural interaction.

Interaction Techniques

The DirActNI framework provides the possibility to handle several individual interaction techniques. Therefore the 3D interaction was divided in typical interaction tasks like locomotion and orientation. This provides the possibility to implement several techniques for the same task separately and interchange them during the execution on the fly to test their pro's and con's. Up to now with the Pointer-based and Whole-body technique two main interaction techniques was realized. Both will be introduced in the following.

Pointer-based interaction technique

The pointer-based technique is based on the same indirect interaction metaphor as a gamepad or a combination of a keyboard with a mouse use for the interaction in a 3D environment. The difference is that not a joystick on the gamepad, the keys on the keyboard or a mouse control the locomotion and orientations but the user's hands do. The intention behind this analogy is, to make this technique intuitive for most experienced users. So the user controls the orientation with one hand and the locomotion with the other hand by moving them in the desired direction. The speed of the locomotion is controlled relatively to the hand’s position whereas the orientation is controlled in a certain interval depending on the absolute position of the user’s hand. Outside this interval the rotation speed is controlled relatively. This combination allows the user a direct control of the orientation in front of him and allows him to rotate completely in the virtual environment without losing a non-surrounding screen out of his field of view. The manipulation also behaves like the manipulation with a gamepad. For the triggering event the opening state of the orientation hand is used, which is following a natural grab gesture. In addition to this triggering event a time-based trigger was also implemented to evaluate the difference for the user experience.

Whole-body interaction

The whole-body interaction technique offers the user the possibility of a realistic natural and (more) intuitive interaction, which is based on natural movements to be more immersive and necessary a lower cognitive load. It provides the possibility of a physically movement in the virtual environment by using the head tracking technique. Unfortunately these techniques only allow a locally limited locomotion due to the tracking and environmental circumstances. Therefore, an additional lean-based navigation was designed by which the user can move freely through the virtual environment while leaning the torso in the desired direction. The speed of the locomotion is controlled by the leaning angle with predefined thresholds. This kind of locomotion is similar to the usage of a Segway. The orientation is also based on already known natural gestures. Thus the horizontal orientation is controlled by the natural rotation of the shoulders and the vertical rotation by looking up and down gesture through moving the head. Both are controlled absolute to the particular rotation angles. In addition, the horizontal orientation also behaves like the pointer-based technique beyond a certain interval relatively to the rotation angle. Thereby, the user can physically move in a limited area and furthermore use his body for an unlimited locomotion in the whole virtual environment.

Direct manipulation

As the user does not need his hands for the Navigation, he can use them for a natural direct manipulation similar to the real and everyday grabbing and holding gestures. Therefore, the movement of his hands is translated isomorphically to his virtual hands in the virtual 3D environment. By touching the desired object with an open hand and closing the hand at the same position he can select the virtually touched object. While keeping the hand closed, the virtual object is being hold by the virtual hand and follows through the virtual environment until he opens the hand again. Similar to the one hand interaction a two-handed interaction was especially implemented for carrying bigger virtual objects.