RICERCA ITALIANA     

Research program

Computational and neural mechanisms of sensorimotor learning and control

University Co-ordinator

Università degli Studi di BOLOGNA - ELETTRONICA, INFORMATICA E SISTEMISTICA - BOLOGNA(BO)

Research Unit Leader

Lorenzo CHIARI

Description

To date, rehabilitation applications of VR have primarily used visual sensory input while the addition of audio and haptics is less developed. Wright et al (2003) demonstrated, in an fMRI-based study, that multi-sensory audio-video stimulation has an important action on the reinforcement of cortical excitability compared to a mono-sensory stimulus. Haptic interface devices including gloves, pens, joysticks and robotic arms provide users with a sense of touch and allow the user to feel a variety of textures or to be exposed to external force-fields. There is increasing evidence that haptic information is an effective addition towards the accomplishment of certain treatment objectives such as increasing joint range of motion and force (Jack et al, 2001). Haptic information has also been identified as a significant signal for improving a subject's performance in more difficult tasks. For example, Shing et al (2003) report a specific benefit of adding auditory and haptic information to an upper extremity movement when the difficulty of the task, in this case a 3D pick and place task, was high. It is reasonable to expect that integration of visual and haptic interfaces with motion tracking could allow the user to become fully immersed in 3D virtual environments, including 3D sound, and facilitate his interaction with virtual objects for more and more effective motor learning accomplishments.

Moving from this evidence, the goals of this research unit (UR_UniBO) will be: (1) the completion and installation of the virtual visual environment (VVE), a VR platform with immersive audio-visual displays as shown in fig. 2 (WP2); (2) the selection and implementation of upper extremity motor tasks to be proposed to the user in the VVE (WP2); (3) the experimental investigation of the construction of muscle synergies during motor tasks executed in the VVE (WP4); (4) the integration of the VVE interface with the haptic interface (VHE) developed in WP1 (WP6); (5) the selection and implementation of motor tasks to be proposed to the user in the integrated virtual environment (WP6); (6) the validation of computational models of multisensory integration for motor learning on the experimental data acquired during WP1, WP2 and WP6 (WP7).

Fig. 2 – Expected Set-up of the VVE

The activities of UR_UniBO are listed in the following. They are subdivided into the two phases of the project and they are articulated across the 4 work-packages (WP) of the project, which we will address. In the first phase of the project UR_UniBO will contribute to WP2 (as a coordinator) and WP4, whereas in the second phase UR_UniBO will contribute to WP6 (as a coordinator) and WP7. For each WP we report the percentual resource allocation and expected deliverables.

Phase 1

WP2: Virtual Visual Environments (VVE)
The first task of this activity is to complete the hardware/software set-up of a VR platform with immersive audio-visual displays and haptic feedback. The available hardware resources include:
(1) graphics displays - projector-based displays (NEC VT660K, TFT active-matrix with Micro Lens Array, resolution 1024x768 pixels) and high-res computer monitors;
(2) sound displays – Bluetooth stereo headphones (Hp), loudspeaker-based 3D sound system (Harman/Kardon);
(3) 3D kinematic trackers - optoelectronic stereophotogrammetric system equipped with 6 CCD cameras (SMART, eMotion), portable inertial measurement unit (MT9, XSENS), accelerometers.
The workstation will be enriched by adding
(1) a personal graphics display (Head Mounted Display - HMD),
(2) an haptic feedback section including a tactile mouse (like the iFeel Mouse, Logitech co., including 1 vibrotactile actuator, maximum force 1.18 N at 30 Hz), a force feedback joystick (like the WingMan Force 3D joystick, Logitech co., including 2 DC electrical actuators, maximum force exerted 3.3 N), and an haptic glove (self assembled);
(3) a high-performance computer including a graphics accelerator card for allowing real-time rendering of VR scenes.
The software environment will be compatible with the one developed in WP1 by relying on the same Matlab modules (Simulink, Virtual Reality Toolbox etc). The haptic rendering experimented in this WP is complementary to the haptic rendering experimented in WP1 (given by a haptic manipulandum). It will be given as a vibrotactile feedback (mouse, glove) or as a force feedback (joystick) and it will allow 2D or 3D movements of the arm with a different involvement of the joints (wrist, elbow, shoulder). In WP6 we plan to carry out a comparative analysis of the two platform in such a way to outline a future integrated scenario.

In general, this experimental platform is conceived as a computational bloc that manipulates the information on the state of the subject (described in kinematic, dynamic, and electromyographic terms by suitable sensors and motion-capture devices) and generates in real-time visual, acoustic, and tactile stimuli driven by a model of the task and/or interaction scenario (see fig.2).
The VVE software will allow to configure the system according to different scenarios, that corresponds to the main activities carried out in this WP:

VR (Virtual Reality) Scenario - Based on the information provided by the measurement module (EMG, prehension force, error signal of the end-effector position, inter-segmental kinematics), the limb will be recreated in the artificial world in which the user could move without being totally immersed. The subject will join a double interaction with the VVE platform: the first through the haptic interface, the second through the images and the sounds, which will be provided using stereoscopic glasses. The virtual environments that will be displayed will be either facilitatory to the exercise the subject is asked to perform (in other words the sensation coming from the proprioception and from the VVE will be consistent and synergic) or conflictual (when the VVE will simulate prism-like shifts or more complicated distortions of the visuomotor transformation, static or dynamic). This latter class of experiments (1) will make it possible to investigate relevant motor learning paradigms (like the divide-and-conquer proposed by Ghahramani and Wolpert, 1997), and (2) will enable us to evaluate the multi-sensory integration strategies performed by the subjects for limb control and the possible dominance of one sensory channel over the other channels (Chiari et al, 2000; Ravaioli et al, 2004).

AR (Augmented Reality) Scenario – The information provided by the measurement module can be used for creating virtual objects which are superimposed to the real ones in the field of view of the user. For example it will be possible to evaluate the effect on the motor task of offering a digitized image of the subject's limb superimposed to a graphic representation of the instant distance of the target or of the force field acting on the end-effector. Or even it will be possible to implement the learning by imitation principle, eg, by displaying at the same time the actual and the desired trajectories of the end-effector.
In general, each one of the signals concerning the movement acquired from the user using the measurements module can be encoded and translated, in real-time, in an audio, video or haptic signal. For example the amplitude and/or frequency modulation of the sound provided by earphones or its 3D representation, or the instantaneous visualization on the screen of the measured signal, or the vibro-tactile skin stimulation, can all adequately represent the feedback variable. In this way the restitution module translates the signals coming from the upper-limb in immediately comprehensible signals for the user who can learn, for example, to associate new sensation to the real muscle activation level, in order to develop later on new muscle synergies in which the muscles are contracted only as much and as long as strictly necessary. The paradigm that is being realized is the sensory augmentation one.

UR_UniBO has maturated a large experience in visual and auditory VR/AR system implementation (Chiari et al, 2005; Ravaioli et al, 2005) and in the successful application of auditory feedback for the balance stabilization during stance in bilateral vestibular loss subject (Horak et al, 2003-2004; Dozza et al, 2005). In this context the AR will enable, complementary to a VR approach, to test separately the role played by the different sensory channels in the sensori-motor transformations (Jeka et al, 2000-2003; Ravaioli et al, 2005; Carver et al, 2005). Also, it will be able to bring, in an easy and direct way, to the identification, among the kinematic and dynamic quantities acquired by the measurement module, of the most efficient ones as control variables in a neurorehabilitation setting.

Depth information Scenario - This activity is intended to explore the role of the depth information determined by binocular view in the process of learning visuo-motor transformations. This is background knowledge valuable for the optimal configuration of the VVE platform. It will take into account the duality of retino-centric and viewer-centered representations and it will investigate how people learn a common reference system where different sources of information can be integrated: visuo-spatial information, proprioceptive information (position and movements of the arm), and gaze-position signals. The approach is to use alterations of the binocular disparity information determined by 3D affine class deformations, which modify the values of absolute and relative disparity, in a static or dynamic manner.

Percentual resource allocation: 50%
Deliverables:
D2.1: first release of the VVE, that includes software and a report (month 3; UR_UniBO)
D2.2: report on the results of the VVE in the VR configuration (month 12; UR_UniBO)
D2.3: report on the results of the VVE in the AR configuration (month 12; UR_UniBO)

WP4: Electrical Muscle Activity (EMG)

This activity is coordinated by UR_UniRoma3. The study of muscular synergies construction can provide insights on motor learning mechanisms. The analysis of motor learning mechanisms needs experimental protocols aiming at destructuring existing internal models by altering the environment. The alteration of the environment can be obtained by dynamical force fields (WP1) and by dynamical visual distortions (WP2). Muscular synergies built during motor tasks execution in altered VVE environment will be investigated by means of pattern recognition techniques (eg, fuzzy clustering, factor analysis, discriminant analysis) on the basis of electrical indices extracted by surface EMG signals (sEMG).

Percentual resource allocation: 10%
Deliverables:
D4.3: Report on experiments on the construction of muscle synergies in the VVE (month 12; UR_UniRoma3, UR_Polito, UR_UniBO)

Phase 2

WP6: Integration of the two virtual platforms
In this activity we shall take advantage of what was accomplished in the first phase of the project. First of all we shall repeat some of the learning experiments performed in WP1 and WP2 by correlating the modifications of behavioral parameters (precision, speed, etc) with neural parameters, both at the central level (modification of the cortical sensitivity by means of the TMS: WP3) and at the peripheral level (regularization of the muscle activation patterns: WP4). We shall also investigate the influence of muscle fatigue on the learning process by performing similar learning experiments involving haptic objects with increasing levels of muscle effort and monitoring the EMG spectral modifications induced by fatigue. WP5 will allow us to investigate if electrical stimulation of muscles, driven by a dynamic model of the task, can facilitate the acquisition of the internal model and thus speed up the learning process. The result of this activity will be relevant for the pilot study with patients in WP8. Moreover, we shall perform some experiments in which the interactive paradigms developed in WP1 and WP2 are combined in different ways. The purpose is to determine to which extent vision and haptic perception influence each other during learning.
In this WP we shall take advantage of what was accomplished in the first phase of the project and explore useful ways of combining the VVE and the VHE. The literature on this topic is quite limited (Shing et al, 2003) and so it is difficult to envision a precise scenario. In general, the goal is to gain some understanding on how visuomotor learning and force field adaptation can facilitate each other or enter in conflict. Given the complexity of the experimental setup and the limited time span of the project only a preliminary pilot study will be carried out but we think that its novelty could open the door to an important new road in the study of motor learning. The activities in this WP will be split into two steps, an implementation step and an experimental step that includes several pilot experiments:
1) integration in the VHE of the systems components of the VVE (either in the VR or AR configuration) that proved to be more promising in the first phase of the project.
2) performance of force-field adaptation experiments in which the visual feedback is modified on-line in order to magnify the ‘visual error': is this manipulation of the visual afference capable to speed-up the learning process?
3) performance of visuomotor adaptation experiments (eg, virtual prism adaptation) in which the ‘motor error' is gently opposed by a facilitating force field (haptic facilitation). Again, is the motor facilitation capable to speed-up the learning process?
4) evaluation of some neural correlates of the combined haptic, visual adaptation; this activity will focus on the most promising experimental protocol (determined in the two previous activities) and the most promising procedures of estimation of neural correlates evaluated in WP3 or WP4.

Percentual resource allocation: 30%
Deliverables:
D6.1: Integration in the VHE of selected systems components of the VVE: it includes software and a report (month 18; UR_UniBO, UR_UniGE1)
D6.2: Report on experiments of visual error amplification during field adaptation (month 24; UR_UniGE1, UR_UniBO)
D6.3: Report on experiments of haptic facilitation during visuomotor adaptation (month 24; UR_UniBO UR_UniGE1)
D6.4: Report on the estimation of some neural correlates during the combined adaptation experiments (month 24; all URs)

WP7: Computational models
This activity is coordinated by UR_UNIRoma3. We shall analyze the experimental data acquired with the two virtual platforms and we shall investigate the suitability of existing models of multisensory integration for explaining the results obtained. Specifically, we shall verify the validity of stochastic models of multisensory fusion, inspired by the works by Jeka et al, 2000 and Carver et al, 2005.

Percentual resource allocation: 10%
Deliverables:
D7.5: Report on multisensory integration modeling (month 23; UR_UniBO)