RICERCA ITALIANA     

Research program

Computational and neural mechanisms of sensorimotor learning and control

University Co-ordinator

Università degli Studi di GENOVA - INFORMATICA, SISTEMISTICA E TELEMATICA - GENOVA(GE)

Research Unit Leader

Pietro Giovanni MORASSO

Description

WP1: Virtual Haptic Environments (VHE)
In this activity, for which this research unit is leader, we intend to develop a Virtual Haptic Environment by using a new robotized haptic interface (Casadio et al 2005), shortly named Braccio di Ferro. This device, shown in the figure below, has been developed from our previous experience in two complementary fields: (i) measurement of the hand mechanical impedance (Tsuji et al 1995), and (ii) quantitative assessment of cerebellar ataxia (Sanguineti et al 2003).
It has been designed by having in mind the range of forces and the frequency bandwidth that characterize the physical interaction between two human beings, for example a patient and a physical therapist, or between a human being and manipulated objects in everyday life and a number of requirements that we think are essential for allowing a natural haptic interaction, such as back-drivability, very low friction and inertia, and mechanical robustness.

In the design of haptic devices to be used in the study of motor learning there are three basic requirements: 1) large workspace, 2) high level of the force at the end effector, 3) high degree of manipulability throughout the workspace. The first two requirements are conflicting: the workspace increases with the size of the manipulandum, whereas the Force to Torque (F/T) ratio decreases. A lower F/T ratio implies the need for more powerful motors in order to guarantee a specified level of hand force. Moreover, a good degree of manipulability enhances the backdriveability of the manipulandum, thus simplifying its control. All together, the following design specifications were used for Braccio di Ferro:
- 800 x 400 mm elliptically shaped workspace;
- Possibility of rotating the workspace plane around a horizontal axis in order to allow experimental protocols in the sagittal plane;
- Nominal spatial resolution: better than 0.1 mm;
- Dry friction forces: less than 0.1 N;
- Continuously deliverable force at the handle (in all directions and over the whole workspace): greater than 50 N;
- Maximum deliverable force (in all directions and over the whole workspace): about 200 N;
- Variation of the manipulability index (Yoshikawa 1985): less than 10%;
- Anisotropy index: less than 2;
- Force/Torque ratio (in all directions and over the whole workspace): better than 2 N/Nm.
In addition to this, we required that the bi-directional interface is updated at a frequency of at least 1000 Hz because below that level it is known that the user may have misperceptions of the virtual objects. The requirements above were satisfied by a four-bar linkage with a suitably bent forearm, whose dimensions were determined by optimising the Global Isotropy Index (Stocco et al 1998).
The mechanical linkage is operated by a pair of AC brushless servo drive systems (Ultract II, Phase Motor Control, Genova), based on rare earth permanent magnets, which provide high levels of dynamic performance and torque density, even at very low speed (the continuous torque is 35 Nw and the peak torque is 98 Nm). With these motors and the corresponding mechanical linkage it is possible to emulate the range of forces at the end-effector that characterize human movements. Each motor is equipped with a digital encoder that provides a resolution of 17 bits/revolution. The control architecture includes two digital controllers, that update the commanded currents and the encoder readings at a rate of 4 kHz, and two PCs that run the real-time control software.

Of the two PCs one operates as a Configuration and Command workstation, that runs RT-LAB, and the other as a Real-Time workstation that operates as RT-LAB real-time target, running QNX. The two PCs interact over a TCP/IP link. With this architecture it is possible to implement an impedance control scheme, which is the heart of the haptic interface. This combination of features is not present in commercially available haptic interfaces.

On top of this haptic interface we intend to build an articulated Virtual Environment that allows to implement Virtual Interactive Models (VIMs). A VIM is a complex reactive system, in which we may distinguish two basic components:
- a haptic component;
- a visual (or, more generally, audio-visual) component.
The former one consists of a functional relationship between the contact force and the motion during the interaction of the user with a virtual object and is implemented by means of a suitable impedance control block on the target machine. The latter one is a real-time display of the interaction between the user and the virtual object, which may include:
- motion/deformation of the virtual object;
- motion of the subject's haptic interaction point (HIP);
- visualization of the interaction force.
Haptic objects will be represented as impedances and implemented as Simulinl blocks. Visual objects will be represented by using the Matlab Virtual Reality Toolbox, which is based on the Virtual Reality Modeling Language (VRML). The interaction among the haptic and the visual part of a VIM will be described by a finite state machine, implemented by means of another Matlab tool, namely Stateflow, which is based on the theory of finite state machines. The figure below summarizes the computational organization of the Virtual Environment and the corresponding haptic rendering scheme.

In the figure J is the Jacobian matrix of the robot, T is the commanded torque vector, q is the robot joint rotation vector, and x is the end-effector position vector; the filled boxes correspond to transformation that occur in the real world whereas the empty boxes are actual computations that occur in the virtual world and are carried out by the haptic architecture.
The main function of the system is to implement dynamical force fields or to emulate different haptic tasks.
As regards the former function, which is the one more heavily investigated in the last decade, the underlying task is typically a reaching task and the force field is an organized dynamical disturbance that the subject is required to learn by building an internal model that allows to compensate the disturbance and thus recover the normal coordination pattern. The focus, in this case, is on the modalities that characterize the acquisition of the internal model. In particular, we shall implement the classical velocity-dependent curl field to be used as a sort of benchmark for integrating in the virtual environment the novel information provided by WP3, WP4, WP5.
Another function of the haptic interface, emulating a variety of haptic interactions in the real world, will be used in order to improve our understanding of the fundamental mechanisms of motor control during manipulation, which typically must find a compromise between trajectory formation/position control and force control. In particular, we shall compare the motor patterns that characterize the well studied reaching tasks, with stationary or moving visual targets, with tasks that involve haptic targets (e.g. reaching the target under visual guidance and then keeping the contact at a prescribed level of contact force while the target moves on a predictable path). If the task is difficult it involves a process of sensorimotor learning or adaptation for which the virtual environment described above is an ideal experimental tool. We shall also study hitting tasks, which requires to reach the target with a precise direction of impact and impact velocity, as in a billiard game. Hitting is more demanding than reaching and it is the logic step further in the study of adaptation to velocity-dependent force fields. Another interesting haptic task is virtual cutting, where cutting is implemented as a suitable frictional force and the subject is required either to "cut" thin slices without hurting adjacent slices or to cut along a visually specified shape. Also in this case the basic problem is to understand the mixture of position control/force control as well as stiffness control.
In this WP we shall also address the issue of generalization by evaluating to which extent the knowledge acquired in a given situation can help solving analogous tasks in slightly or vastly different situations.
We also intend to integrate in the control system a real-time module, indicated in the previous figure, that compensates at least part of the dynamic disturbance to the subjects that is induced by the intrinsic dynamics of the robot.

Percentual resource allocation: 30%
Deliverables:
D1.1: First release of the VHE, that includes software and a report (month 3)
D1.2: Report on experiments on haptic psychophysics (month 12)
D1.3: Report on experiments on haptic motor control (month 12)
D1.4: Report on experiments on virtual hitting (month 15)
D1.5: Report on experiments on virtual cutting (month 15)



WP2: Virtual Visual Environments (VVE)
This activity is coordinated by UR_UNIBO and the goal of our contribution is to explore the role of the depth information determined by binocular view in the process of learning visuo-motor transformations, by taking into account the duality of retino-centric and viewer-centered representations. In particular we are interested in investigating how people learn a common reference system where different sources of information can be integrated: visuo-spatial information, proprioceptive information (position and movements of the arm), and gaze-position signals.
The activity will be articulated as follows:
- Alteration of the binocular disparity information by using 3D affine class deformations, which modify the values of absolute and relative disparity, in a static or dynamic manner;
- Integration of monocular and binocular cues (e.g., Knill 2005);
- Expansion/compression of the depth range;
- Replacement of the disparity planes with respect to the horopter (crossed vs uncrossed disparity);
- Alteration of the spatio-temporal variations of disparity (such as the motion-in-depth) that are either real or illusory, as in the Pulfrich effect .
The Pulfrich effect is an optical illusion according to which an object actually moving in the fronto-parallel plane appears to move out of this plane (either approaching the subject or going away according to the specific conditions) if an eye is dimmed for the purpose of increasing the visual latency and, as a consequence, the delay with which the retinal image reaches the brain.
The fixation point can be varied according to the task and/or during the task, thus measuring the effect of such variable. Moreover we intend, in perspective, to study the effects of voluntary binocular saccadic eye movements by integrating an eye movement tracker in the HMD.
The experimental setup for carrying out the investigation summarized above cosists of the following items:
- Pair of cameras in a convergent stereo configuration, calibrated on the fixation point of the subject:
- LCD stereoscopic visual display system (HMD Visette Pro, Cybermind);
- On-line deformation of the binocular disparity (e.g. warping of the images acquired by the two cameras) by means of a suitable software packege.

Percentual resource allocation: 25%
Deliverables:
D2.4: report on the results of the study of the role of the depth information (month 13). It will include the following items:
- experimental set-up for the real-time binocular acquisition of the workspace and for the perceptual rendering of the altered visual information
- a software module for static alteration of 3D visual feedback
- a software module for dynamic alteration of 3D visual feedback

WP5: Electrical muscle stimulation (FES)
In this WP we shall collaborate with UR_Polito for carrying out experiments of force field adaptation in which we employ the closed-loop FES system, developed by UR_Polito, in conjunction with the VHE developed in WP1.

Percentual resource allocation: 5%
Deliverables:
D5.2: Report on experiments with the VHE (month 15)

WP6: Integration of the two virtual platforms
In this activity we shall take advantage of what was accomplished in the first phase of the project. First of all we shall repeat some of the learning experiments performed in WP1 and WP2 by correlating the modifications of behavioral parameters (precision, speed etc) with neural parameters, both at the central level (modification of the cortical sensitivity by means of the TMS: WP3) and at the peripheral level (regularization of the muscle activation patterns: WP4).
We shall also investigate the influence of muscle fatigue on the learning process by performing similar learning experiments involving haptic objects with increasing levels of muscle effort and monitoring the EMG spectral modifications induced by fatigue.
WP5 will allow us to investigate if electrical stimulation of muscles, driven by a dynamic model of the task, can facilitate the acquisition of the internal model and thus speed up the learning process. The result of this activity will be relevant for the pilot study with patients in WP8.
Moreover, we shall perform some experiments in which the interactive paradigms developed in WP1 and WP2 are combined in different ways. The purpose is to determine to which extent vision and haptic perception influence each other during learning.

Percentual resource allocation: 20%
Deliverables:
D6.1: Integration in the VHE of selected systems components of the VVE (month 18, in collaboration with UR_UniBO)
D6.2: Report on experiments of visual error amplification during field adaptation (month 24, in collaboration with UR_UniBO)
D6.3: Report on experiments of haptic facilitation during visuomotor adaptation (month 24, in collaboration with UR_UniBO)
D6.4: Report on the estimation of some neural correlates during the combined adaptation experiments (month 24, in collaboration with all URs)


WP7: Computational models
We shall analyze the experimental data acquired with the two virtual platforms and we shall verify to which extent they can be explained by means of a class of models proposed by Todorov and Jordan (2002) and by Todorov (2004) that are based on stochastic optimal feedback control as a theory of motor coordination.
We shall analyze the weight of binocular information in visual-haptic and visual-proprioceptive integration. Specifically, we shall verify that the optimal cue integration strategy evidences differences in how the brain combines depth cues for motor control and perception (Knill, 2005).

Percentual resource allocation: 10%
Deliverables:
D7.3: Report on the internal modelling scheme (month 22)
D7.4: Report on the application of the stochastic feedback control model (month 22)



WP8: Neurorehabilitation
In this activity we intend to continue the study of cerebellar ataxia reported by Sanguineti et al (2003) by using the virtual environments developed in WP1 and the closed-loop FES developed in WP5.

Percentual resource allocation: 10%
Deliverables:
D8.3: Report on rehabilitation experiments focused on closed-loop FES (month 24, in cooperation with UR_Polito)