RICERCA ITALIANA     

Adaptive, Context-aware, Multimedia Guides on Mobile Devices

Università degli Studi di Udine

Abstract

The project aims at deepening those theoretical and technical aspects allowing one to design a full-scale mobile guide in different application domains. The project will propose an integrated framework for a context-aware mobile guide, characterized by: (i) a standard and shared format for the representation of data allowing interoperability between different devices and applications, (ii) ability to derive information on the state and behaviour of the user from common sensors data (e.g., GPS data), (iii) automatic generation of natural language presentations, (iv) device-dependent generation and adaptation of the user interface, (v) usability-centered input/output modalities based on visual interfaces and multimedia and multimodal techniques, (vi) multi-user features (mobile communication, community). Three applications for three different domains (tourism, culture and sport) will be developed to prove the effectiveness and generality of the proposed solution. <<<

Principal Investigator

Luca CHITTARO Università degli Studi di UDINE

Research Objectives

Recent advances in mobile devices (PDA, SmartPhones, mobile phones), wireless communication technologies (GPRS, UMTS, Wi-Fi) and localization technologies (GPS, EGNOS) are making it possible to develop complex applications, named mobile guides, that support users in their work and free time activities, any time and in any place, replacing the need for human guides and for the time-consuming and often unsuccessful examination of paper-based information.
The main goal of this research project is to deepen the theoretical and technical aspects allowing one to design a full-scale mobile guide in different application domains. The project will propose an integrated framework for a context-aware mobile guide, that will be used to develop three applications in three different domains (tourism, culture and sport).
The main requirements for the guide will be:
• The ability to provide users with services in an easy and effective way.
• The ability to infer contextual information exploiting common sensors data (e.g., GPS).
• The ability to automatically adapt contents, presentation modalities and interaction with the system, according to the specific user (e.g., her interests, goals), the current context (e.g., position, luminosity, noise), the current device (e.g., PDA, SmartPhone).
• The ability to provide the user with automatically generated content and natural language presentations.
• The ability to provide the user with communication and collaboration capabilities.

Building a really effective mobile guide, indeed, requires a joint effort, in order to put together different skills, both at design level and at development level, in order to guarantee that all the involved aspects are properly handled.

Due to the great number of possible mobile guide applications and mobile devices, a goal of the project is the representation of different kinds of knowledge in an interoperable format, by exploiting a shared semantic representation of the entire system knowledge. Since the mobile guide is characterized by adaptive features, the system knowledge base will include knowledge about the content and services offered by the guide, knowledge about the users the guide has to adapt to, knowledge about the devices used by users, about the environment and social contextual conditions.

A second goal of the project is the design of appropriate solutions that allow the user to access the services provided by the mobile guide in an intuitive, simple and effective way. This goal may be subdivided in different sub-goals.
The first sub-goal concerns context-awareness. A system is context-aware if it employs the context (e.g., the user's position) to provide the user with the most relevant information and/or services for a given situation, thus increasing the flexibility and the ease with which content can be used. During the project, basic information constituting the context will be identified and appropriate solutions will be developed to measure this information and to exploit it in different (and innovative) ways in the guide.
The second sub-goal concerns the design of an appropriate user interface and the interaction modalities it provides, which are the aspects of a system that more affect its perception by users. To this end, it is necessary to face different problems such as the limitations of mobile devices when compared to desktop computers (e.g., limited screen size and poor resolution, limited processing capabilities, etc.), the necessity to quickly satisfy the user's information needs (e.g. to find a hotel, a parking, etc…), the influence of use and environmental factors (e.g., use in open environments, standing, presence of noise, etc.). Therefore, during this project appropriate solutions to this problems will be studied and implemented, on one side by using Information Visualization and input simplification techniques, on the other side personalizing the provided information and adapting the interface in terms of layout (thus allowing also portability on devices with different features).
A third sub-goal concerns the automatic generation and exploitation of natural language presentations, which are nowadays possible also for devices with limited capabilities, thanks to the evolution of speech synthesis and interaction technologies. During the project these new technologies will be experimented, exploiting them to provide mobile guides with speech capabilities in the form of comments dynamically adapted to the context and the user. The guide will not simply present predefined audio messages; the system instead will be able to generate texts and dynamically synthesize speech in real time depending on the circumstances.

An additional goal of the project is to exploit the interaction capabilities of mobile devices in order to stimulate collaborative group activities in the exploration of environments such as museums or cities. Communication and social interaction among users are in fact key elements for making the experience enjoyable and involving, and to facilitate learning. The interaction and communication among users will be promoted by engaging them in educational games, to be played collaboratively by means of mobile devices, and using a shared communication infrastructure.

The flexibility and reusability of the framework that will be developed will be tested in three different application domains.
The first application allows one to access museum information and, more generally, cultural information about a city. Solutions which are able to foster and stimulate the fruition of the cultural patrimony may have a significant impact in the cultural as well as economic growth, as they could constitute an additional stimulus to tourism. In particular, the proposal of cooperative explorations aims at increasing the involvement of people in cultural visits, by turning them into socializing occasions rather than individual experiences.
The second application concerns the support to outdoors training activities (e.g., jogging, biking, etc.). In particular, the application that will be developed aims at improving the training experience by providing the user with features (such as personalized information supply during the activity, navigation support, etc.) that allow her to perform the training more effectively, improving her satisfaction and health.
The third application is a city tourist mobile guide, which will offer a set of services that can satisfy typical tourist needs, such as information about accommodations, restaurants, sightseeings, etc., as well as daily needs, such as traffic information, or even special needs, like information about events (e.g., conferences, exhibitions, etc.). <<<

Timescale

24 months

National and international background

The first mobile guide prototypes have been developed during the '90 in the Cyberguide (Abowd et al., 1996) and GUIDE (Cheverst et al., 2000) projects. In recent years, there has been an increase in the interest towards these applications, thanks to the continuous progress of mobile devices and technologies.
The research literature refers to a broad spectrum of different application areas for mobile guides. Mobile city guides provide the user with maps and other relevant information such as the location of restaurants, public offices, tourist sites, etc. adapted to the user's location (e.g. Iacucci et al. 2003, Schmidt-Belz and Poslad 2003). Mobile guides for specific tourist-sites provide users with the opportunity to follow guided tours or access additional information about the site being visited (e.g. Bornträger et al. 2003), while mobile museum guides provide additional information about the specific items on display (e.g. Rocchi et al. 2003). Other mobile guide systems support the user's social or personal life, such as mobile personal guides and event planners keeping track of friends and events (e.g. Fithian et al. 2003, Paulos and Goodman 2004). Assisting people commuting and travelling, mobile navigation guides provide route planning information and directions for wayfinding (e.g. Chincholle et al. 2002, Holland et al. 2001), while mobile travel planners keep track of users' itineraries, upcoming meetings etc. (e.g. Ricci et al. 2002).

What really sets apart a mobile guide from more traditional solutions (such as paper guides and desktop guides) is context-awareness. The most important contextual information that is usually employed by mobile guides is location. Indeed, by knowing the position of the device, it is possible to easily provide users with location-aware services, thus increasing the usefulness of the guide. The current position of the user may be obtained by means of different techniques. Very common is the use of satellite-based tracking approaches, such as GPS, while, for indoor applications, it is necessary to use other technologies such RFID, Bluetooth, IrDA or ultrasounds (Vossiek et al. 2003). Exploiting contextual information, such as location, time, other sensor data, it is possible for a mobile guide to integrate different adaptation techniques, increasing its flexibility. Current mobile guides offer different levels of adaptation and exploitation of contextual information. Among the most relevant systems, Crumpet (Poslad et al., 2001) personalises services according to the user's position, interests and previous interactions, both on smarthphones and PDAs. Crumpet, furthermore, adapts the presentation to the modified technological context and uses different technologies to localize the user. GUIDE (Cheverest et al., 2000) is a guide that is adaptive according to user's position, walking speed, visited places, the time of the day and the user's interests and language. Lol@ (Pospischil et al., 2003) is a tourist guide that provides adaptation only according to the device.
With respect to the interface and the modality with which information is presented to users, most context-aware mobile guides use 2D maps of the area where the user is located, pinpointing her position and usually providing visual information on the nearest points of interest and on the paths to reach specific destinations (e.g., Pospischil et al. 2001). Maps are powerful tools for navigation because of the richness of information they can supply and the rate at which people can absorb this information. At the same time, maps require users to repeatedly switch from their egocentric perspective of the world to the exocentric perspective provided by the map and vice versa. This often requires significant mental effort and affects performance (Aretz and Wickens 1992). Recently, to solve this problem, some attempts have been made at exploring 3D graphics for mobile guides. Rakkolainen and Vainio (2001) have proposed a system that combines a 2D map of an area with a 3D representation of what users are currently seeing in the physical world, studying the effects of 3D graphics on navigation and wayfinding in a urban environment. Other experiments with 3D graphics have been made by Kulju and Kaasinen (2002) e Laakso et al. (2003). In general, these experiments have aided researchers in understanding how to use 3D graphics in mobile guides, but further studies must be carried out to assess the real effectiveness of this solution. More generally, regarding the automatic generation of user interfaces that are adaptive to different devices, many works can contribute to define the methodologies and techniques for handling this kind of issues; for example, we can mention (Conino et al., 2003; Dees et al., 2003; Mori et al., 2003; Pashtan et al., 2003).
With respect to the dynamic generation of adaptive natural language comments, the different approaches in the literature vary significantly in complexity: the simplest systems rely on predefined and suitably annotated text fragments, which are composed in various ways; in template-based systems variable elements are inserted in predefined schema; more complex systems provide for planning of sentences and discourse and usually require a more structured representation of contents in a formal language.
Porzel e Strube (2002) discuss the wider issue of context-adaptive natural language interaction systems, and offer interesting suggestions for a general architecture.
Concerning mobile guides, adaptive comments generation has been explored in some research prototypes, mainly in the context of museum and tourism applications.
The Hyperaudio system (Sarini and Strapparava, 1998) provides audio comments on a palmtop computer. It takes into account the physical location of the user and the time the user spends at a location as a measure of interest in the pieces of work that are situated close by. Höök (1997) addresses the problem of finding information that is interesting to the user on the basis of a predefined network of semantically related topics. ILEX (Cox, 1999) is a system that dynamically generates text labels for exhibits in a museum jewellery gallery, by concatenating static and dynamic text.
Marucci e Paternò (2002) present a system which provides navigation assistance in a Web based virtual museum. The system, in addition to adapting to the user profile and navigation history, introduces different information categories: introductory information, when a new topic or aspect of interest (e.g. a new artist or a new section) is accessed by the user; summary information, to provide an overall perspective on the items that have been accessed in the current session; comparison and difference information, in order to talk of objects in relation to other previously encountered objects; curiosity information, additional peculiar information which can increase the visitor's involvement, highlighting those features of the work that can raise the user's interest. Text fragments are selected by a rule-based mechanism and adapted to form a coherent discourse.
Other works in the field of natural language comments generation have considered rhetorical structures (Not e Zancanaro, 2000). Other authors (Milosavljevic, 1997) have considered a richer set of comparisons (illustrative, explanatory, direct).

With respect to the development of a semantic knowledge base, the background area is certainly the so-called Semantic Web, theorized by Berners-Lee (Berners-Lee, 2001). The project of semantic web requires a common model for representing information that must be shareable, expressive, semantic and machine-accessible.
In particular, for the development of ontologies for semantic knowledge representation, some of the main works are those of Guarino (1998), Uschold and Grüninger (1996) and Gruber (1993).
About adaptive hypermedia, there is an increasing interest in problems of Semantic Web and Semantic Web services. In the past, most of adaptive systems adapted a limited corpus of known resources. However, the use of shared ontologies and of standard semantic web markup languages (e.g. RDF, RDFS, DAML, OWL) allow the application of adaptation techniques beyond the local context.
Starting from the possibility to apply adaptation techniques in a global context - e.g. information retrieval in a e-learning environment - AHA! (Aroyo et al., 2003)-, a lot of systems that gather both the advantages of adaptive systems and semantic web open corpus has been developed. The main projects are about e-learning (Dolog et al., 2004; Brusilovsky et al., 2004; Conlan and Wade, 2004; Power et al., 2004), communities of practice (Garlatti and Iksal, 2004), mobile guides (Van Setten et al., 2004) and knowledge management (Razmerita et al., 2003).
In the field of web information systems, Frasincar e Houben (2003) have developed a methodology (HERA) to design these kind of systems. In their adaptive framework they distinguish a concept model, expressed in RDF(S), and an application model regarding navigation and presentation aspects, in which the base elements are parts of the ontology expressed in the conceptual model.
Heckmann and Kruger (2003) developed UserML, an XML-based language to represent the user model in a standard way, shared by different applications and devices in an ubiquitous computing context. UserML can be used to represent metadata, user model, inferences explanations, context and environment models.
In the user modeling server area, the group led by Judy Kay (2002) developed a user modeling server (Personis) which allows the user to examine her/his user model as a "view" on the application s(he) is using.

With respect to multi-user functionalities, it must be observed that visits to museums or cities are usually supported by mobile guides as individual experiences; guides are not designed to promote social interaction among visitors. On the other hand, the cultural visit experience, as Falk and Dierking (2000) claim, is influenced by the social context, which includes the interactions among visitors. Several studies underline the fact that the interaction with the exhibition, as well as communications and interactions among visitors are key points for a successful learning environment (Hindmarsh, 2002; Leinhardt, 2002).
Multi-user support for collaborative and group visits has been explored in some international research projects. The SottoVoce project (Grinter, 2002), developed at Xerox PARC, is a mobile companion, based on the iPaq technology, that provides audio content of artwork descriptions and acts as an audio media space between visitors, offering a mean for awareness and sociability.
As part of the Equator project (Brown, 2003), a system has been developed at the lighthouse in Glasgow, a museum dedicated to the work of the designer Mackintosh. The system considers three ways for visiting the museum: (i) a real visit using a PDA; (ii) a virtual reality visit in a 3D world; (iii) a Web visit. With this system, visitors are able to share their museum experience and navigate jointly through mixed realities: the Web, the virtual and the physical reality. Information is provided about each visitor location and orientation. In addition, they may communicate through audio channels. The authors have observed that voice interaction, location and orientation awareness, and mutual visibility are essential to the success of museum co-visiting between remote users.
In the Equator project, as well as in the Sotto Voce project, researchers have noted that information about location and orientation of companions is essential in cooperative visits in order to maintain group awareness.
The Ghost Ship project (Hindmarsh, 2002), compared to the previous projects, is more oriented to an artistic experience of the museum co-visiting. The goal of this work is to analyse and consider informal and social interactions between visitors through video interaction recordings.
VeGame (Bellotti, 2003) is another project making use of mobile technology in the exploration of the city of Venice and to learn its history through games based on observation, reflection and action (e.g., videogames).
The History Hunt project (Fraser, 2003) has been performed at the Nottingham castle. The authors have considered two types of activities: searching for objects in the castle or looking for traces of the past on the basis of written clues on pieces of paper with RFID tags (Radio Frequency IDentification) distributed to visitors at the entrance. When visitors approach an interactive display, they receive additional information on the object or clues described in the RFID tag. <<<