Agnes Lisowska - Projects

Home - Projects - Publications - Abbreviated CV

Interactive Multimodal Information Management (IM2) - Archivus (2003-2007)

The Interactive Multimodal Information Management (IM2) project is a Swiss National Science Foundation project in the field of Human-computer Interaction. The aim of the project is the advancement of research "particularly concerned with technologies coordinating natural input modes (such as speech, image, pen, touch, hand gestures, head and/or body movements, and even physiological sensors) with multimedia system outputs, such as speech, sounds, images, 3D graphics and animation."

One aspect of work in the project involves the development of specially equipped SmartRooms which are capable of recording everything that happens in a meeting (video, audio, slide capture and electronic whiteboard use). Meetings that take place in these rooms leave an extensive series of browsable and searchable artifacts in the form of audio and video files, electronic copies of documents used during the meeting, and a text transcript of what was said. This multimedia data can also be annotated (topics, dialogue acts, arguments, etc.) and all of this information, along with meta-data about the meeting such as the names of participants and the date and location of the meeting can be stored in a multimedia database. The result is that people have access to a much richer resource of information about meetings than they ever had before, and possibly new ways to explore this data.

Archivus screenI was part of a small group that tried to address the issue of how people would want to access that stored information - both in terms of the types of information that they would want to find, and the interactions that they prefer to use to find it. In order to address these questions we designed, implemented and evaluated a multimodal dialogue-driven system for multimedia meeting data retrieval and browsing called Archivus. Archivus acts as a portal to the multimedia meeting database, allowing users to browse and search the data using any of mouse, keyboard, voice or stylus - independently, or in combination with one another.

I was heavily involved in several parts of the application development lifecycle including user requirements gathering (developed questionnaires, analysed and synthesized responses for reports), design specification (ensured user requirements were met, created mockups and low-fidelity prototypes of the system to facilitate communication about the design), and user evaluation (designed the evaluation protocol, developed accompanying documents - instructions for subjects, tutorial and system manual, organized and ran large-scale Wizard of Oz evaluations using a dual-wizard protocol, acted as experimenter and also as one of the wizards during the evaluations, and analysed and interpreted the resulting data.

In addition to work on the Archivus system, I was also involved in the development (and later transcription) of realistic meeting scenarios that could be used for mock meetings. Recording of mock meetings was necessary to develop a database of meetings which could be used by various groups in the project and which were easy to understand by outside parties and contained no sensitive information.

Parmenides (2003-2005)

The Parmenides project was a European Union project concerning 'Ontology driven temporal text mining on organizational data for extracting temporal valid knowledge' whose aim was 'to develop novel techniques for building and maintaining domain specific ontologies, automatic detection and extraction of events in textual data, integrating the textual temporal information which has been extracted in a document warehouse and temporal knowledge discovery tasks for trends analysis, temporal validity of knowledge, etc.'. The project involved academic, public and private sector partners.

Our role in the project was user-oriented system evaluation. My responsibilities involved helping to determine an evaluation plan and appropriate evaluation metrics for the system under development. Our group's work resulted in the conception, design and initial implementation of two software tools:

Quality Model Builder screen The Quality Model Builder (QMB) allows software designers and end-users to develop a hierarchical quality model to help in the evaluation of complex software.

Relative Ordering Tool for Evaluation screen The Relative Ordering Tool for Evaluation (ROTE) is designed to help end-users rank the relative importance of their user requirements in order to guide software developers as to the priorities that should be attached to individual requirements.

Both these tools are available online at

Regulus (2008, consultant)

Regulus GUI screen Regulus is an open-source Prolog-based toolkit for building spoken dialogue systems. Recent developments include the addition of a graphical user interface for the Regulus environment, which had previously been command-line based. I have been performing heuristic evaluations of early versions of the Regulus GUI.

Mobile MedSLT (2008, consultant)

Mobile MedSLT screen The MedSLT system is a speech-to-speech translation system for the medical domain. It was initially developed for use on a desktop or laptop computer, but work has recently begun to move the system onto smaller mobile devices such as PDAs. I have been working with the project team to design the interface and interactions for the mobile version of the system.