Quality models and resources for
the evaluation of machine translation

The main objective of this project is to develop a web-based resource (FEMTI) dedicated to context-based evaluation of machine translation software. This novel approach to MT evaluation defines a formal correspondence between contexts of use and required quality characteristics, and implements an interface enabling evaluators to define contextual quality models.

During the first year (Oct. 2004 - Sept. 2005), work focussed on: (1) defining a linking mechanism between contexts of use and quality characteristics, in terms of context/quality vectors and generic contextual quality models (matrices); and (2) implementing a user-friendly interface that generates a quality model on the basis of the intended context of use as specified by an evaluator.

The development of this interface and the migration of FEMTI to a dynamic document server based on Cocoon has created a clear separation of content and form, thus enabling content refinement to proceed independently of work on interface development. In parallel, a thorough assessment of the content of FEMTI (contexts of use, MT qualities and metrics) has led to the revision of the data structures, and has provided an overview of the current state of the resource which will guide upcoming work.

• Dr. Andrei Popescu-Belis - applicant, senior research assistant
• Prof. Margaret King - co-applicant, head of department
• Ms. Paula Estrella - research assistant, doctoral student at ETI since April 2005
• Ms. Nancy Underwood - research assistant

For more information, see the following web pages

FEMTI guidelines, a Framework for MT Evaluation

http://www.issco.unige.ch/projects/isle/femti/

Former ISLE Evaluation Working Group:

http://www.issco.unige.ch/projects/isle/ewg.html