Quality models and resources for
the evaluation of machine translation

SNSF Project n. 200021-103318

October 2004 - September 2006

The main objective of this project is to develop a web-based resource (FEMTI) dedicated to context-based evaluation of machine translation software. This novel approach to MT evaluation defines a formal correspondence between contexts of use and required quality characteristics, and implements an interface enabling evaluators to define contextual quality models.

During the first year (Oct. 2004 - Sept. 2005), work focussed on: (1) defining a linking mechanism between contexts of use and quality characteristics, in terms of context/quality vectors and generic contextual quality models (matrices); and (2) implementing a user-friendly interface that generates a quality model on the basis of the intended context of use as specified by an evaluator.

The development of this interface and the migration of FEMTI to a dynamic document server based on Cocoon has created a clear separation of content and form, thus enabling content refinement to proceed independently of work on interface development. In parallel, a thorough assessment of the content of FEMTI (contexts of use, MT qualities and metrics) has led to the revision of the data structures, and has provided an overview of the current state of the resource which will guide upcoming work.


For more information, see the following web pages

FEMTI guidelines, a Framework for MT Evaluation
Former ISLE Evaluation Working Group:

Back to:  ISSCO -- ISSCO's Projects

Andrei Popescu-Belis -- Last modified: Mon Nov 21 2005