EVALUATION WORKING GROUP

ISSCO

CST

Many measures and methods of evaluating MT systems have been developed over the years. The ISLE project, funded jointly by the European Union and the National Science Foundation of the USA, is continuing with work started in the EU's EAGLES project on systematizing these measures and procedures. The main thrust of the work is to build up schemes that classify various aspects of import for MT, including user needs, the suggested system characteristics and associated metrics for measuring each of these. The work is intended to be useful to those who are considering using machine translation, those interested in comparative evaluation of several MT systems and to MT system developers.
The ISLE project has been organizing a series of workshops on MT evaluation, namely at LREC in Athens, AMTA in Mexico, Geneva Workshop, NAACL in Pittsburgh, MT Summit in Santiago de Compostela. Most of these are hands-on exercises in MT. The results of the workshops bring feedback that serves directly the goal of developing proposals of concrete usefulness to the whole community. Information about the workshops can be found at:
Workshop on the Evaluation of Machine Translation at LREC 2000 (Athens) Hands on MT Evaluation Workshop at AMTA 2000 (Mexico) MT-Eval Workshop "An Invitation to get Your Hands Dirty", University of Geneva Workshop on MT Evaluation at NAACL 2001 (Pittsburgh) MT Evaluation Workshop "Who Did What to Whom" at MT Summit VIII (Compostella) MT Evaluation: "Human Evaluators Meet Automated Metrics" - a Workshop at LREC 2002 (Las Palmas, Canary islands) MT Evaluation Workshop at USC/ISI, Marina del Rey, February 10-14, 2003 (internal workshop) MT Evaluation Workshop: "Towards Systematizing MT Evaluation" - a Workshop at MT Summit IX (New Orleans, Louisiana, USA - Sept. 2003) MT Evaluation Tutorial: "Context-based Evaluation of MT Systems: Principles and Tools" - at MT Summit XI (Copenhagen, Denmark - Sept. 2007) MT Evaluation Tutorial: "Evaluating MT in Use: from theory to practice" - at 6th LREC Conference (Marrakech, Morocco - May 2008)

Working Group Documents

FEMTI: Framework for the Evaluation of MT in ISLE

EAGLES 7-step Recipe for NLP Evaluation

Georges Van Slype (1979) - Critical Study of Methods for Evaluating the Quality of Machine Translation. Final Report, Bureau Marcel van Dijk / European Commission, Brussels.
List of NLP evaluation initiatives and projects

List of web-based MT systems

Language pairs offered by MT systems

Translator's viewpont on MT vs. inexperienced human translators

ISSCO Admin

Last modified: Mon Oct 1 15:29:13 MET DST 2007