MT Evaluation:
Who Did What To Whom

Contents: Proceedings Introduction Programme Original Call for Papers Schedule and Venue About Santiago and the Venue Contact Points / Organizing Committee

Proceedings Introduction to the workshop and to the ISLE project Session 1: The ISLE taxonomy for MT Evaluation and its use Towards a Two-stage Taxonomy for Machine Translation Evaluation (p.1-8) -- Andrei Popescu-Belis, Sandra Manzi, Maghi King. Comparative Evaluation of the Linguistic Output of MT Systems for Translation and Information/Browsing Purposes (p.9-14) -- Elia Yuste-Rodrigo, Francine Braun-Chen. Evaluating MT Output for an Unknown Source Language: an ISLE-based Investigation (p.15-20) -- Keith Miller, Donna M. Gates, Josemina Magdalen, Nancy L. Underwood. Scoring Methods for Multi-Dimensional Measurement of Machine Translation Quality (p.21-27) -- Michelle Vanni, Keith Miller. Session 2: Correlations between evaluation measures Automatically predicting MT systems rankings compatible with Fluency, Adequacy or Informativeness scores (p.29-34) -- Martin Rajman, Anthony Hartley. Predicting Intelligibility from Fidelity in MT Evaluation (p.35-37) -- John S. White. Session 3: Analytic measures of output quality, focusing on noun phrases Quantitative Evaluation of Machine Translation Systems: Sentence Level (p.39-43) -- Palmira Marrafa, Antonio Ribeiro. Predicting MT fidelity from noun-compound handling (p.45-48) -- Monika Forner, John S. White. Setting a Methodology for Machine Translation Evaluation (p.49-54) -- Widad Mustafa El Hadi, Ismail Timimi, Marianne Dabbadie. The Naming of Things and the Confusion of Tongues: an MT Metric (p.55-59) -- Florence Reeder, Keith Miller, Jennifer Doyon, John S. White. Session 4: MT Evaluation in relation to other domains Evaluating the Operational Benefit of Using Machine Translation Output as Translation Memory Input in the Translation Process of Software as Documentation (p.61-65) -- Christine Bruckner, Mirko Plitt. In One Hundred Words or Less (p.67-71) -- Florence Reeder. Introduction Many measures and methods of evaluating MT systems have been developed over the years. The ISLE project, funded jointly by the European Union and the National Science Foundation of the USA, is continuing with work started in the EU's EAGLES project on systematizing these measures and procedures. This workshop will be the fourth in a series that report on and flesh out portions of the systematization. The main thrust of the work is to build up schemes that classify various aspects of import for MT, including user needs, the suggested system characteristics and associated metrics for measuring each of these. The classification schemes relate to ISO work on software evaluation. The work is intended to be useful to those who are considering using machine translation, those interested in comparative evaluation of several MT systems and to MT system developers. A fuller account of the ISLE evaluation work and an overview of the current classification schemes can be found at http://www.issco.unige.ch/projects/isle/taxonomy2/. By the time of the MT Summit, the ISLE project will have organized or been involved in a number of workshops on MT evaluation, namely at LREC in Athens, AMTA in Mexico, Geneva Workshop, and NAACL in Pittsburgh. Participants are encouraged, but not required, to get involved with the Geneva Workshop or NAACL Workshop as these are hands-on exercises in MT. The results of the previous workshops, as possible, will be presented here and feedback based on them serves directly the goal of developing proposals of concrete usefulness to the whole community. This workshop will be seen as a preliminary attempt at reaching conclusions which synthesize results. Information about the previous workshops can be found at: LREC: http://www.icp.grenet.fr/ELRA/lrec2000.html AMTA: http://www.isi.edu/natural-language/conferences/AMTA2000.html NAACL 2001: http://www.cs.cmu.edu/~ref/naacl2001.html MT Evaluation workshop: An invitation to get your hands dirty! http://www.issco.unige.ch/projects/isle/mt-eval-workshop.html, and finally at the MT Summit VIII: http://www.eamt.org/summitVIII. Programme 10:00 - 14:00 (with 15' break)      15:30 - 19:00 (with 30'break) The schedule follows the general conference schedule. Papers will be presented at the beginning of each session, and discussions willfollow every group of papers. Average timing is 25' per paper plus discussion. 10:00-10:15 - Introduction to the workshop and to the ISLE project 10:15-11:55 - Session 1: The ISLE taxonomy for MT Evaluation and its use Towards a Two-stage Taxonomy for Machine Translation Evaluation (p.1-8) -- Andrei Popescu-Belis, Sandra Manzi, Maghi King. Comparative Evaluation of the Linguistic Output of MT Systems for Translation and Information/Browsing Purposes (p.9-14) -- Elia Yuste-Rodrigo, Francine Braun-Chen. Evaluating MT Output for an Unknown Source Language: an ISLE-based Investigation (p.15-20) -- Keith Miller, Donna M. Gates, Josemina Magdalen, Nancy L. Underwood. Scoring Methods for Multi-Dimensional Measurement of Machine Translation Quality (p.21-27) -- Michelle Vanni, Keith Miller. 11:55-12:10- BREAK 12:10-13:00 - Session 2: Correlations between evaluation measures Predicting the Fluency of MT Output from its Parsabilty (p.29-34) -- Tony Hartley, Martin Rajman. Predicting Intelligibility from Fidelity in MT Evaluation (p.35-37) -- John S. White. 13:00-14:00 - Intermezzo: A hands-on exercise in MT Evaluation - Eduard Hovy. 14:00-15:30 - LUNCH BREAK 15:30-17:10 - Session 3: Analytic measures of output quality, focusing on noun phrases Quantitative Evaluation of Machine Translation Systems: Sentence Level (p.39-43) -- Palmira Marrafa, Antonio Ribeiro. Predicting MT fidelity from noun-compound handling (p.45-48) -- Monika Forner, John S. White. Setting a Methodology for Machine Translation Evaluation (p.49-54) -- Widad Mustafa El Hadi, Ismail Timimi, Marianne Dabbadie. The Naming of Things and the Confusion of Tongues: an MT Metric (p.55-59) -- Florence Reeder, Keith Miller, Jennifer Doyon, John S. White. 17:10-17:40 - BREAK 17:40-18:30 - Session 4: MT Evaluation in relation to other domains Evaluating the Operational Benefit of Using Machine Translation Output as Translation Memory Input in the Translation Process of Software as Documentation (p.61-65) -- Christine Bruckner, Mirko Plitt. In One Hundred Words or Less (p.67-71) -- Florence Reeder. 18:30-19:00 - Final discussions and conclusion of the workshop Schedule and Venue The workshop will take place on September 22nd, 2001, after the regular conference. The main conference site contains information about registration and accommodation for both the conference and any satellite events, including this workshop. About Santiago Santiago de Compostela in North West Spain (Galicia) was the most visited pilgrimage shrine in the Middle Ages, and to this day the Road to Santiago is a popular route for walkers through France and northern Spain. The goal is the imposing 11th century cathedral containing the relics of St. James the Great. But the well-preserved historical center of the city offers much more: medieval streets, excellent restaurants, ancient churches, monasteries, palaces, etc. The venue for the conference is the Hostal de los Reyes Católicos, founded in 1499 as a Royal hospital for pilgrims; a magnificent building in the Spanish Renaissance style, and now a hotel (the oldest and one of the most beautiful in the world). Those staying at the Hostal will have a once-in-a-lifetime experience, but there is also accommodation nearby in many other good hotels, all within easy walking distance of the center and the Hostal. Santiago de Compostela has its own airport with daily direct flights from Barcelona, Madrid, Brussels and London. Contact Points and Organizing Committee: For any further information, please contact Sandra Manzi, TIM/ISSCO University of Geneva 40 blvd du Pont d'Arve CH-1211 GENEVA 4 (Switzerland) Tel: +41-22-705.8680 Fax: +41-22-705.8689 Organizing Committee E. Hovy, ISI, hovy@isi.edu B. Maegaard, CST, bente@cst.ku.dk S. Manzi, ISSCO, M. King, ISSCO, margaret.king@unige.ch F. Reeder, MITRE, freeder@mitre.org A. Popescu-Belis, ISSCO,