MT Evaluation:
Who Did What To Whom

Contents: Introduction Papers Submission Guidelines Important Dates Schedule and Venue About Santiago and the Venue Contact Points / Organizing Committee

Introduction Many measures and methods of evaluating MT systems have been developed over the years. The ISLE project, funded jointly by the European Union and the National Science Foundation of the USA, is continuing with work started in the EU's EAGLES project on systematizing these measures and procedures. This workshop will be the fourth in a series that report on and flesh out portions of the systematization. The main thrust of the work is to build up schemes that classify various aspects of import for MT, including user needs, the suggested system characteristics and associated metrics for measuring each of these. The classification schemes relate to ISO work on software evaluation. The work is intended to be useful to those who are considering using machine translation, those interested in comparative evaluation of several MT systems and to MT system developers. A fuller account of the ISLE evaluation work and an overview of the current classification schemes can be found at http://www.isi.edu/natural-language/mteval/. By the time of the MT Summit, the ISLE project will have organized or been involved in a number of workshops on MT evaluation, namely at LREC in Athens, AMTA in Mexico, Geneva Workshop, and NAACL in Pittsburgh. Participants are encouraged, but not required, to get involved with the Geneva Workshop or NAACL Workshop as these are hands-on exercises in MT. The results of the previous workshops, as possible, will be presented here and feedback based on them serves directly the goal of developing proposals of concrete usefulness to the whole community. This workshop will be seen as a preliminary attempt at reaching conclusions which synthesize results. Information about the previous workshops can be found at: LREC: http://www.icp.grenet.fr/ELRA/lrec2000.html AMTA: http://www.isi.edu/natural-language/conferences/AMTA2000.html NAACL 2001: http://www.cs.cmu.edu/~ref/naacl2001.html MT Evaluation workshop: An invitation to get your hands dirty! http://www.issco.unige.ch/projects/isle/mt-eval-workshop.html, and finally at the MT Summit VIII: http://www.eamt.org/summitVIII. Papers We invite submission of papers along the themes discussed above. The questions and issues to be answered are diverse. All papers on evaluation and evaluation issues will be considered, but preference will be given to papers relating to the ISLE framework. The following questions suggest possible evaluation threads within the framework. What kind of metrics are useful for what system characteristics? What system characteristics reflect what user needs? Is there a radical difference between evaluation focusing on research or development needs and evaluation focusing on end-user needs? When should real-world data be used, and what is the impact of using it? What constitutes a valid metric? How can you demonstrate that a metric is valid? What are the advantages/disadvantages of specific metrics? What kinds of tools automate the evaluation process? Can the process (or any part of it) be automated? What are the difficulties inherent in choosing particular metrics for automation? What kinds of tasks are suited to which evaluation schemes? How can we use the evaluation process to speed or improve the MT development process? How can we evaluate MT when MT is a small part of the data flow? How independent is MT of the subsequent processing? Cleaning up the data improves performance, but does it improve it enough? How do we quantify that? Submission Guidelines Papers may be submitted electronically to Sandra Manzi, (Sandra.Manzi@issco.unige.ch) by the submission date. The format may be RTF, Microsoft Word, or PostScript. Any specialized fonts should be included in the submission package. Users are advised to follow the main conference formatting as described at: http://www.eamt.org/summitVIII/guidelines.html. Submission for review: papers of no more than 6000 words are expected. Submission for publication: A template will be provided for accepted papers. Important Dates Paper submission deadline April 15, 2001 Notification of acceptance May 31, 2001 Final camera-ready due July 1, 2001 Schedule and Venue This one-day workshop will take place on September 22nd, 2001, after the regular conference. The main conference site contains information about registration and accommodation for both the conference and any satellite events, including this workshop. About Santiago Santiago de Compostela in North West Spain (Galicia) was the most visited pilgrimage shrine in the Middle Ages, and to this day the Road to Santiago is a popular route for walkers through France and northern Spain. The goal is the imposing 11th century cathedral containing the relics of St. James the Great. But the well-preserved historical center of the city offers much more: medieval streets, excellent restaurants, ancient churches, monasteries, palaces, etc. The venue for the conference is the Hostal de los Reyes Católicos, founded in 1499 as a Royal hospital for pilgrims; a magnificent building in the Spanish Renaissance style, and now a hotel (the oldest and one of the most beautiful in the world). Those staying at the Hostal will have a once-in-a-lifetime experience, but there is also accommodation nearby in many other good hotels, all within easy walking distance of the center and the Hostal. Santiago de Compostela has its own airport with daily direct flights from Barcelona, Madrid, Brussels and London. Contact Points and Organizing Committee: For any further information, please contact Sandra Manzi, (sandra.manzi@issco.unige.ch) TIM/ISSCO University of Geneva 40 blvd du Pont d'Arve CH-1211 GENEVA 4 (Switzerland) Tel: +41-22-705.8680 Fax: +41-22-705.8689 Organizing Committee E. Hovy, ISI, hovy@isi.edu B. Maegaard, CST, bente@cst.ku.dk S. Manzi, ISSCO, sandra.manzi@issco.unige.ch M. King, ISSCO, margaret.king@issco.unige.ch F. Reeder, MITRE, freeder@mitre.org A. Popescu-Belis, ISSCO, andrei.popescu-belis@issco.unige.ch