A workshop on MT evaluation organised during the AMTA conference in Cuernavaca in 2000 included a series of practical exercises on machine translation evalauation. Carrying out the exercises provided insights into the difficulties and subtleties of MT evaluation, thus inspiring several of those present to suggest the organisation of a longer workshop whose primary focus would be to design and carry out portions of a thorough evaluation.
At the same time, the Evaluation Working Group of the ISLE project (funded by the EU, the NSF in the USA and by the Swiss and Danish governments) has been working on the provision of support material for those involved in MT evaluation. This material takes the form of classification schemes intended to be helpful in the definition of user needs, the choice of system characteristics of importance to the specific evaluation and the choice of metrics to be applied to system characteristics. The current version of the ISLE proposals can be seen at http://www.issco.unige.ch/projects/isle/taxonomy2 .
We invite you to a practical workshop to be held in Geneva between April 19th and 24th 2001.
Here are some directions
on how to reach our building.
The workshop will be take place on the 6th floor, welcoming and general sessions in room 6032, and the practical work in the computing rooms. Please follow the ``ISLE MT Evaluation Workshop'' signs.
The programme of the workshop is available at:
Participants in the workshop will be provided with a scenario
describing a practical situation in which an evaluation of an MT
system or systems might be undertaken. The organisers will ensure
that the scenario(s) reflect real life situations. Particpants will
then spend two days designing an evaluation which is appropriate to
their scenario, using a unified framework (ISLE) described in the
introductory talks. They may choose to work alone or in small groups.
Participants will have free access to the machine translation systems
available on the web, and to the considerable computing support
available at the University of Geneva School of Translation and
Interpretation. As much as possible, the evaluations will be carried
out. Results and experience will be pooled and discussed in the final
day of the workshop. This workshop can be seen as one in an-going
series (LREC 2000, AMTA 2000, NAACL 2001, MT Summit VIII 2001), where
each workshop builds on the experience and the results of previous
workshops. Potential participants need not however have participated
in earlier workshops, although we would of course like to encourage
them to participate in later ones. More information about the
conferences where workshops have been held or will be held can be
NAACL 2001: http://www.cs.cmu.edu/~ref/naacl2001.html
MT Summit VIII: http://www.eamt.org/summitVIII.
Participation in the workshop is free of charge, although particpants must pay their own travel and living expenses. Because of the nature of the exercise, participation is limited to a maximum of 20 persons, and will be on a first come first served basis. Note, though, that if there is a team which would like to participate as a team, these restrictions may be relaxed in order to accomodate them.
Further information can be obtained from Maghi King at Margaret.King@issco.unige.ch or from Florence Reeder at firstname.lastname@example.org.
Important dates and Deadlines
REGISTRATION -- MARCH 15TH 2001
LATEST NOTIFICATION OF ACCEPTANCE -- MARCH 22ND 2001
WORKSHOP -- APRIL 19-24 2001