MT Evaluation: An invitation
to get your hands dirty!

Organizers

In conjunction with the MT Evaluation Working Group of the
ISLE project: Sandra Manzi and Andrei Popescu-Belis, ISSCO

Daily schedule

Exceptions: on day 1, welcome between 09:30 and 10:00, workshop begins at 10:00.
On last day, end of workshop at 15:30 for early departures.
 

Useful www links

Location

Thursday, 19 April 2001 

   Welcome, registration, coffee and orange juice (09:30-10:00)
   Introduction to the workshop, outline of the program, handing out of materials
   Overview of evaluation guidelines from ISO-9126, 14598 and EAGLES
   Formal model of evaluation (measure, appreciation, integration)

11:20-11:40 - BREAK

11:40-13:00 - Initial presentations and teaming

   Brief self-introduction of each participant (interest in MT evaluation,
     competencies, language pairs and tools/resources of interest,
     data brought to the workshop)

   Presentation of the requirements for teams 

   Informal discussions to set up teams start during lunch 

13:00-14:00 - LUNCH 

14:00-15:30 - Introduction to the construction of evaluation tasks (1/2)

     Presentation of the latest ISLE taxonomy: how to use it, how to make
       comments

     Outline of evaluation challenges, with a discussion of available data
       and tools

15:30-15:50 - BREAK

15:50-17:00 - Introduction to the construction of evaluation tasks (2/2)

     Guidelines for the design of evaluation procedures

     Possible tasks for MT systems

     Overview of available MT systems

     Informal discussions to set up teams continue

17:30 - WELCOME RECEPTION COURTESY OF THE ETI (Ecole de Traduction et d'Interpretation)
 

Friday, 20 April 2001 

     Recording of teams, tasks and language pairs

     Presentation and distribution of linguistic resources with appropriate
       warnings and caveats

     Practical information 

     Optional tutorial on the use of Reverso (Softissimo)

10:50-11:10 - BREAK

11:10-13:00 - Design and execution of evaluation procedures (1/4)

     Organizers will float to answer questions, join discussions, provide 
       technical support on resources (tools and data).

     Participants are encouraged to allow sufficient time for each of the points
       below (a through d) in the four time slots devoted to the evaluation
       procedure (1/4 to 4/4).

    (a)  Design (using the ISLE taxonomy)

    (b)  Execution (using a system, resources, metrics)

    (c) Interpretation of scores and metrics (how these fit into the taxonomy, 
       how they might be automated, how they contribute to evaluation in general)

    (d) Synthesis of results, writing up of summaries and presentations

13:00-14:00 - LUNCH

14:00-17:00 - Design and execution of evaluation procedures (2/4)

19:30 - Get together dinner at a local restaurant
 

Saturday, 21 April 2001

13:00-14:00 - LUNCH 

14:00-17:00 - Free time or working sessions if groups so choose
 

Sunday, 22 April 2001

Monday, 23 April 2001

13:00-14:00 - LUNCH 

14:00-15:30 - Design and execution of evaluation procedures (5/4!)

15:30-17:30- Presentation of results by each team (1/2)

     Reports on the evaluation procedures may for instance include the
       following points: task/system/data; how the evaluation procedure was
       designed, why, is it conformant to the ISO/EAGLES guidelines, how was
       the ISLE taxonomy used (or why not); which metrics were used, why, how
       they were integrated/balanced; comments on the scores; conclusion

     Time allotted to each team depends on number of teams (minimum of 15
       minutes for each team).

19:30 - Get together dinner at a local restaurant
 

Tuesday, 24 April 2001

09:30-11:20- Presentation of results by each team (2/2)

11:20-11:40 - BREAK

11:40-13:00 - General discussion 

     Wrap-up discussion of results

     Comments/discussion on specific issues, selected from the list of
       questions proposed in the call for participation to the workshop

     Write-up for MT Summit Workshop report, for the taxonomy & ISLE report

13:00-14:00 - LUNCH

14:00-15:30 - Conclusions and plans for the future

     Discussion of future actions, outline of final report for organizers
       and interested participants. Proposals for submissions or invited
       papers to future workshops

     Ideas for future evaluation, possible follow-ups

15:30 - End of workshop