next previous contents
Next: COBALT Up: Ongoing projects Previous: Ongoing projects

TEMAA - A testbed study of evaluation methodologies

The TEMAA project is an LRE project carried out by researchers and developers at Center for Sprogteknologi (Copenhagen), ISSCO (Geneva), Stichting Taaltechnologie (Utrecht), Language Technology Group (Edinburgh) and Claris Ireland (Dublin) (see also (Thompson94)).

The users involved in this project are users of spelling checkers in English, Italian, Danish and Dutch -- mother tongue users, as well as foreign language users. The classes of users, apart from the managers and end-users, which will profit from the results are: developers, vendors and funders.

The objective of the project is to provide a generally applicable framework for the evaluation of NLP products. The project also studies project evaluation to some extent, but the focus is on adequacy evaluation. The framework is expressed as a number of guidelines to be followed when setting up an evaluation of a specific type of product. These guidelines are implemented in an object-oriented environment to facilitate modelling of user requirements and system characteristics.

The result is the development of a Parameterized Test Bed (PTB). Test materials for testing the lexical coverage of spelling checkers have also been constructed. The TEMAA framework is modelled in CLIPS following an object-oriented programming (OOP) paradigm. The project has also developed an experimental tool supporting automatic spelling checker checking (ASCC) which is interfaced with CLIPS. The tool is a collection of programs in WINBATCH, C and Perl. The results of the project will be documented in reports which will hopefully be widely accepted as constituting a methodological framework for evaluation to be further elaborated. Other results include an evaluation package, covering some of the test materials necessary for the evaluation of spelling checkers for Danish, English and Italian, a program, ASCC, which automatically transforms correctly spelled words into misspelled words and measures the ability of the spelling checker to find the errors and to produce a good suggestion, and a prototype of a test modelling tool for the description of an evaluation of some NLP product, i.e. an assessment of the way in which system characteristics are mapped onto different sets of user requirements. The final report will give a good description of the system. The PTB implemented in CLIPS is an experimental prototype aiming at demonstrating the soundness of the general approach, covering a subset of the user requirements and system parameters relevant to spelling and grammar checkers. Likewise, only an interesting subset of typing and spelling errors has been implemented in ASCC. The project envisages that the results could be taken over by organisations such as ISO. As for new partner types, the project envisages developers of the product type in question and standardisation bodies or user organisations.

next up previous contents
Next: COBALT Up: Ongoing projects Previous: Ongoing projects