This report represents the work done by the EAGLES Evaluation Working Group in the first 30 months of its existence. Although it is a final report on the work done in that period, there is no sense in which it is a report on a fully completed and polished piece of work, presenting a thoroughly worked out and integrated methodology for the design of evaluations. The problem area we have been attacking is vast, and there was little previous work available as a starting point. Nevertheless, the long term objective of the group is to identify and specify the components of a compendium of evaluation criteria and associated techniques, together with guidelines for their use, from which the individual evaluation user can select those techniques which are relevant to his purpose.
The work of the group has been entirely concerned with natural language processing systems and products. Within this broad area, the group initially selected three types of system for study, writers' aids, translators' aids and knowledge management systems. Work has been narrowly focused on a particular type of evaluation, namely adequacy evaluation, which is the activity of assessing the adequacy of a system with respect to some intended use of that system. The resources available to the group were rather unevenly distributed. In consequence, writers' aids and translators' aids have received rather different treatment from the knowledge management systems, where all that could be done was to set up a set of desirables for such systems or, in other words, to work out a requirements specification from which criteria for evaluation could be deduced.
In what follows, although some sections are, we hope, pre-final in the sense that no complete re-thinking of the basics should be necessary, nothing is to be thought of as totally definitive. With this in mind, we welcome all comments and suggestions, which can be sent either to email@example.com (EWG Editor), firstname.lastname@example.org (Editorial Board) or email@example.com (EAGLES secretariat). The reader is equally welcome to contact individual members concerning particular points of interest.