As already mentioned, the long term objective of the group is to identify and specify the components of a compendium of evaluation criteria and associated techniques, together with guidelines for their use, from which the individual evaluation user can select those techniques which are relevant to his purpose.
Once this compendium exists it can be used as a starting point to determine a standard against which new and existing products can be measured and compared. It may also be useful for systematising the definition of a goal or set of goals for new projects and for measuring progress towards those goals.
Although the group will seek to keep abreast of the development of evaluation and assessment methods for speech technology, and may indeed find some inspiration there, the primary emphasis is on evaluation of systems applied to written text. Within that constraint, the group will in the long term consider a wide range of application areas for natural language processing systems, ranging from spelling, grammar and style checkers at one end of the market to natural language interfaces and machine translation systems at the other.
Application areas to be considered will include:
This list is not intended as either definitive or exclusive. Note too that although the list is organised around the notion of an application area, this is not meant to imply that the group will have no interest in the evaluation of basic or applied research.