We have emphasized many times throughout this report that we make no claim to finished work: we have concentrated our work during this first thirty months on one particular kind of evaluation, adequacy evaluation of on market or near market products. We have only been able to carry out hands-on work with products in two areas, writers' aids and translators' aids, and we have only been able to do very preliminary work on the evaluation of knowledge management systems. Although we feel that substantial progress has been made towards the overall goal of defining a general framework for the design of evaluation methodologies, we are aware that much remains to be done. In particular, work during this first complete period has led to the identification of areas where very little indeed has been done, either within the group or in the larger community and which we feel should be given priority in follow-up work.
Obvious directions for future work include applying the framework to other kinds of evaluation and to other areas, perhaps even outside the strict domain of language engineering.
In addition to these obvious areas for future work, the work to date has also revealed the importance of work on user profiling and requirements analysis undertaken in the spirit of identifying and formalising the needs of classes of users. Very little has yet been done on this topic which is of importance to a very wide community, going beyond the bounds of language engineering products to touch the concerns of industry as a whole and potentially providing insights useful to research policy and research and development projects.
On a very practical level, we have been able to sketch the possibility of semi-automated evaluation test-bed which can be parametrized to take account of the characteristics of different objects of evaluation and of different kinds of users who are interested in the results of an evaluation, We think this, too, worthy of further effort and potentially of interest to both industry and the wide community of users.
The territory when we started was almost virgin: it is perhaps worthwhile to close by summarizing where we consider we have started to make inroads into it. We believe that we have achieved the following:
It would be wrong to close this report without thanking again all those who have contributed to it, whether directly by volunteering their labour, by writing sections of it and by providing feedback and comment on this and earlier versions, or indirectly by providing additional financial and moral support. We hope the same spirit of collaboration which has informed our efforts to date will continue into the future.