The purpose of EAGLES evaluation work on knowledge management systems (KMSs) is twofold:
Initially, we aimed at concentrating on general information retrieval, possibly with a natural language front end, with emphasis on different kinds of semi-automatic (computer-assisted) and automatic indexing for text (document) retrieval. However, due to lack of manpower resources and after having scanned the market for IR products that can, at least to a certain extent, support NLP applications, such as the analysis of corpora or the construction of term banks, we have decided to focus on one specific area of application, namely technical documentation and document production. In the near future, this area will be essential for massive industrial and academic research and development, e.g. for product localisation operations, value-added services and multilingual language technologies, in particular when taking into account the amalgamation of computers, media and telecommunications.
The focus of work will be on technical documentation, because general texts would be too imprecise due to the fact that such texts can be interpreted very differently depending on the intellectual interests, background and knowledge of a potential user of an IR system; technical texts, however, are usually not subject to broad interpretation. Therefore, we see an IR system not only as a general information provider (broad view) for a specific subject, but also as a knowledge-based support tool for the extraction and maintenance of the heterogeneous knowledge needed for technical documentation (application-oriented view).
The result of our market survey was disappointing in that commercially available systems provide only very limited functionality with respect to their applicability to NLP. Although IR provides good solutions for storing textual data, in contrast to general database management systems, it does not allow for complex structuring of the data and flexible access methods, because of its strict character string orientation.
Today, several industrial companies have sophisticated tools available for supporting their in-house information retrieval procedures and their technical documentation. However, these products are not yet available to the public and therefore they cannot be taken into account for EAGLES purposes.
Given this situation, we will not evaluate existing systems (or only to a limited extent), as is done in other sections of this chapter, but will rather define application oriented requirements for a KMS that is able to support large-scale NLP related applications. Based on these requirements we will be able to specify (to some degree) the functionalities of such a KMS, which will also lead to basic evaluation methods (perhaps a kind of de-facto standard). These methods will take into account the methods applied in other sections, i.e. adequacy, diagnosis and progress evaluation. However, relevant research and development systems in this area are also taken into account, such as the CODE system (Skuce93), RELATIO/IR (Rahmstorf94), SYSTEM QUIRK (Ahmad93) of the ESPRIT project Translator's Workbench (TWB) and The Information Theater (XeroxParc92).
As already mentioned, the area to be studied is technical documentation and the focus will be on an information (knowledge) repository that supports a technical writer during the various processes of documentation and manual production, such as planning, information (knowledge) acquisition, drafting, reviewing and production.
Given the limited time and budget frame of EAGLES , this work will be mainly based on work-in-progress and the research interests of the author. However, we will keep the results as open to generalisation as possible.
The impact of the work will be twofold. On the one hand, it will provide developers with guidelines for the design of future knowledge-based IR systems, thus promoting the creation of competition which may improve products, and, on the other hand, it will allow potential users to evaluate new products according to their needs.
This section is organised as follows: first, in section The KMS scenario we will set up the general KMS scenario which will form the basis for the specification of a knowledge-based assistant for technical writers which is briefly presented in section Case study: KBAS-TW. The focal point of this system will be its knowledge management component that will facilitate the building of the system's knowledge base. We will describe the general requirements for such a system and give an overview of its architecture. The requirements are mainly derived from empirical research results which are available as a separate document(Ross, forthcoming) . This then leads us to different evaluation criteria outlined in section Towards evaluation. The criteria will be initially defined here and will thus form the basis of future work within the EAGLES .