Originating in ESPRIT projects, KADS (Hayward87) is widely used within the EU and increasingly beyond as a practical KBS development methodology.
KBS development according to KADS is essentially a process of modelling at various levels. Relating this to the ideas of Requirements Engineering, KADS provides a framework of representations and process suggestions for producing system descriptions at different levels of abstraction.
Three levels of analysis are distinguished. At the Process Level, a Process Model identifies the tasks involved in the domain, data flows and stores, and the assignment of ownership of tasks and data stores to agents. At the System Level, the Co-operation Model describes in detail the interactions between the system and external agents, and how `internal agents' (components of the system) will interact; there may also be separate a User Task Model and System Task Model. The third level, the Expertise Level corresponds to an Expertise Model, which divides the task of describing the expertise level of the system into a number of layers:
A major advantage of the KADS approach is in the idea of generic task models (hereafter GTMs), also known as interpretation models. These are partially instantiated skeleton models for typical tasks or task fragments -- such as ``classification'' or ``system diagnosis'' (Tansley93) -- which are stored in generic task libraries, promoting reuse and standardisation. Knowledge engineers can use suitably chosen generic task models to guide the knowledge acquisition process in a new domain, refining and combining GTMs to produce a fully specified model.
The basic approach of identifying a sequence of modelling steps and supporting the modelling activity with reusable elements is echoed in the proposals made later in this document. KADS has been suggested before as a method for KA for NLP (Mikheev94) , in which the descriptions of the expertise level were used to structure NLP knowledge acquisition. The proposals made in the main part of this report draw on the whole KADS process as a useful model for the requirements modelling stage of NLP evaluation.
KADS does have weaknesses, which any attempt to adapt it must be wary of. For instance:
KADS does not itself specify the representation types in terms of which its various models are to be couched. The field of Requirements Engineering encompasses representations ranging from long discursive reports associated with ethnomethodological approaches, to formal languages like Z. A relatively common representation which might be seen as intermediate between these approaches is conceptual graphs (Mikheev94) -- it is sometimes proposed on the grounds that it is more acceptable to users than stricter logical formalisms, particularly since it can be given a graphical representation.
The last point is important, since we must decide what our needs are for representations. Suitable representations must have a two-sided functionality: on one hand, it must be able to express the language of our testing techniques, and on the other it must be able to describe systems in such a way that they are recognisable to those who must contribute to the development of evaluation models.
Strictly formal specification languages are not likely to be very usable for large, real, complex NLP systems and their users. However, in the absence of such methods, the organisation and procedure of requirements development needs to be particularly disciplined and thorough, with lots of check-lists to prompt the preparer of the evaluation to obtain relevant information.