We shall conclude our discussion by showing how techniques for evaluation can be derived from the design stages of one specific system (product) and how they can be combined into an integrated evaluation methodology for knowledge management systems. The goal of the integration is to create a top-down evaluation process that reduces user and data requirements to a standard evaluation structure (ideally interpretable by computers). Such a process shall be called an evaluation methodology and contains:
The evaluation process should be supported by appropriate software tools. These will assist evaluators in two ways. First, they will act as a repository for the documentation generated during the evaluation, thereby enabling evaluators to draw up models, to store them and to change them. Secondly, such tools can help to develop accurate models and to convert a model at one level to that used at the next level of the evaluation process. This is essential for diagnostic and progress evaluation.
Evaluation methodologies are constructed by:
The evaluation stages must follow the natural design process of a product to be evaluated, which begins by collecting and analysing the data regarding the application that aims at employing the product (purpose of use). The evaluation specification is then prepared and the technical evaluation follows. The evaluator must make the following choices when selecting a methodology:
Each choice means selecting evaluation techniques and a method for documenting the output from each evaluation stage.
Documentation serves two purposes in an evaluation methodology:
Documentation is a record of all the decisions made during evaluation and it contains the final model produced at each stage, which can be used to monitor the project's progress (useful for progress evaluation).
The goal of the data collection is to produce a formal document of user requirements. This document must include the data needed by users as well as the use made of the data. Evaluators should interview the users and then obtain even more detailed information from other sources, such as:
The latter source comprises a detailed examination of the envisaged activities. This analysis differs from the functional descriptions in that it defines, principally, the inputs and outputs of the application's functions. Scenario analysis is thus a deeper examination of the relationships among objects in the function, searching mainly for the activities in the application and building from them.