next previous contents
Next: Functionality requirement - part I Up: Evaluation of Writers' Aids Previous: Reportable attributes of writers'


Assuming we have a list of attributes we want to report to the customer, we need to find ways of measuring these. The specification of methods is the definition of how to build up a value for a reportable attribute by combination of results of various component attributes, down to measurements that can be made directly, `by inspection'.

The method has to define at least the following:

Concerning the amount of test material, the method should specify a critical threshold, i.e. the minimal amount of material for the results to be reliable and generalisable. In testing and evaluating natural language processing systems, we cannot obtain exhaustiveness in the linguistic coverage of the test material. A higher degree of completeness can, however, be achieved in types of phenomena to be tested than in tokens. Yet even as regards types, aiming for a complete coverage seems unfeasible in practice, which makes the issue of composition of the test material all the more important. The structuring of the material is rather a practical matter, as it has to do with systematising the material in a sound and transparent way, such that interpretational schemes can be readily applied to the results of the measures.

Here, we will concentrate on the composition and structuring of the test material.