Next: Functionality requirement - part I
Up: Evaluation of Writers' Aids
Previous: Reportable attributes of writers'
Assuming we have a list of attributes we want to report to the
customer, we need to find ways of measuring these. The specification
of methods is the definition of how to build up a value for a
reportable attribute by combination of results of various component
attributes, down to measurements that can be made directly, `by
inspection'.
The method has to define at least the following:
- Amount, composition and structuring of test material;
- Instantiations of interpretational scheme.
Concerning the amount of test material, the method should specify a
critical threshold, i.e. the minimal amount of material for the
results to be reliable and generalisable. In testing and evaluating
natural language processing systems, we cannot obtain exhaustiveness
in the linguistic coverage of the test material. A higher degree of
completeness can, however, be achieved in types of phenomena to be
tested than in tokens. Yet even as regards types, aiming for a
complete coverage seems unfeasible in practice, which makes the issue
of composition of the test material all the more important. The
structuring of the material is rather a practical matter, as it has to
do with systematising the material in a sound and transparent way,
such that interpretational schemes can be readily applied to the
results of the measures.
Here, we will concentrate on the composition and structuring of the
test material.
ceditor@tnos.ilc.pi.cnr.it