The first step in any evaluation procedure should be to analyse user requirements, from which relevant quality characteristics are deduced and broken down in terms of reportable attributes. In parallel to the definition of the user, the objects of evaluation are also defined in terms of attributes. Relevant metrics are then developed, taking into account both definitions. Having defined attributes and metrics for the particular evaluation environment, the next step is to decide what methods should be applied in order to perform the measurement and to obtain values for the attributes. This section sketches the contents of a library of methods for system measurement from which the most appropriate test type and methods for data collection and data reporting can be chosen.
Among the broad software engineering community, there is little consensus concerning the exact nature of tests that are appropriate for software evaluation. Much has been said about the importance of software quality assurance from the viewpoint of project management and developers: little has been presented from the viewpoint of the end-user, who so far has played a rather passive role in software development and evaluation. Moreover, the principal aim of existing testing methods, as reported in the literature, is to uncover software errors and not to allow the adequacy of systems for a particular user environment to be assessed: naturally enough, it is this latter which is primarily of interest to users.
The methods presented here as a library for system measurement are user-oriented in the sense that evaluation is seen as being performed on behalf of a user or of a class of users. Furthermore, they are based on general software engineering principles, taking into account development-oriented testing expertise using both glass box and black box testing. (By glass box testing here we mean that access to intermediate results of processing is possible, by black box testing that only input and output behaviour can be observed.)
For ease of presentation and in order to facilitate the decision process, we make a clear distinction between the motivation underlying testing, which leads to a choice of the test type suitable for the particular purpose, and the description of methods for data collection, i.e. the instruments that can be used in combination with the test types.
This section contains only very brief descriptions of possible methods for system measurement.