next previous contents
Next: Scenario tests Up: Methods for System Measurement Previous: Black box testing

A user-oriented model of test types

This section has two principal motivations, i.e. (i) to provide a comprehensive and theoretically sound definition of types of test which allow the judgement of the performance of a piece of software as seen from the eyes of the user and (ii) to give some practical advise to all those people who need to assess the quality of a certain piece of software for a particular reason.

Considering both methodological attempts to define software evaluation and practical test reports in the broad software engineering area, one may roughly distinguish between three principal motivations behind testing, i.e.

  1. to assess the appropriateness of a piece of software for every-day work
  2. to examine the behaviour of software under specific conditions
  3. to check the actual functionality of a piece of software

While both software developers and users may share the principal motivation behind testing to a certain extent, it is obvious that the actual testing procedures will differ largely. Despite this and the fact that there is no terminological consensus among leading software engineers (Sneed87), (Musa87), (Hausen84), (Hausen87), (Hausen82), (Hausen87), (Fagan76) and (Howden80), it is necessary that the extensive testing experience of software engineers be considered when developing a model for user-oriented testing, while at the same time introducing new user-oriented testing terminology. The terminology used here partly goes back to (Ahmad93a, 9). Accordingly, the three major test types discussed here will be called scenario tests Scenario Tests, systematic tests Systematic Testing, and feature inspection Feature Inspection.

Apart from the principal objective of evaluation, the practical choice of test type and instruments to be used depends on various factors such as (i) the testing budget, (ii) the testing environment, i.e. whether a professional testing team or a user company undertakes the software tests; (iii) the availability of evaluation and testing expertise and equipment; (iv) time that is going to be invested in testing; and (v) the availability of users as subjects for tests (Crellin90, 331). In addition to a theoretical definition of the different types of test, the practical guidelines provided will discuss the typical ISO quality characteristics which can be assessed, the (type of) personnel involved, the (type of) instruments employed, the (type of) results achieved and their constraints on objectivity and informativeness.

The following model of test types is meant as a starting point for anyone who has to perform user-oriented software tests. For each practical evaluation scenario with its specific environmental constraints, however, individual decisions have to be made concerning the most appropriate test type(s). For most cases it will become obvious that ``in order to evaluate the usability of a system it is often necessary to use several methods in combination.'' (VainioLarsson90, 324) and (Hausen87, 141).

next up previous contents
Next: Scenario tests Up: Methods for System Measurement Previous: Black box testing