next previous contents
Next: Context and validity in Up: Issues for requirements analysis Previous: Issues for requirements analysis

Reporting evaluations to the customer

The main purpose of a requirements statement from the point of view of software development is to support the subsequent design and implementation phases. When our aim is evaluation, this is no longer a priority. A new end-audience for the statement is important, however: the customer of the evaluation. In fact, this is not so different from the classical view; one of the most talked-of issues in requirements representation is the need to re-present requirements for agreement to the management, the commissioning organisation, experts whose knowledge is being represented, etc. (this is one of the biggest difficulties with formal methods).

The Consumer Report Paradigm mostly assumes a certain familiarity with the products under test on the part of the customer of the evaluation. This allows the reporting of attributes which relate fairly directly to measurements of the system, and whose relevance to individual customers' needs is taken to be evident. Nonetheless, such attribute listings are usually accompanied by explanatory text which can help to make clear the relationship between characteristics of a customer's requirements that are easy to recognise (because they are couched in the customer's own problem vocabulary) and the attributes whose values relate to those requirements. For Linguistic Engineering, the factors of technical performance are not well understood by any mass market audience, and indeed there may not be an adequate vocabulary for the linguistic facts of the problem domain, regardless of system considerations. Explanation of the relevance of attributes might therefore be considered an important part of the function of an evaluation in the area of LE .

How wide should the external bounds of the evaluation be set? One of the criticisms levelled at consumer reports is that they take a given user task, but then assume a certain product class as the candidates to satisfy that need (taking a restricted view of the task). Innovative consumer reports sometimes take seriously the idea of taking the user's need as primary, and preparing evaluation criteria at the top level which do not take for granted a particular type of satisfier for the need. For instance, instead of evaluating cars as though they were a given natural class of satisfiers, some reports have evaluated cars along with combinations of cycling, public transport, and walking, even choice of living area, as potential satisfiers of users' mobility or access requirements. In the case of spelling checkers, it must be relevant to the impact of spelling checkers on the overall task of document quality assurance to know that the errors that spelling checkers are able to spot are also those that people find it easiest to spot (namely, errors that do not result in other legal but unintended words) (Cohen80) .

If we take seriously the needs of the customer, then, we need to analyse their requirements for explanation, as well as just technical requirements.

In knowledge acquisition for expert systems, it has often been remarked that once you try to get information from more than one `expert', you are in trouble, even if these are supposed to be experts in the same field, playing the same rôle in the situation being analysed. In fact, the gathering of requirements is complicated by the fact not only of different rôle-players, but multiple rôles in the situation, which may result in completely different viewpoints on the problem. These may simply involve different vocabularies to describe compatible requirements, but they may also involve genuine clashes in requirements between different stake-holders, e.g., management and end-users, writers and editors. There is a good deal of work being done in Requirements Engineering on how to represent and manage such multiple viewpoints, some of which may be relevant to our needs (Nuseibeh94) .

Where multiple viewpoints do not involve clashes, they can once again be seen as a special case of the redescription activity which is a fundamental way of conceptualising the requirements analysis process.

In a sense, to explain the relationship between descriptions of requirements that are comprehensible to the customer or audience of an evaluation, and those attributes of systems that can be directly measured is to explain why the system-level attributes are valid measures of customer-level requirements, and hence is related to work in the traceability of requirements and issues about the validity of requirements and measures, which are discussed in the following section.

next up previous contents
Next: Context and validity in Up: Issues for requirements analysis Previous: Issues for requirements analysis