Regarding the whole process of product testing, it is then possible to list various criteria (Siderius89):
Most of these criteria are interrelated: reliability is a necessary but not sufficient condition to obtain validity; reliability and validity together are necessary but not sufficient conditions to obtain relevance.
The test procedure consists of several successive phases (Willenborg85; Siderius89). These different stages refer to requirements concerning the testing procedure and the methods involved, complementary to the ones listed above:
The first stage is described as being related to the priorities and targets of product information research. These two aspects affect the selection process.
The second stage tries to answer the question -- which product characteristics are essential when comparing product alternatives (of one class of products)? According to (Willenborg85), essential product characteristics are those allowing one to judge:
Other selection criteria, mentioned by (Box79), are: (assumed) relevance for the consumer (needs and patterns of use) and testability (objectively, in a standardized way).
With regard to product characteristics, generally two types are distinguished: non-external and external characteristics. The latter type can be established and evaluated by simply observing the product or making inquiries, whereas the former requires a much more complicated/extensive analysis. Comparative product testing is primarily involved in testing non-external characteristics.
(Willenborg85) stresses the importance of consumer research for selecting and defining product characteristics (nevertheless he is aware of possible temporal and financial restrictions). In this respect he reproduces a working method described by (Cuthbert79):
As to the third step, developing measuring methods, (Willenborg85) distinguishes two types of measuring method:
Test methods that lead to results having a clear relationship to the performance of a product in practical use and that are to be used as a basis for information to consumers about the performance characteristics of the product.These methods are available for a limited number of products and they result from international activities on standardization (HellmanTuiter83) .
A much more elaborate set of measuring methods has been developed and applied by various consumer associations and collected (more than 800) by the International Organization of Consumer Unions (IOCU).
(Siderius89) stresses the importance of choosing the right measuring level and discusses the four most important levels:
According to (Siderius89), it is indispensable to measure at least at interval level to guarantee the independence of the measuring method with respect to the other phases of the test procedure.
The fourth stage, performing the measurements, produces the test results concerning the measurable quantities of the product characteristics.
During the fifth stage, the test results are interpreted as data forming part of a real user situation.
The sixth stage coincides to some degree with the previous one, but has a more subjective aspect: the test results are translated into plus and minus signs. (Willenborg85) does not specify how this can be done, but he stresses the importance of basing the judgements on data collected with the help of consumer research. Other variables he considers important for the interpretation of the test results are norms concerning product quality minimally wished for, average product quality and variation of the product quality.
The plus and minus signs serve as a basis for determining a global test result per product. The individual characteristics can have different relative weights (weighting should also be based on the results of consumer research). After having compared the total test results of the various products, the consumer organizations select the best buy.
Critical remarks to be made concerning the reliability of this weighting procedure include:
Comparative product tests can be described roughly as consisting of two successive phases: a testing phase and an information phase (Siderius89) . The second phase consists of interpreting the test results as product information and therefore coincides with the last step of the test procedure discussed in paragraph Interpreting and evaluating data.
The main activity during this second phase can be described as evaluative transformation: the test results at interval or ratio level are translated into data situated at the ordinal level; the ordered data are then evaluated in terms of, for instance, the annotation satisfactory, almost satisfactory or unsatisfactory. According to (Siderius89) the consumer will assimilate data more easily when they are presented in a certain arrangement expressing a value judgement.
This transformation implies applying to the test results firstly the criteria used for evaluating the products, secondly their ultimate values and thirdly the weighting factors (see paragraph Interpreting and evaluating data). These three variables are set prior to the actual product testing. The procedure of setting the criteria, ultimate values and weighting factors should not only refer to the product's functionality, but also to the consumer. As for the criteria, these are mostly derived from the functional analysis of the product. However, determination of the ultimate values and weighting factors often requires additional research of a technical or sociological nature.
The following criteria can be applied to judging the product information resulting from comparative product tests (Willenborg85). The information has to:
The information phase is completed when the product information has been presented (e.g. in tables) and distributed (e.g. monthly magazines). As for the presentation of the information, (Box79) makes the following suggestions: