Basic requirements of comparative product testing

Regarding the whole process of product testing, it is then possible to list various criteria (Siderius89):

Reliability of product testing:

Degree to which repetition of the measurement produces the same/similar results. Subcriteria:

Conformity:

Degree to which repetition of the measurement with different specimens of a product (with identical characteristics) produces the same/similar results (product variation);

Reproducibility:

Degree to which repetition of the measurement on the same object produces the same/similar results.

Validity of product testing:

Degree to which the measurement results reflect the functionality of a product as experienced by the consumer. Subcriteria:

Internal validity:

Degree to which variables of use and environment are included in the research conditions;

Measurability:

Degree to which the product characteristics/functions of the product can be translated into measurable quantities;

Interpretability:

Degree to which the test results can be translated into different aspects of the functionality of the product.

Relevance of product testing

: degree to which the product information enables consumers to buy and use products that contribute as much as possible to the satisfaction of their needs. Subcriteria:

• Importance of the product tested and its characteristics;
• Degree of detail in which the test results on the product's functionality are presented;
• Value judgement of certain levels or ultimate values concerning test results on the product's functionality;
• Comprehensibility/clarity of the presentation of the product information;
• Availability of the product information.

Feasibility of product testing:

Amount of resources that has to be activated and checked to do the testing and to have the required methodologies at one's disposal.

Most of these criteria are interrelated: reliability is a necessary but not sufficient condition to obtain validity; reliability and validity together are necessary but not sufficient conditions to obtain relevance.

A procedure for comparative product testing

The test procedure consists of several successive phases (Willenborg85; Siderius89). These different stages refer to requirements concerning the testing procedure and the methods involved, complementary to the ones listed above:

1. Selecting the product class and the products to be tested;
2. Selecting and defining the attributes;
3. Developing measuring methods (measures and measuring process);
4. Performing the measurements;
5. Analysing the measuring results;
6. Interpreting and evaluating the data.

Selection of product class and products

The first stage is described as being related to the priorities and targets of product information research. These two aspects affect the selection process.

Selecting and defining the attributes

The second stage tries to answer the question -- which product characteristics are essential when comparing product alternatives (of one class of products)? According to (Willenborg85), essential product characteristics are those allowing one to judge:

• The primary function of the product (characteristics such as efficiency and output);
• The correct use and maintenance of the product;
• The economic value of the product (characteristics such as expenses of use, the material, contents and, if relevant, the price per standard amount);
• External effects of purchasing and/or using the product (characteristics such as aspects concerning the environment, raw material, energy, sound production, ...);
• Whether the health and safety of the user and his environment may be endangered;
• What avenues lead to legal recourse, such as the name of the manufacturer.

Other selection criteria, mentioned by (Box79), are: (assumed) relevance for the consumer (needs and patterns of use) and testability (objectively, in a standardized way).

With regard to product characteristics, generally two types are distinguished: non-external and external characteristics. The latter type can be established and evaluated by simply observing the product or making inquiries, whereas the former requires a much more complicated/extensive analysis. Comparative product testing is primarily involved in testing non-external characteristics.

(Willenborg85) stresses the importance of consumer research for selecting and defining product characteristics (nevertheless he is aware of possible temporal and financial restrictions). In this respect he reproduces a working method described by (Cuthbert79):

1. By means of questionnaires and/or interviews, inquiries can be made about the ways in which consumers actually use a product. It is important that this consumer research is representative of all users.
2. Product characteristics have to be identified that are of importance to consumers. If the population of users consists of separate groups with different usage habits, it is necessary to identify the essential product characteristics for every group.
3. It should be established how the population of users appreciates the product characteristics. According to Cuthbert, the best method to obtain this information is to combine specific product characteristics into more general, complex ones, like price, lifetime, performance, expenses of use and safety, and to have them evaluated by consumers as to their relative importance. Again, it is important to consider the degree of homogeneity of the users.

Developing measuring methods

As to the third step, developing measuring methods, (Willenborg85) distinguishes two types of measuring method:

1. Standard methods of measuring performance (SMMPs):

   Test methods that lead to results having a clear relationship to the performance of a product in practical use and that are to be used as a basis for information to consumers about the performance characteristics of the product.

   These methods are available for a limited number of products and they result from international activities on standardization (HellmanTuiter83) .
2. Methods developed by consumer associations:

   A much more elaborate set of measuring methods has been developed and applied by various consumer associations and collected (more than 800) by the International Organization of Consumer Unions (IOCU).

(Siderius89) stresses the importance of choosing the right measuring level and discusses the four most important levels:

1. Nominal level (presence/absence of a product characteristic);
2. Ordinal level (e.g. ordering in terms of high, middle, low);
3. Interval level (absolute numbers, averages);
4. Ratio level (relative numbers: relation between two amounts).

According to (Siderius89), it is indispensable to measure at least at interval level to guarantee the independence of the measuring method with respect to the other phases of the test procedure.

Performing measurements

The fourth stage, performing the measurements, produces the test results concerning the measurable quantities of the product characteristics.

Analysis of measuring results

During the fifth stage, the test results are interpreted as data forming part of a real user situation.

Interpreting and evaluating data

The sixth stage coincides to some degree with the previous one, but has a more subjective aspect: the test results are translated into plus and minus signs. (Willenborg85) does not specify how this can be done, but he stresses the importance of basing the judgements on data collected with the help of consumer research. Other variables he considers important for the interpretation of the test results are norms concerning product quality minimally wished for, average product quality and variation of the product quality.

The plus and minus signs serve as a basis for determining a global test result per product. The individual characteristics can have different relative weights (weighting should also be based on the results of consumer research). After having compared the total test results of the various products, the consumer organizations select the best buy.

Critical remarks to be made concerning the reliability of this weighting procedure include:

• Consumers may weight the importance of different product characteristics differently from consumer associations;
• Individual consumers may perform weightings differently;
• Specimens of the same product can have different functionalities resulting from product variation;
• Weighting and best buy recommendations are to a certain degree subjective.

Translating test results into product information

Comparative product tests can be described roughly as consisting of two successive phases: a testing phase and an information phase (Siderius89) . The second phase consists of interpreting the test results as product information and therefore coincides with the last step of the test procedure discussed in paragraph Interpreting and evaluating data.

The main activity during this second phase can be described as evaluative transformation: the test results at interval or ratio level are translated into data situated at the ordinal level; the ordered data are then evaluated in terms of, for instance, the annotation satisfactory, almost satisfactory or unsatisfactory. According to (Siderius89) the consumer will assimilate data more easily when they are presented in a certain arrangement expressing a value judgement.

This transformation implies applying to the test results firstly the criteria used for evaluating the products, secondly their ultimate values and thirdly the weighting factors (see paragraph Interpreting and evaluating data). These three variables are set prior to the actual product testing. The procedure of setting the criteria, ultimate values and weighting factors should not only refer to the product's functionality, but also to the consumer. As for the criteria, these are mostly derived from the functional analysis of the product. However, determination of the ultimate values and weighting factors often requires additional research of a technical or sociological nature.

Some basic requirements of product information

The following criteria can be applied to judging the product information resulting from comparative product tests (Willenborg85). The information has to:

• Be complete, that is, refer to all essential product characteristics, positive and negative, that are important when evaluating different products;
• Be factual, that is, relate to product characteristics referring to technical and economic functions that can be tested objectively;
• Enable the consumer to compare various products as to their essential characteristics and to make their choice;
• Be standardized to simplify the comparison of various products: when evaluating product characteristics, the same criteria and measuring methods have to be used for every single product and the product information has to be presented unambiguously;
• Refer, if possible, to the entire collection of products belonging to one product type available on the market;
• Be topical;
• Be presented in a structured way for easy assimilation by the reader and effortless application to the evaluation of the products.

The information phase is completed when the product information has been presented (e.g. in tables) and distributed (e.g. monthly magazines). As for the presentation of the information, (Box79) makes the following suggestions:

• Mark product attributes (in tables) that scarcely differ among products or that have little importance as to the quality of the product;
• Put the attributes (in tables) in rows instead of the brands, to simplify the comparing activity;
• Add data on nearby service facilities, product lifetime, etc.