Even among the broad software engineering community, there is little consensus concerning the exact nature of tests that are appropriate for software evaluation. Much has been said on the importance of software quality assurance from the viewpoint of project management and developers. Little has been presented as seen from the viewpoint of the end-user who so far has occupied a rather passive role in software development and evaluation. However, whereas software testing until recently was the sole domain of software developers, a gradual change in the underlying philosophy can now be perceived, giving the user a higher priority in both the overall software development process and as a customer. The fact that software developers are not necessarily the best choice when it comes to testing their own programming product is gaining general acknowledgement (Thaller94, 61,84,90), cf. also (Hausen82, 119).
Moreover, a new philosophy has emerged concerning the goals of testing: whereas so far the principal motivation for applying software tests was ``... to look actively for problems so that these may be corrected thus improving the software'' (Bukowski87, 369), cf. also (Sneed87, 10.3-2), the evaluation and comparison of existing software products is gaining more and more importance. This is due to the fact that the PC market is overloaded with competing software packages for different application areas and users are at a loss when it comes to deciding which of the products suits their purpose best. The type of information which is needed in this situation is very different from what existing testing exercises can provide. Given this rather new user-centered development philosophy, it is obvious that there is a lack of methodologies for user-centered evaluation and testing. This is even more true for the NLP area which has so far produced only a relatively small number of operable systems.
This report strives to develop a user-oriented testing framework which is based on general software engineering (SE) principles, considers development-oriented testing expertise of both glass box and black box testing, and, last but not least, applies and further develops testing approaches to the needs of user-oriented assessment, particularly for NLP applications. Despite the great variance in terminology and the differences of the actual testing environments, common goals and procedures in testing will be identified, considering elementary aspects of testing such as the objectives behind testing, the persons involved in testing, the instruments used and the testing procedures. Survey of Glass and Black Box Testing Techniques presents an overview of glass box and black box testing techniques as they are practised by the SE community. Test Types presents a new model of test types which applies SE testing techniques to user-oriented testing and Survey of Instruments produces an overview of the instruments that are generally used for testing, their merits and drawbacks.