This section is based on the idea that the ultimate goal for evaluation of language engineering would be to have some automated procedure into which new products are fed which are then evaluated and compared with other products. We call the device that does this a parameterisable test bed (PTB).
A PTB would have several components, to be described below. In our opinion, a major goal for future EAGLES-related work on evaluation would be to collect such components and work on the design and use of a PTB.
In this section, we will first formally define some key concepts in evaluation and then briefly describe the components of a PTB.