next previous contents
Next: Evaluating local terminology management Up: Support activities Previous: The computing correspondents

Evaluating translator's work bench systems

Three commercially available translators' work bench systems (TWBs) were under evaluation at the time of the interviews, with translators from the language units directly involved. For most, the main feature of the TWB is perceived to be the translation memory function, although one translator reported having used the term recognizer of one system with very satisfactory results: in the text he was translating there were three terms where he could not find help even from a specialist he consulted. With the term finder he got the terms and context very quickly and from a source he could trust, the Bulletin. Another reported using a TWB in order to create glossaries for the documents he was working on. The document was sent from the PC to the server which preprocessed the document and returned it together with the glossaries. Then, whilst he was working, the system proposed any solutions found during the preprocessing.

Testing the translation memory function relies on having something in the memory. One of those doing the evaluation had created a number of small memories, mainly representing text which had already been rationalised, another had simply taken the originals and translations of a repetitive text that arrives for translation every month and used those to stock the memory. Yet another had taken previous texts and their translations, without trying to select text that was thought specially suitable for machine treatment, for example because it was repetitive or followed a standardized format. The importance of what is in the memory was emphasized by the person who reported a very positive experience with a very long text which came back to him for modification: with the old version in the translation memory he had been able to complete the job in an afternoon, whilst his unfortunate colleague without a translation memory took a week for the same job. Several of those involved in evaluating translation memories said that it would be useful to be able to edit the translation memory, which is not possible with some commercial systems.

Most of the comments offered related to the interfaces of the system, rather than directly to functionality or efficiency, although several people said that they strongly disliked having to wait whilst processing happened. One person talked about what he called the "motorway syndrome": moving at all feels better than sitting waiting, so even though one might be aware that sitting in the queue is likely to get one to the destination quicker than leaving the motorway and tackling country roads, one still opts for the country roads. In a rather similar way, one is very irritated if EURODICAUTOM offers twenty-five solutions and it takes time to choose between them, even if doing so actually reduces the total time required to find a solution, or if a translation memory takes a perceptible time to come up with a proposal, even if the proposal fits exactly.

It is important to notice that two interfaces are involved, one for the user of the TWB, one for the management of linguistic resources such as the translation memory. Often the user interface was thought more satisfactory than the resources management interface.

The number of comments relating to the screen indicate how important presentation is to most people. There was much discussion of whether translation and original should appear on the screen together, whether the translator should have to over-type the original, whether it was satisfactory to have only one sentence at a time appearing in the translation window and so on. Several people remarked that it was important for the user to be able to organize the screen in the way he wanted. The number and size of windows was important, as was the ability to work without being distracted. A critical feature was that the focus of the screen should be on the translation being done. It was agreed too that it should be very easy for the user to use the system: for example it should be integrated with the text-processor the user was familiar with rather than involve use of a special system.

There was common agreement too that it would be useful if the unit of treatment were shorter than a whole sentence.

There was much less general agreement on other features. There were those who only wanted text and translation to go into memory when the translator decided to put it there, others who wanted automatic recording so that colleagues could have almost immediate access to what had been done. There were those who wanted search for similar texts and their translations only to be done when the translator requested, others who wanted it done automatically all the time. Many of these questions relate to the general issue of validation in an environment where translation memories are to be shared, an issue which has already been discussed in an earlier section. Several people suggested that it would be useful to work in batch mode rather than interactively.

Reactions to using an alignment facility varied enormously, and seemed to have a great deal to do with the particularities of specific products. Therefore, apart from reporting the obvious and general desire that alignments when presented should be reliable, this question will not be further discussed here.

The stage of the evaluation described here is now over, and final reports on each of the three products were presented at the end of June 1995. The evaluation will now continue in a real production environment in the framework of the translation workshop and modernisation network already mentioned, both of which are due to start in the autumn of 1995. It is felt that, although the first round of evaluation did provide a very good overview of what was available and the current state of the art, it did not allow for what is sometimes called in the evaluation literature ``scenario testing'', testing of a product by putting it into use to solve real problems in a realistic work environment.

The workshop will be staffed by volunteers from the translation units and from the horizontal services, who, although they will remain attached formally to their own groups, will be temporarily relocated in thirty or so offices in Brussels. The workshop will acquire an expertise in using computerized tools to help in the task of translating documents identified as suitable material for such treatment. Liaison with the Heads of Department, Central Planning and the translation units will be done conjointly by the Director of AGL and the Head of Department A. When the staff of the workshop return to their own group, the savoir faire they have acquired will be transmitted to their colleagues.

The network is a parallel operation in Luxembourg, whereby the participants stay in their own Units, and a loose organisational structure coordinated by a member of the AGL staff links them together. With the aid of the coordinator, suitable documents are identified and are treated with new tools, primarily pre-processing tools and translation memories. The aim here is to investigate how well new tools can be integrated into existing structures and the need to create new structures avoided.

next up previous contents
Next: Evaluating local terminology management Up: Support activities Previous: The computing correspondents