The aim of validation workpackage was to test each element of DicoPro tool not only in its strictly technical aspect (good implementation of functionalities) but also in its practical aspect, i.e. its compliance with the needs of the final user (user friendliness, relevance, speed of the tool). This document presents the different cycles of tests that were carried out; it shows the difficulties that were met during testing; it reports the remaining bugs and proposes some improvements for the tool. As a conclusion, the point of view of the potential user reveals the eventual interest of DicoPro tool.

Table of Contents

1. Introduction

1.1 Goals of Validation Workpackage

1.2 Background to WP10 report

1.3 DicoPro specificity and WP10 report structure

2.1 Drawing up of the document

2.2 Starting tests without validation agreement

2.3 Signing of the final validation agreement

3.1 Partners involved in testing and testing environment

3.2 Release of testing versions and rounds of tests

3.3 Conclusion on the final version

4.1 DicoPro Server available

4.2 Installation problems

5.1 Dictionaries available

5.2 Data display and structure

5.3 Relevance of queries

6.1 Organizing difficulties

6.2 Validation limitations

6.3 Perspective for DicoPro tool

DicoPro interface is independant from a specific platform and can run regardless of the type of Operating System. The widest the range of client material, the better it was for a complete validation of DicoPro tool. Partners hardware and environment were indeed diverse.

• DicoPro Client was tested at L & H Mendez by two persons on Toshiba Satellite Pro 460 CDT, Window 95, 64 MB RAM and PC Pentium Desktop, Windows 95, 32 MB RAM.
Testers qualifications: translator/terminologist and director R&D.

• DicoPro Client was tested at MTL by four persons on networked Solutions PC (233 MHz), Windows NT 4.0.
Testers qualifications: Language engineering consultant, plus three translators.

• DicoPro client was tested at Hachette Livre by two persons on two Macintosh G3 (300 MHz and 350 MHz) Mac OS 8.1. and 8.6. The Client software was installed on PC Windows 98 but eventually this computer could not be used for testing.
Testers qualifications: lexicographer and multimedia projects manager.

• DicoPro Client should have been tested at Harper-Collins by one person on Sun Ultra-5, Solaris 2.5.
Tester qualifications: analyst-programmer, typesetter.

• DicoPro Client should have been tested at Xynos by on Pentium 200 MMX 96 MB RAM, Windows 98.
Tester qualifications: technical manager and translators.

ISSCO team was working on a Client computer, on Unix (Sun/Solaris 2.6).

Three rounds of tests were necessary to get to version 4, considered as the final version of the project and tested in a fourth round.

On May 14 1999, ISSCO asked WP10 partners to load Java environment in a version compatible with the software (Java 1.2). A problem appeared with Macintosh and OS/2, because this version of Java was not available for such environment: Hachette Livre and LIM could not participate to the first round of testing.

L & H Mendez and MTL could load and install DicoPro_01 Client easily, but the connection to ISSCO server turned out to be more complex. L & H Mendez could connect to ISSCO on May 31, MTL on June 17. Both L & H Mendez and MTL experienced problems with the display of the dictionaries, which were later resolved.

Harper-Collins and Xynos did not participate to the first round of tests.

Bug reports put emphasis on cosmetic problems in the interface, typographical errors in the layout of the dictionaries entries (fonts –especially IPA fonts–, size of characters, too many spaces between words).

A design problem was raised during this round of tests: some arrows ("right" and "left") enable the display –by alphabetical order of the concerned dictionary– of the entries in the right part of the screen, whereas the list of headwords displayed in the left part of the screen can be the result of a search without relation to the entry displayed. This point that seemed immediately illogical to the testers have not been corrected yet. Will the professional user be bothered by this?

DicoPro_02 Client showed very few changes compared to the previous version, exept for its compatibility with Java 1.1. Some of the interface problems were fixed. In the new version of dictionaries, the bad display of accented letters on Macintosh was solved. Nevertheless, until today, IPA fonts display is still a problem for the Macintosh. It seems that only a planned new version of Java for Macintosh could solve the problem, but this version is still not available.

DicoPro_02 was installed without problems at L & H Mendez, MTL and Hachette Livre. The server software remained at ISSCO.

Harper-Collins and Xynos did not participate to the second round of testing.

Bug reports put emphasis on queries: some queries on words included in the dictionaries got no answer. Moreover, the starting of DicoPro was considered to be quite time consuming.

In version 02, translators at MTL appreciated the possibility of automatic display of the first word in a list of results. In fact, nine times out of ten these translators felt they would make queries on exact words, instead of pattern or suffix.

Problems in the typographical layout of the entries were focussed on, but at that time ISSCO’s urgency was the development of the server with a new version of the client software. Working on the dictionary data was postponed later. But the layout as it stood then was quite worrying since the logical structure of entries was not respected.

DicoPro_03 Client was no longer compatible with the server version installed at ISSCO. It had to be tested in an intranet environment since DicoPro_03 server was made available on July 26 1999.

Because WP10 partners had some difficulties to install their own server, ISSCO allowed testers to connect to the University server during the full testing period.

The new round of tests was performed by MTL and Hachette Livre. Xynos did not participate and both Haper-Collins and L&H Mendez tried but did not succeed in getting a working installation.

Two essential functions of the tool were still not available in this version: advanced and multiple searches. Typographical layout of the entries was not changed. Changes in the program were above all designed to make the tool more performant.

Testers spent a huge amount of time installing the Server and consequently had much less time to test the new version of the Client. Nevertheless, remarks were made about multiple windows, slowness of the tool and many other points (cf. http://www.issco.unige.ch/projects/dicopro/mail/dicopro-test).

The main problems raised in DicoPro_03 Client were:

• the layout of the entries did not comply with their lexical structure

• bad results of queries on words with accented characters (with suffix and pattern queries especially).

DicoPro_04 was tested in a systematic way by filling out an exhaustive grid for all the projected functionalities. The exhaustive test carried out by L & H Mendez, MTL, Hachette Livre and Xynos can be consulted in annex B (p. 19). The included remarks shall be taken into account so to implement a commercial version of the product.

MTL, Hachette Livre and Xynos were able to test DicoPro in an Intranet environment, L & H Mendez in Internet through ISSCO’s server. Harper-Collins tested in both Intranet and through ISSCO’s server.

Nearly all functions are correctly implemented. But there are still some problems:

- searches on words with accented letters are impossible on Macintosh

- index that appear at the top of some entries do not work

- lack of stability of the open dictionaries after multiple or advanced searches

- problems with personal bookmarks

- necessary fit of the functions "save" and "cancel" in the preferences

- difficulties or ambiguity when using various frames in different dictionaries. This function should be considered again and certainly simplified.

The speed of the tool is correct in an Intranet context, however it seems to be still an issue at Harper-Collins where the Client and the Server are running on the same machine. It will be necessary to confirm that searches are always time efficient (above all when patterns or suffix searches are made), when all the corpora of dictionaries are on line.

The interface of "dictionaries consultation" is simple and effective. The icons could be more homogeneous on a design point of view and some of them could be more representative of their functions. The contextual inactivity of some icons should be clearer.

The user documentation is adapted to the version and was very clear. Yet, it is still not always available on line and, above all, it is not contextual.

The documentation for the installation of the tool (with its Java environment and fonts) should be checked before any industrial distribution.

The first tests on DicoPro Client were carried out using a connection to the ISSCO server. For security, each computer assigned to validation at the partners firms was registered with its IP number at ISSCO. The difficulties that were encounterd with connecting to ISSCO server delayed the starting of the tests. They did not seem that serious because they were due to a connection problem to internet and because the goal of the validation step was to prove the efficiency of the program, i.e. the running of the Client and Server software on Intranet.

Expected installation platforms for servers:

- L & H Mendez. Hardware: PC Pentium II (128 MB RAM), Windows NT 4. Qualifications of the person who have tried to install the server: System engineer.

- MTL. Hardware: networked Solutions PC (233 Mhz), Windows NT 4. Qualifications of the person who have installed the server: Language engineering consultant and systems administrator.

- Hachette Livre. Hardware: Sun Sun SPARCStation 5, Solaris 2.7. Qualifications of the person who have installed the server: manager of Hachette intranet sites, Net engineer.

- Harper-Collins. Hardware: Sun Ultra-5, Solaris 2.5. Qualifications of the person who have installed the server: analyst-programmer, typesetter.

- Xynos. Hardware: Intel Pentium II, 350 MHz, 128MB of RAM, Windows NT 4.0, Service Pack 5. Qualifications of the person who have installed the server: technical manager.

On July 26 1999, the first version of DicoPro sever was available. It was called DicoPro_03 Server in order to bring it into line with the client software DicoPro_03 Client.

A second server DicoPro_04 was available on October 4 1999. It is compatible with DicoPro_04 Client.

WP10 partners, except Hachette Livre, met serious difficulties when installing and defining parameters for their server. These problems have been solved at the end of the project at MTL, Harper-Collins and Xynos, but not at L & H Mendez.

When Hachette Livre loaded and installed the server, the connection was successful but access to the dictionaries was denied because of a licence problem. The bug, soon resolved, was due to time difference between the partners countries!

Bug reports show the great efforts made by MTL, Xynos and L & H Mendez to identify and explain their problems and those of ISSCO to answer all questions as soon as possible. Problems were diverse: installation of the different modules, definition of parameters, connection, etc. MTL succeeded their first connection client-server on intranet on September 24 1999, Harper-Collins on October 6 1999, Xynos on October 7 1999.

A major difficulty in the validation step was the connection to the Client and Server software via Intranet: it seems essential to verify that WP10 testing cycles have really shown all the problems related to the implementation of DicoPro in a firm and have enabled to solve them.

For the main part of the validation cycle, tests have been performed on the data of letter A in three dictionaries only:

- Harper-Collins English-Italian

- Oxford-Hachette English-French

- Hachette-Oxford French-English

Other dictionaries have been available at the end of the validation cycle.

Bug reports have put emphasis on the typographical layout that did not comply with the data structure. From August 16 until September 23 1999, ISSCO worked on a new processing of the dictionary data and submitted a few samples to Hachette Livre. The problem turned out to be more complex than thirst thought: a post-processing would have had to be followed by a detailed checking and the "cleaning up" of lexical entries. There was no more time to define such a new processing step that, besides, was to be adapted according to each dictionary.

On September 24 1999, a new version of letter A for English-Italian, French-English and English-French dictionaries was available on ISSCO server.

Also on September 24 1999, letter A of Italian-English and monolingual French dictionaries was available on ISSCO server .

On September 26 1999, letter A of Spanish-English and English-Spanish dictionaries was available on ISSCO server.

These dictionaries were also available for partners to download to install on their own servers..

In this project, priority was not given to correcting the typographical layout of data. Obviously, this point would have deserved a huge amount of time, so that publishers and users could think about the desirable layout of an on-line dictionary.

A question emerged during the validation step: shall DicoPro make the translator work easier by a uniform presentation of all the dictionaries or shall each dictionary retain its own identity and that of its publisher?

Data as they were supplied to ISSCO presented structure and SGML tagging discrepancies from one publisher to another. Conversion of dictionaries from SGML to HTML was eventually more complex than it was thought, because SGML format did not comply systematically with the intimate and detailed structure of the entries, but sometimes only took into account the typographical layout wished by the lexicographer. Post-processing designed to disambiguate some lexical entities could not solve all the problems.

Please report to the Dictionary testing form filled out by L & H Mendez, MTL, Hachette Livre and Xynos in Annex D (p. 65) for precise remarks about the presentation of search results and the layout of entries.

Suggestions for a better readibility of the list of entries after a request and the layout of a dictionary entry:

- in a list of multi-word compounds, only the first word appears, so it is necessary to display the whole entry to see if the relevant compound word is really defined

- for homonyms, it would be helpfull if more information was given in the headwords list

- the number of words matching a search should appear at the bottom of the window for instance.

- the information could be more colour coded to separate the sense tags from the lexical items themselves, to emphasize compound words

- a use of full potentiality of the hyperlinks and cross-references

- avoid the italic font

- avoid the vast amount of blank space that makes the use of the scrolling bar necessary

- more performant links between elements in the left and right windos should be studied.

As the tool stands, the layout of entries remains too similar with the paper dictionaries layout. All the potentiality of the tool has not been used.

It is quite difficult to assess the relevance of the searches in a professional context since the tests were not performed in a real scale.

It seems that the tool is somewhat too powerful compared to the everyday needs of the translators and that searches on patterns and suffix are little used. It seems that the boolean logic of the advanced search is not particularly intuitive for the end user

A first difficulty in DicoPro validation planning came from the tool itself: a software and its functions, a user interface and ergonomics, then the layout of data that was to conform with a logical structure, were to be checked at the same time. If the first two points could be tested by everyone following a defined method, the third point came within the competence of publishers who had supplied the structured data. However, such a work required much more time and staff than publishers had.

The second stumbling block was the number of partners involved in WP10 (a natural tendency is to rely on the work of other partners), from the distance between them (it is quite difficult to follow a discussion via e-mail) and apparently from the different competences of the partners (diversity of qualifications is indeed precious but sometimes makes things more complex).

The third difficulty was obviously linked to the holidays period: holidays for partners involved in testing but also for IT personnel, whose assistance to the partners turned out to be crucial during the validation step.

Finally, changes of WP10 actors in ISSCO and Hachette Livre made much more complex the necessary continuity of a project like DicoPro. In addition, ISSCO had to move at the end of September when partners were behind with the schedule.

We cannot claim that exhaustive tests have been carried out as far as hardware and OS are concerned, but WP10 partners could demonstrate the principle of independance of DicoPro software from the platform since the Client could run in NT 4.0, Windows 95, Windows 98, Unix and MacOS environment and the Server could be installed on NT 4.0 and Unix platform.

The difficulties with having Client and Server running together on intranet (or Internet) show that tests should be carried on from a Net implementation perspective if this product were to be distributed in an industrial context. A systematic analysis of the different types of nets should enable a complete set of documentation to be drawn up that could be used in any technical situation and provide the opportunity to correct bugs if needed.

As said before, only letter A of three dictionaries were available during the testing period. The delay in signing the validation agreement meant that a full-scale testing of DicoPro, as it was wished by all the partners, could not be performed. This agreement is now signed and the real validation of lexical data could start in better conditions. It would be the next step of the project.

Problems linked to different installations (Client, Server, connection to ISSCO server, connection to intranet servers, fonts setting, etc.) required a lot of time to be solved and had a real impact on the users tests.

The fact that all the dictionaries were not available and that all the partners but one were not able to install DicoPro in their intranet system prevented WP10 partners from validating the tool in a professional context.

The final user should have validated the product in his/her standard work conditions. But due to technical difficulties and delay, L & H Mendez, MTL and Xynos staff could not work with DicoPro tool as it was expected.

As a conclusion, DicoPro tool still needs some corrections and improvements before beeing sold. Its use by language professionals must now be tested in a real professional context. The professional opinion will be crucial for the finalization of the product. Publishers will have to take into account the professional assessment to organize their dictionary data in an efficient layout.

Consequently, it is important to conclude this report with both MTL’s assessment (as a professional translator) and Hachette Livre proposition (as a dictionary publisher), which give a positive perspective for DicoPro. These conclusions are taken from the Dictionary testing forms (see annexe D p. 65)