next previous contents
Next: Results of tests on Up: Introduction Previous: Testing on the UBS

Testing the ad hoc sentences

A few sentences were especially created to test the checker, with the aim of checking the coverage of the grammar and ascertaining whether the program would false flag some intentional traps. Focus has been on NPs, maintaining a certain equivalence between the English and French testing, even if some changes had to be made to account for differences between the two languages.

The list is not intended to be exhaustive, but can definitely be a starting point for an evaluation of grammar checkers.

Grammar checker for English

The testing of ad hoc sentences has been concentrated on a specific class of Noun Phrase errors, i.e. determiner-noun disagreement (see section Requirements). Some initial testing has also been performed in relation to phonetic adjacency agreement and the genitive.

It has been possible to make some comparisons between different versions of the same program. Many erroneous results found in the older version of the grammar checker were no longer present in the newer one, e.g. concerning mass nouns.

As proposed in section Composition and structuring of test material, a set of correct inputs, a set of erroneous inputs and a set of misleading ones have been considered. These inputs are defined such that the same construction may appear both in simple realisations and in more complex ones. In some cases this has turned out to be very useful. For example, let us consider the noun phrase:

All this work.

The input has been considered correct. But the input:

All this good work

was considered wrong. A third combination:

All this interesting work

was correct.

Most of the erroneous results given by the program were due to confusion between nouns and verbs, even in the presence of verbs. Usually, correct sentences were recognised as such, but, in many incorrect sentences, the error reported by the grammar checker did not match the supposed one.

Another weakness presented in E1 concerns ordering problems. The following erroneous sentences were in fact considered correct:

Many these painters.

Some fresh other milk.

Grammar checker for French

In general, F1 could spot straightforward errors of agreement (noun-article or noun-adjective subject-verb agreement and agreement of past participles). In some cases, a complex subject NP (e.g. the kittens near the door which is shut are ...) would make the checker interpret the wrong noun as the subject, flagging for wrong agreement.

The checker is capable of spotting typographical errors, e.g. mis-matched brackets, spaces before a full-stop, non-inclusion of a question mark within inverted quotation marks, etc.

The test sentences focus on a few key points. Some are aimed at the agreement of tout (`all'), which can be either an adjective or an adverb, and will consequently change its behaviour in agreement. Testing of the list of NPs showed that straightforward errors would be caught, but F1 would overlook errors in a slightly more complicated construction.

The test sentences were kept fairly simple and short. After having used F1 on the UBS texts, there was no need to check it further for side effects due to long and convoluted sentences.

One of the tests was to see if the program would recognise non-existent verb forms: in French, some verbs are never used in some tenses, which is difficult to keep in mind when writing. F1 failed to spot that the verb parfaire (`to perfect, put the finishing touches to') is only used in compound tenses or in the infinitive, and thus did not flag the sentence:

Albert parfaisait la statue que Robert avait laissée incomplète
(``Albert was giving the finishing touch to the statue that Robert had left incomplete'').

In another case, for the verb renaıtre (`to be reborn'), rarely used in compound tenses, the checker points out, erroneously, that the user should have written avoir (`to have') as the auxiliary.

Some sentences were aimed at testing, somewhat more exhaustively, agreement for words that can be both an adjective and an adverb. Tout (`all') is among those words, and a few sentences were written to represent most of the possible constructions in which tout can appear. Here, the checker seems to base its recognition of tout as an adjective or adverb simply on the basis of the word's proximity to a noun or a verb. For instance, in the sentence

Ils sont tout partis
(``They have all left'')

where tout refers to the pronoun and should therefore be in the plural form tous, the checker does not spot the error. Conversely, it marks as wrong the correct sentence

Ils se sonts tous brûlé les mains
(``They all got their hands burned'')

pointing out that tout used as an adverb should not be in the plural form. Since these kinds of error are fairly frequent even for French mother tongue speakers, the grammar checker does not help as much as it could and in fact confuses the user even further.

As a side effect of testing for adjective-adverb usage, the following false flagging was detected. The sentence

Ils sont tous partis
(``They have all left'')

was tested on F1. The checker correctly assumes that tous is an adjective referring to the subject, but also assumes incorrectly that partis is not the main verb (past participle of partir (`to leave')), but the noun parti (meaning `(political) party'. This is evident because it prompts the user with a message warning for possible misuse of a word, in this case as a false friend or homonym. (The message reads more or less: Do not confuse ``party'': union of interests, with ``part'': element of a whole).

Another common source of error is homophones. These are difficult to catch, since most of the time the words concerned share the same syntactic category and it is only at the semantic level that they differ. However, tout is pronounced like toux (`cough'). In a sentence like

l'enfant était au lit avec la tout
(intended:
l'enfant était au lit avec la toux
`the child was in bed with a cough')

the checker should have been able to spot the error. Tout can be used as a noun, in which case its gender is masculine, but in the sentence above the article is feminine.

When working on the list of NPs, F1 could not correctly recognise recognition of vowels/consonants that guide the choice of determiners. As in English, where there is a difference in using a/an according (grossly) to the first letter of the following word, in French some determiners have a double form and the choice of which to use is dependent on the following phonetic environment. For instance, ce (`this') should be written cet when followed by a vowel or an unpronounced h. The checker flagged correctly

ce homme
(``this man'')

but

ce énorme travail
(``this enormous work'')

was allowed to pass un-noticed.

However, F1 was accurate in spotting the correct usage of the article le/l' or la/l', where the vowel of the article is dropped and the apostrophe appears before a word beginning with a vowel or an unpronounced h.


next up previous contents
Next: Results of tests on Up: Introduction Previous: Testing on the UBS

ceditor@tnos.ilc.pi.cnr.it