next previous contents
Next: Evaluation of the grammar Up: Second round of testing Previous: Method

Evaluation of the grammar checker wrt stated requirements

Here the set of errors on which the evaluation should be based is the set of errors that the system states it detects. These errors are listed here:

To cover the total amount of grammatical problems mentioned above would be very time consuming, and as the usefulness of our framework, rather than the actual performance of the grammar checker, is our main concern, only some of the problems are covered in this study.

It should also be mentioned that we have not had access to already existing test suites for this testing. Consequently, the test material we have used is far from being sufficient for a real evaluation. So again, the testing below is meant to show the feasibility of the methodology, not to be an example of a full scale testing.

In the testing, we have used the methodology outlined above, i.e. provided sentences in pairs, a wrong one and a correct one.

Results

Adjectives/adverbs:
E1 did not perform very well on errors of adverbs and adjective usage, and the distinction between the use of these two parts of speech.

For example there are certain adjectives that can only function as either premodifiers or postmodifiers but not both. E1 did not react to any of these. Below are two examples of sentences with an adjective that can only be premodifying or postmodifying, respectively.

The fool is utter.

The afraid man.

As the methodology indicates, the phenomena were also checked with the corresponding correct sentences below. These were also considered correct by the grammar checker:

He is an utter fool.

The man is afraid.

The checker also accepted adjectives which were incorrectly used as adverbs, so for example the following ungrammatical sentence was accepted:

He spoke brief and frank.

The checker accepted ill-formed comparatives, such as the following:

The house is beautifuller than I thought.

The mouse is more below the table.

In the two examples above we actually have two problems of ill-formed comparatives, (i) a problem of comparative of an adjective that can have an comparative form, but with more not by inflection, and (ii) there is an attempt to form a comparative of a preposition which is not possible.

Articles:
E1 produced some surprising results on missing articles.

The checker sometimes caught a sentence with a missing article, sometimes it did not. Thus, the first of the following examples was not caught by the checker but the second was.

He was brave man.

He was good boy.

Clauses:
In general E1 handles clauses satisfactorily, with some exceptions where it falsely flags correct sentences as errors for example in some adverbial constructions such as;

As was natural, he married her.

E1 flags this as an error. The error message states that 'was' seems to be lacking a subject.

E1 also falsely flags some appositive clauses as incorrect, e.g.

A message that he would be late arrived by special delivery.

In this case the error message is that the sentence does not seem to be complete.

Prepositions:
The correct choice of prepositions is a well-known problem for second language writers of English -- or any other second language for that matter. Prepositions also posed a problem for the grammar checker.

Most of the tested sentences with wrong prepositions were not flagged by the checker. This was all the more surprising since most of the errors were constructed around the fact that certain verbs subcategorise for only one or a few prepositions. So for example,

The package consisted with many things.

was not flagged as an error although consist subcategorises for the prepositions in or of. Similarly, the following sentence was also accepted by the checker,

He pays attention on it.

although the preposition in this phrasal verb should always be to.

However, some of the errors are not possible for the grammar checker to correct without semantic knowledge, and these should, therefore, not be considered to be errors in this connection, e.g.

He laughed with my words.

is semantically ill-formed but this is not due to any subcategorisation restrictions since laugh does not subcategorise for a prepositional phrase complement. So the following sentence

He laughed with the others.

is perfectly acceptable.

Subject/verb disagreement:
The problems relating to concord are primarily subject-verb concord in sentences starting with there, uncountable nouns, indefinite pronouns, or where the verb is divided from the subject noun. The checker catches some of these errors but not all of them as can be seen in the following examples:

The police has never found his murderer.

This sentence was caught by E1 (police is a plural noun). The next two sentences, however, were not.

There are violence in the school.

Some maintains that this is correct.

Comments

As can be seen, it is maybe time consuming, but otherwise fairly easy to test a grammar checker according to its specifications.


next up previous contents
Next: Evaluation of the grammar Up: Second round of testing Previous: Method

ceditor@tnos.ilc.pi.cnr.it