next previous contents
Next: Comments on the grammar Up: Second round of testing Previous: Evaluation of the grammar

Evaluation of the grammar checker wrt user requirements

Above, we tested some of the quality characteristics which are part of the coverage claimed by the producers in their documentation. However, as mentioned earlier, an evaluation has to take its point of departure in actual user requirements. As it is a big task in itself to establish such user requirements, we chose to use the list provided by the producer and to supplement it by requirements of a specific user group. The only indication of intended users of E1 is a small paragraph in the documentation stating that professionals as well as students may benefit from using the grammar checker. We have chosen to focus on second language students.

User group and error types

The second language students whose requirements we describe, are Danish students aged 16-19 years having been taught English in school for 7-10 years.

Below are the ten most frequent error types for the user group according to (Faerch84). The list is not ordered, but the error types mentioned cover about 80 of the total amount of errors, so together they constitute the most important part.

  1. concord (subject/verb concord, usually in sentences starting with 'there', uncountable nouns, indefinite pronouns, or the verb is divided from the subject noun)
  2. articles (usually an overuse of articles)
  3. predicate (very heterogenous group, often direct transfer from Danish or verbal complementation errors)
  4. verb phrase (errors in tense, aspect, auxiliary, modal verbs, conjugation errors)
  5. tense shifts
  6. noun phrase (errors of nominalization of adjectives or numerals)
  7. word order (e.g. placement of adverbs)
  8. adverbial phrase (structure problems)
  9. aspect (mostly progressive aspect problems)
  10. number (no marking of distributive plural, conjugation of uncountables)

Some of these problems are specific to Danish writers, some specific to second language writers of English in general, and some belong to the more general area of errors of writers of English. As could be hoped, some of these error types are already part of the list established by the grammar checker provider. But there are also types which were not explicitly mentioned. Below we have chosen to test two of these, i.e. Tense, Aspect and Modality and Collectives.

Results

Tense, Aspect, Modality:
Here we are concerned with errors in the verb phrase resulting from incorrect use of tense, aspect or modality. The checker got a high recall score, and its major problem is incorrect flagging of uses of auxiliary contractions:

She's been seen.

He's been succeeding.

The reason for the lower precision score is that there seems to be a gap in its coverage of auxiliaries. Incorrect sentences containing 'have' as an auxiliary are typically not rejected:

She used to have succeeding.

She will have succeeding.

Collectives:
The problem here is that singular count nouns may have either singular or plural concord. This specific subtype of subject/verb agreement was not covered by the checker, and it falsely flagged the sentence with the plural verb:

The committee succeeds.

The committee succeed.


next up previous contents
Next: Comments on the grammar Up: Second round of testing Previous: Evaluation of the grammar

ceditor@tnos.ilc.pi.cnr.it