next previous contents
Next: Evaluation of the grammar Up: Second round of testing Previous: Method

Evaluation of the grammar checker wrt stated requirements

Here the set of errors on which the evaluation should be based is the set of errors that the system states it detects. These errors are listed here:

To cover the total amount of grammatical problems mentioned above would be very time consuming, and as the usefulness of our framework, rather than the actual performance of the grammar checker, is our main concern, only some of the problems are covered in this study.

Results

NP Coordination:
E1 got a high precision score for NP Coordination, and was able to detect 90 of errors relating to this phenomenon. Below are two examples of errors detected by the checker. In the first example, the checker correctly objects to a coordination of two different cases, and in the second example it correctly objects to the combination of 'neither' and 'and'.,

He sees I and them.

Neither he or he

As the methodology indicates, the phenomena were also checked with the corresponding correct sentences below. These were considered correct by the grammar checker:

He sees me and them.

Neither he nor he

However, there was a tendency that it could not detect errors in conjunction combinations, and e.g. the example below was accepted:

Neither he both he

Either he both he

Also, it wrongly rejected coordination of accusatives in subject position, which according to TSNLP should be accepted:

Her and him succeed.

Him and me succeed.

NP Complementation:
E1 performed poorly wrt. NP Complementation. True enough it got a perfect recall score, i.e. 100, but the precision score was 0, and consequently it must be assumed that it does not cover this phenomenon. The following examples illustrate incorrect complementation patterns accepted by the checker:

The donation of the building of the manager

The donation of the building from the manager

Relative Clauses:
Wrt. recall E1 handles relative clauses well, with some exceptions where it falsely flags correct sentences as errors, e.g. in constructions such as:

The firms, after whom he looks, succeed.

E1 flags this as an error, because it incorrectly analyses 'whom' as the object of 'looks' which takes a prepositional object. Also, because the relative clause occurs between subject and verb of the matrix clause, the checker incorrectly flags sentences like:

The committees, the firm of whom succeeds, come.

In this case the error message is that singular 'firm' does not take the plural verb 'come'.

In terms of accepting incorrect relative clauses, the major problems were that it did not reject 'that' after prepositions, and did not distinguish between 'who' and 'which':

The manager, around that she revolves, succeeds.

The manager, at which he aims, succeeds.

Verb Complementation:
The correct choice of verb complementation patterns, and especially choice of preposition for prepositional objects, is a well-known problem for second language writers of English -- or any other second language for that matter.

The checker did better wrt. recall than the score would indicate. Most of the sentences wrongly rejected, were rejected because it had problems with proper names ending in 's'.

Abrams fails to hire Browne.

This sentence was flagged as an error, because the checker analysed 'Abrams' as a plural noun.

However, the checker failed to reject 40 of wrong complementation patterns, e.g. a wrong preposition or wrong syntactic category:

She abstains of it.

She considers on her a competitor.

Also, the checker had problems with the expletives 'there' and 'it', i.e. in deciding what verbs allow expletive subjects:.

There manages to be a bookcase in the office.

They rain.

Subject/verb agreement:
The checker does very well on subject-verb agreement, with both high precision and recall scores. The types of construction it wrongly rejects involve pronouns and constructions where the subject and verb are separated by other sentence elements:

Either of them are new.

Patience , friendship , sacrifice - everything she needs - are important.

Incorrect constructions wrongly accepted contained partitives involving pronouns:

Half of them is new.


next up previous contents
Next: Evaluation of the grammar Up: Second round of testing Previous: Method

ceditor@tnos.ilc.pi.cnr.it