Next:
Introduction
Up:
Previous:
Contents
Introduction
Contributors
Authors
Other contributors
Guide to the report
About this report
How to read it
Types of systems considered
Long term
Short term
The Framework Model
The ISO 9126 Standard
ISO 9126
Quality requirements definition
Evaluation preparation
Evaluation procedure
The EAGLES extensions to ISO 9126
Towards formalisation and automation
Key concepts in evaluation - a sketch for a formalisation
The evaluation function
Feature descriptions
Some useful terms
Candidates for standardisation
Parameterisable test bed
Parameterisable Test Bed
Parameters of objects
Parameters of users
Library
Testing
Assessment
Towards a PTB for language engineering
Applying the framework
Set of attributes
Requirements
Methods
Requirements analysis for language engineering evaluation
Requirements analysis and evaluation
Evaluation in the software life cycle
Scope of requirements statements
Functional and non-functional requirements
User profiling and requirements analysis
Issues for requirements analysis
Evaluation of existing products or systems
Reporting evaluations to the customer
The context
Evaluation of components and interactive systems
Evaluation of generic systems
Iteration and validity
Scope of evaluation
Knowledge acquisition for requirements analysis
Conclusion
Requirements analysis in language engineering evaluation
Domain-level requirements
Top level task description
Setup and variable elements
Quality requirements for top level
Transformation to reportable attributes
Measures
Attributes as measures
Validity of measures
Internal validity
External validity
Reliability of measures
Compositional measures
Typing of attributes
Binary values
Classificatory values
Comparative values
Numerical values
Metric values
Conclusion
Methods for system measurement
Test types
Scenario tests
Field tests
Laboratory tests
Systematic testing
Task-oriented tested
Menu-oriented testing
Benchmark testing
Systematic testing and adequacy evaluation
Feature inspection
Instruments
Testing instruments
Reporting instruments
Evaluation descriptions
Test problem reports
Result reports
Assessment reports
Test materials
Test sets
Test suites
Test collections
Conclusion
Related Work
Looking into the consumer report paradigm
Basic notions
Consumer and product quality
Consumer and information
Activities and aims of consumer organizations
Basic requirements of comparative product testing
A procedure for comparative product testing
Selection of product class and products
Selecting and defining the attributes
Developing measuring methods
Performing measurements
Analysis of measuring results
Interpreting and evaluating data
Translating test results into product information
node43.html
Some basic requirements of product information
Discussion (and evaluation) of the literature
Literature suggestions (extracted from SWOKA reports)
Ongoing projects
TEMAA - A testbed study of evaluation methodologies
COBALT
RENOS
TSNLP - test suites for NLP
Recent History
Some history
ALPAC
ARPA
The Hewlett Packard Laboratories Study
MUC-3
Conclusions and Directions for Future Work
ISO Terms and Guidelines
Quality characteristics and subcharacteristics
Functionality
Reliability
Usability
Efficiency
Maintainability
Portability
Subcharacteristics
Functionality
Reliability
Usability
Efficiency
Maintainability
Portability
Terminology
Guidelines
After-sales servicing
Evaluation of importance
Evaluation process
Maintenance of software products
Quality objectives
Quality system elements
Methods for System Measurement
Preface
Introduction to the testing of software
Survey of glass and black box testing techniques
Glass box testing
Static analysis techniques
Dynamic analysis techniques
Generation of test data in glass box tests
Black box testing
Black box testing - without user involvement
Black box testing - with user involvement
A user-oriented model of test types
Scenario tests
Types of scenario tests
Field tests
Laboratory tests
Types of metrics and results in scenario tests
Systematic testing
Task-oriented testing
Types of metrics and results in task-oriented testing
Menu-oriented testing
Types of metrics and results in menu-oriented testing
Benchmark testing
Types of metrics and results in benchmark tests
Feature inspection
Types of metrics and results in feature inspection
Survey of instruments
Testing instruments
Manual test instruments for user-oriented testing
Questionnaires
Checklists
Interviews
Observations
Think-aloud protocols
Automatic test instruments for user-oriented testing
Reporting instruments
Test descriptions
Test problem reports
Result reports
Assessment reports
Conclusion and outlook
Requirements Analysis for Linguistic Engineering Evaluation
Introduction
Requirements analysis and evaluation
Evaluation in the software life cycle
Scope of requirements statements
Functional and non-functional requirements
User profiling and requirements analysis
Issues for requirements analysis
Reporting evaluations to the customer
Context and validity in
LE
evaluation
Knowledge acquisition for requirements analysis
Introduction
KADS
Knowledge acquisition (
KA
) techniques
Requirements analysis in
LE
evaluation
Introduction
Overview
Problem domain requirements
Transformation to reportable attributes
Generic requirements models: adaptation and reuse
Conclusions
Evaluation of Writers' Aids
Introduction
Reviews
Notes from ``Intelligent text processing: A survey of the available products'' by Shona Douglas
Summary of method
The system centred taxonomy
The user centred taxonomy: A high-level specification of functionality
The test suite
Scoring the tests
Progress evaluation for
The Editor's Assistant
PC Magazine Laboratories' performance tests
Chandler's review of grammar checkers
Conclusions
Selection of grammar checkers to be used as test beds
Reasons for the selection
Focus of attention
Quality characteristics
Task model
Model of writer types
Taxonomy of systems
Model of end-user types
Model of unproofed text
Model of proofed text
A Taxonomy of grammar errors
Taxonomy of advice types
Relationships among parts of the model
Requirements
Functionality requirement - part I
The grammatical errors made by the writer
The specification made by the system
The correction of errors by the end-user
Functionality requirement - part II
Functionality requirement - part III
Discussion
Reportable attributes of writers' aids
Methods
Functionality requirement - part I
Functionality requirement - part II
Methods for combination of attributes
Composition and structuring of test material
Instantiations of the interpretational scheme
Evaluation of Translators' Aids
Translators' aids: user profiles
Dimensions of Translation
Relationships between different dimensions
Trends in dimensions
Product typology
Multilingual dictionaries
Multilingual thesauri
Terminology management databases
Translation memories
Miscellaneous
Featurization
Design and function of translation memory
Information contained
Off-line function: import
Import from raw format.
Import from native format.
Off-line function: analysis
Textual parsing.
Linguistic parsing.
Segmentation.
Alignment.
Term extraction.
Text statistics.
Off-line function: export
Control information.
Off-line functions: translation memory combinatorics
On-line functions
On-line function: retrieval
Exact match.
Fuzzy match.
On-line function: updating
On-line function: automatic translation
Automatic retrieval.
Automatic substitution.
Networking
Overview of feature checklists
Specification/inspection catalogues
Feature checklists of translators' tools
Further procedure - types and feature checklists
Evaluation procedure
General aspects of evaluation procedure
ISO on evaluation procedures
Quality requirement definition.
Evaluation preparation.
Evaluation procedure proper.
Desiderata for testing methods
Test types
Checklisting of features (= specification/inspection).
Scenario test (= users test the complete system in a realistic environment).
Benchmark testing (= systematic testing using test tools and materials).
Validation
Benchmarking translation memories
Usefulness
Online properties
Size (capacity of translation memory).
Speed (retrieval time).
Hit rate (number of matches).
Off-line properties
Text analysis methods.
Segmentation and alignment: success rate
Initial (raw) import into translation memory
Translation memory combinatorics
Export of translation memory
Specifying benchmarks
Suggestions for translation memory benchmarks
Translation memory: segmentation and retrieval.
Translation memory, import from external format.
Translation memory, learnability.
Translation memory, portability
Benchmarks suggested by Steenbakker's featurization
User Profiles
Dimensions of translation
Quantity of translated text
Quantity of translation work
Text types
Characteristics of original text
Languages involved
Translation quality
Dimensions of the translation organization
Type of translation organization.
Size of translation organization.
Nature of the entire organization.
Size of the entire organization.
Amount of international activity.
Nature of international activity.
Language policy of the organization.
Typical profiles.
Free-lance translator.
Translation company.
Organization with centralized internal translation department.
Organization with subcontracted translation.
Organization with decentralized translation.
Bi/multilingual organization.
International organization.
Organization with hybrid profile.
Correlations between user profiles and translation aids
User case examples
Translation Company: Transbus/Sweden.
Organization with internal translation department: LM Ericsson, Sweden.
Bilingual organization: Vaisala Oy, Finland.
International organization: WHO.
Public organization: Swiss Chancery.
Free-lance translator: Canadian/Quebecan survey.
Measurement of translation quantity
Measurement of translation quantity
Translation Tools
Introduction
ALPNET
CAP debis
Canadian Workplace Automation Research Center
Collins
GlobalWare
IBM Deutschland
LinguaTech
Microtac
Linguistic Systems
SITE/Eurolang
Soft-Linc
STAR
TA Electronic Publishing
Telesoft
TRADOS
Vertaal
Feature Checklist Examples
Feature checklist for translation memory
TM updating/maintenance
Alignment
Importing an aligned SL- and TL-segment to the translation memory database
Adding a SL-segment and its translation to TM while translating in TM mode
Modifying existing contents of TM's (apart from adding/importing)
TM application
The productivity of the translation memory
Translating a SL-text in TM mode and having TM generate/propose a match per SL-segment
The translator's workbench program
Openness
Editing
Using the termbank program
A Practical Evaluation of Writers' Aids
Introduction
Testing on the UBS corpora
Grammar checker for English
Well spotted errors
False flagging
Grammar checker for French
Well spotted errors
False flagging
Testing the ad hoc sentences
Grammar checker for English
Grammar checker for French
Results of tests on NPs
Tests on English NPs
Tests on French NPs
Second round of testing
Introduction
Functionality requirements
Measure
Method
Evaluation of the grammar checker wrt stated requirements
Results
Comments
Evaluation of the grammar checker wrt user requirements
User group and error types
Results
Conclusion
Evaluation of the framework
Evaluation of Knowledge Management Systems
Introduction
The KMS scenario
Knowledge management systems
Information retrieval
Search units
Precision, recall and specificity
Application and measurement criteria
Summary
Case study: KBAS-TW
General overview
A KBAS for technical writers
Designing and specifying the knowledge base
Design methodology
Knowledge acquisition
Knowledge organisation
Document production and reviewing
Summary
Towards evaluation
Evaluation methodology
KMS evaluation
SdT: A Case Study
Preliminaries
The genesis of the study
Methodology of the study
Status of the study
Structure of the report
Background
The texts
The organisational structure
Computing equipment
Computerized tools
Document preparation tools
Communications tools
Management tools
Translators' aids
Multilingual generation tools
Maintenance, development and future plans
Distribution of tools
The document production chain and its participants
Documents, their origin and their creation
Translation requests and transmission of documents
Reception in the translation services: planning
The department heads
Freelance translation
Machine translation
In-house translation: the unit heads
In-house translation: the translators
Research
Translation
Other activities
In-house translation: the secretaries
Revision
Return to the requester
Support activities
Terminology and the terminology help desk
Documentation
Training in computer tools
The computing correspondents
Evaluating translator's work bench systems
Evaluating local terminology management systems
Developing machine translation systems
Computing support, modernisation, development of multi-lingual tools
Dream tools
Conclusion
Closing remarks
EAGLES Comments Cover Form
References
All trade marks are hereby acknowledged.
ceditor
@tnos.ilc.pi.cnr.it