Contents

 
• Introduction
  • Contributors
    • Authors
    • Other contributors
  • Guide to the report
    • About this report
    • How to read it
  • Types of systems considered
    • Long term
    • Short term
• The Framework Model
  • The ISO 9126 Standard
    • ISO 9126
      • Quality requirements definition
      • Evaluation preparation
      • Evaluation procedure
    • The EAGLES extensions to ISO 9126
  • Towards formalisation and automation
    • Key concepts in evaluation - a sketch for a formalisation
      • The evaluation function
      • Feature descriptions
      • Some useful terms
      • Candidates for standardisation
    • Parameterisable test bed
      • Parameterisable Test Bed
        • Parameters of objects
        • Parameters of users
        • Library
        • Testing
        • Assessment
      • Towards a PTB for language engineering
  • Applying the framework
    • Set of attributes
    • Requirements
    • Methods
  • Requirements analysis for language engineering evaluation
    • Requirements analysis and evaluation
      • Evaluation in the software life cycle
      • Scope of requirements statements
      • Functional and non-functional requirements
      • User profiling and requirements analysis
    • Issues for requirements analysis
      • Evaluation of existing products or systems
      • Reporting evaluations to the customer
      • The context
      • Evaluation of components and interactive systems
      • Evaluation of generic systems
      • Iteration and validity
      • Scope of evaluation
      • Knowledge acquisition for requirements analysis
    • Conclusion
    • Requirements analysis in language engineering evaluation
      • Domain-level requirements
        • Top level task description
        • Setup and variable elements
        • Quality requirements for top level
      • Transformation to reportable attributes
  • Measures
    • Attributes as measures
    • Validity of measures
      • Internal validity
      • External validity
    • Reliability of measures
    • Compositional measures
    • Typing of attributes
      • Binary values
      • Classificatory values
      • Comparative values
      • Numerical values
      • Metric values
    • Conclusion
  • Methods for system measurement
    • Test types
      • Scenario tests
        • Field tests
        • Laboratory tests
      • Systematic testing
        • Task-oriented tested
        • Menu-oriented testing
        • Benchmark testing
        • Systematic testing and adequacy evaluation
      • Feature inspection
    • Instruments
      • Testing instruments
      • Reporting instruments
        • Evaluation descriptions
        • Test problem reports
        • Result reports
        • Assessment reports
    • Test materials
      • Test sets
      • Test suites
      • Test collections
      • Conclusion
• Related Work
  • Looking into the consumer report paradigm
    • Basic notions
      • Consumer and product quality
      • Consumer and information
    • Activities and aims of consumer organizations
    • Basic requirements of comparative product testing
      • A procedure for comparative product testing
        • Selection of product class and products
        • Selecting and defining the attributes
        • Developing measuring methods
        • Performing measurements
        • Analysis of measuring results
        • Interpreting and evaluating data
      • Translating test results into product information
        • node43.html
      • Some basic requirements of product information
    • Discussion (and evaluation) of the literature
    • Literature suggestions (extracted from SWOKA reports)
  • Ongoing projects
    • TEMAA - A testbed study of evaluation methodologies
    • COBALT
    • RENOS
    • TSNLP - test suites for NLP
  • Recent History
    • Some history
      • ALPAC
      • ARPA
      • The Hewlett Packard Laboratories Study
      • MUC-3
• Conclusions and Directions for Future Work
• ISO Terms and Guidelines
  • Quality characteristics and subcharacteristics
    • Functionality
    • Reliability
    • Usability
    • Efficiency
    • Maintainability
    • Portability
  • Subcharacteristics
    • Functionality
    • Reliability
    • Usability
    • Efficiency
    • Maintainability
    • Portability
  • Terminology
  • Guidelines
    • After-sales servicing
    • Evaluation of importance
    • Evaluation process
    • Maintenance of software products
    • Quality objectives
    • Quality system elements
• Methods for System Measurement
  • Preface
  • Introduction to the testing of software
  • Survey of glass and black box testing techniques
    • Glass box testing
      • Static analysis techniques
      • Dynamic analysis techniques
      • Generation of test data in glass box tests
    • Black box testing
      • Black box testing - without user involvement
      • Black box testing - with user involvement
  • A user-oriented model of test types
    • Scenario tests
      • Types of scenario tests
        • Field tests
        • Laboratory tests
      • Types of metrics and results in scenario tests
    • Systematic testing
      • Task-oriented testing
        • Types of metrics and results in task-oriented testing
      • Menu-oriented testing
        • Types of metrics and results in menu-oriented testing
      • Benchmark testing
        • Types of metrics and results in benchmark tests
    • Feature inspection
      • Types of metrics and results in feature inspection
  • Survey of instruments
    • Testing instruments
      • Manual test instruments for user-oriented testing
        • Questionnaires
        • Checklists
        • Interviews
        • Observations
        • Think-aloud protocols
      • Automatic test instruments for user-oriented testing
    • Reporting instruments
      • Test descriptions
      • Test problem reports
      • Result reports
      • Assessment reports
  • Conclusion and outlook
• Requirements Analysis for Linguistic Engineering Evaluation
  • Introduction
  • Requirements analysis and evaluation
    • Evaluation in the software life cycle
    • Scope of requirements statements
    • Functional and non-functional requirements
    • User profiling and requirements analysis
  • Issues for requirements analysis
    • Reporting evaluations to the customer
    • Context and validity in LE evaluation
  • Knowledge acquisition for requirements analysis
    • Introduction
    • KADS
    • Knowledge acquisition (KA) techniques
  • Requirements analysis in LE evaluation
    • Introduction
    • Overview
    • Problem domain requirements
    • Transformation to reportable attributes
  • Generic requirements models: adaptation and reuse
  • Conclusions
• Evaluation of Writers' Aids
  • Introduction
    • Reviews
      • Notes from ``Intelligent text processing: A survey of the available products'' by Shona Douglas
        • Summary of method
        • The system centred taxonomy
        • The user centred taxonomy: A high-level specification of functionality
        • The test suite
        • Scoring the tests
        • Progress evaluation for The Editor's Assistant
      • PC Magazine Laboratories' performance tests
      • Chandler's review of grammar checkers
      • Conclusions
  • Selection of grammar checkers to be used as test beds
    • Reasons for the selection
    • Focus of attention
  • Quality characteristics
  • Task model
    • Model of writer types
    • Taxonomy of systems
    • Model of end-user types
    • Model of unproofed text
    • Model of proofed text
    • A Taxonomy of grammar errors
    • Taxonomy of advice types
    • Relationships among parts of the model
  • Requirements
    • Functionality requirement - part I
      • The grammatical errors made by the writer
      • The specification made by the system
      • The correction of errors by the end-user
    • Functionality requirement - part II
    • Functionality requirement - part III
    • Discussion
  • Reportable attributes of writers' aids
  • Methods
    • Functionality requirement - part I
    • Functionality requirement - part II
    • Methods for combination of attributes
    • Composition and structuring of test material
    • Instantiations of the interpretational scheme
• Evaluation of Translators' Aids
  • Translators' aids: user profiles
    • Dimensions of Translation
    • Relationships between different dimensions
    • Trends in dimensions
  • Product typology
    • Multilingual dictionaries
    • Multilingual thesauri
    • Terminology management databases
    • Translation memories
    • Miscellaneous
  • Featurization
    • Design and function of translation memory
      • Information contained
      • Off-line function: import
        • Import from raw format.
        • Import from native format.
      • Off-line function: analysis
        • Textual parsing.
        • Linguistic parsing.
        • Segmentation.
        • Alignment.
        • Term extraction.
        • Text statistics.
      • Off-line function: export
        • Control information.
      • Off-line functions: translation memory combinatorics
      • On-line functions
      • On-line function: retrieval
        • Exact match.
        • Fuzzy match.
      • On-line function: updating
      • On-line function: automatic translation
        • Automatic retrieval.
        • Automatic substitution.
      • Networking
    • Overview of feature checklists
      • Specification/inspection catalogues
      • Feature checklists of translators' tools
      • Further procedure - types and feature checklists
  • Evaluation procedure
    • General aspects of evaluation procedure
      • ISO on evaluation procedures
        • Quality requirement definition.
        • Evaluation preparation.
        • Evaluation procedure proper.
      • Desiderata for testing methods
      • Test types
        • Checklisting of features (= specification/inspection).
        • Scenario test (= users test the complete system in a realistic environment).
        • Benchmark testing (= systematic testing using test tools and materials).
    • Validation
    • Benchmarking translation memories
      • Usefulness
      • Online properties
        • Size (capacity of translation memory).
        • Speed (retrieval time).
        • Hit rate (number of matches).
      • Off-line properties
        • Text analysis methods.
      • Segmentation and alignment: success rate
      • Initial (raw) import into translation memory
      • Translation memory combinatorics
      • Export of translation memory
      • Specifying benchmarks
      • Suggestions for translation memory benchmarks
        • Translation memory: segmentation and retrieval.
        • Translation memory, import from external format.
        • Translation memory, learnability.
      • Translation memory, portability
      • Benchmarks suggested by Steenbakker's featurization
• User Profiles
  • Dimensions of translation
    • Quantity of translated text
    • Quantity of translation work
    • Text types
    • Characteristics of original text
    • Languages involved
    • Translation quality
    • Dimensions of the translation organization
      • Type of translation organization.
      • Size of translation organization.
      • Nature of the entire organization.
      • Size of the entire organization.
      • Amount of international activity.
      • Nature of international activity.
      • Language policy of the organization.
      • Typical profiles.
      • Free-lance translator.
      • Translation company.
      • Organization with centralized internal translation department.
      • Organization with subcontracted translation.
      • Organization with decentralized translation.
      • Bi/multilingual organization.
      • International organization.
      • Organization with hybrid profile.
    • Correlations between user profiles and translation aids
    • User case examples
      • Translation Company: Transbus/Sweden.
      • Organization with internal translation department: LM Ericsson, Sweden.
      • Bilingual organization: Vaisala Oy, Finland.
      • International organization: WHO.
      • Public organization: Swiss Chancery.
      • Free-lance translator: Canadian/Quebecan survey.
    • Measurement of translation quantity
    • Measurement of translation quantity
• Translation Tools
  • Introduction
  • ALPNET
  • CAP debis
  • Canadian Workplace Automation Research Center
  • Collins
  • GlobalWare
  • IBM Deutschland
  • LinguaTech
  • Microtac
  • Linguistic Systems
  • SITE/Eurolang
  • Soft-Linc
  • STAR
  • TA Electronic Publishing
  • Telesoft
  • TRADOS
  • Vertaal
• Feature Checklist Examples
  • Feature checklist for translation memory
    • TM updating/maintenance
      • Alignment
      • Importing an aligned SL- and TL-segment to the translation memory database
      • Adding a SL-segment and its translation to TM while translating in TM mode
      • Modifying existing contents of TM's (apart from adding/importing)
    • TM application
      • The productivity of the translation memory
      • Translating a SL-text in TM mode and having TM generate/propose a match per SL-segment
    • The translator's workbench program
      • Openness
      • Editing
      • Using the termbank program
• A Practical Evaluation of Writers' Aids
  • Introduction
    • Testing on the UBS corpora
      • Grammar checker for English
        • Well spotted errors
        • False flagging
      • Grammar checker for French
        • Well spotted errors
        • False flagging
    • Testing the ad hoc sentences
      • Grammar checker for English
      • Grammar checker for French
  • Results of tests on NPs
    • Tests on English NPs
    • Tests on French NPs
  • Second round of testing
    • Introduction
    • Functionality requirements
    • Measure
    • Method
    • Evaluation of the grammar checker wrt stated requirements
      • Results
      • Comments
    • Evaluation of the grammar checker wrt user requirements
      • User group and error types
      • Results
    • Conclusion
    • Evaluation of the framework
• Evaluation of Knowledge Management Systems
  • Introduction
  • The KMS scenario
    • Knowledge management systems
    • Information retrieval
    • Search units
    • Precision, recall and specificity
    • Application and measurement criteria
    • Summary
  • Case study: KBAS-TW
    • General overview
    • A KBAS for technical writers
    • Designing and specifying the knowledge base
      • Design methodology
      • Knowledge acquisition
      • Knowledge organisation
      • Document production and reviewing
    • Summary
  • Towards evaluation
    • Evaluation methodology
    • KMS evaluation
• SdT: A Case Study
  • Preliminaries
    • The genesis of the study
    • Methodology of the study
    • Status of the study
    • Structure of the report
  • Background
    • The texts
    • The organisational structure
    • Computing equipment
    • Computerized tools
      • Document preparation tools
      • Communications tools
      • Management tools
      • Translators' aids
      • Multilingual generation tools
      • Maintenance, development and future plans
      • Distribution of tools
  • The document production chain and its participants
    • Documents, their origin and their creation
    • Translation requests and transmission of documents
    • Reception in the translation services: planning
    • The department heads
    • Freelance translation
    • Machine translation
    • In-house translation: the unit heads
    • In-house translation: the translators
      • Research
      • Translation
      • Other activities
    • In-house translation: the secretaries
    • Revision
    • Return to the requester
  • Support activities
    • Terminology and the terminology help desk
    • Documentation
    • Training in computer tools
    • The computing correspondents
    • Evaluating translator's work bench systems
    • Evaluating local terminology management systems
    • Developing machine translation systems
    • Computing support, modernisation, development of multi-lingual tools
  • Dream tools
  • Conclusion
  • Closing remarks
• EAGLES Comments Cover Form
• References

All trade marks are hereby acknowledged.