Fraunhofer Institute: Evaluation Report

The well-reputed Fraunhofer Institute for Industrial Engineering IAO extensively tested Personal Translator with regard to efficiency and usability. Forty candidates, each with different levels of knowledge of the English language, translated technical texts and business letters using Personal Translator and, also as an alternative, using only a conventional dictionary.

Test Personal Translator – Summary

1. Objectives  

2. Operationalization of the measured variables

3. Results 

Test summary

According to tests carried out by relevant specialist magazines the translation program Personal Translator (PT) has become the best automatic translation software on the market within only a few years. To test the extent to which Personal Translator helps increase the quality and efficiency of a translation as compared with the use of a conventional dictionary, the Usability Engineering marketing strategy team of the Fraunhofer IAO carried out a usability test with potential users.

1. Objectives

The test focused on the extent to which Personal Translator simplifies the completion of typical user tasks as compared with the use of a conventional dictionary (on paper). These typical tasks included:

• Translating English texts into German
• Writing English texts
• Translating German texts into English

In detail, the following points were tested (each compared with the use of a conventional dictionary):

• Can the time and resources required for completing the above-mentioned tasks be reduced by using PT?
• Can productivity (number of words per time unit) be increased by using PT?
• Can the quality of the final results be improved by using PT for the above-mentioned tasks?
• Can the above-mentioned tasks be made more enjoyable by using PT?

In this way, the three most important usability aspects of an interactive software product were analyzed in accordance with ISO 9241-11. The test evaluated the extent to which PT can be used by specified users to translate and write texts effectively, efficiently and with satisfaction (ISO 9241-11, ISO, 1996). The test also evaluated whether the use of PT saved time and increased productivity. Another interesting aspect was the extent to which PT helped users with a specific level of knowledge to achieve better results with different types of text. In addition, optimization recommendations for the design of the translation software were derived from the test.

2. Operationalization of the measured variables

Effectiveness characterizes the quality of the performed task. To operationalize effectiveness, the results of the participants were assessed separately by four teachers of the English language and graded from 0-15. To avoid biased evaluations due to the origin of the results, the assessment was carried out blind, i.e. the teachers did not know that the test was an evaluation of PT. They did not know the number of test subjects or that they were given two texts from each person. Once the test was completed, the texts were all numbered so that the test managers were able to link them to the test subjects. The teachers, however, had no idea who had written the texts and where they came from. They were only told that different translation aids were to be compared.

As the second criterion of usability, efficiency measures the resources expended in relation to the effectiveness: efficiency decreases with growing expenditure of resources. The expenditure of resources was measured using both objective and subjective characteristic variables. The time required for completing the task was an objectively measurable value used in the evaluation. Subjective variables were the ratings of the test subjects in terms of effort and time required on two scales from 0-100.

The satisfaction with the application was taken into account via a subjective scale: the test participants specified how much fun they had had in completing the task.

3. Results

• Using Personal Translator resulted in an average time saving of 41.4 % as compared with the time required using a conventional dictionary. 
• Using Personal Translator increased productivity by 164.5 %.
• The quality of the texts translated with Personal Translator was marginally better; but this was not significant statistically.
• The efficiency (quality in relation to resource expenditure) was increased significantly by using PT.
• The subjective assessment of the test participants showed that they felt that working with Personal Translator was less of an effort, less time-consuming and more fun than translating or writing texts using a conventional dictionary.