EUROCALL 2021 human/TTS results by language and condition. Yellow = TTS not worse, orange = TTS within 10%.

Condition
Everything language Danish English Farsi French Icelandic Irish Italian Mandarin Spanish Swedish
(#raters) (7) (26) (33) (7) (12) (29) (7) (2) (3) (4)
(#items) (25) (18) (32) (23) (16) (39) (23) (30) (16) (14)
(#annotations) (175) (468) (1056) (161) (192) (1131) (161) (60) (48) (56)
human_acceptable 97.1 90.2 97.7 91.3 95.8 97.5 88.8 100.0 100.0 98.2
tts_acceptable 64.0 97.2 70.1 98.1 85.4 98.1 85.1 76.7 56.2 98.2
human_better 63.4 16.2 63.2 8.7 54.2 30.9 31.7 33.3 93.8 55.4
tts_better 12.6 47.4 5.9 22.4 15.1 12.5 25.5 6.7 0.0 5.4
(same) (24.0) (36.3) (31.0) (68.9) (30.7) (56.7) (42.9) (60.0) (6.2) (39.3)
Teacher language Danish English Farsi French Icelandic Irish Italian Mandarin Spanish Swedish
(#raters) (7) (16) (13) (4) (9) (25) (2) (2) (1) (1)
(#items) (25) (18) (32) (23) (16) (39) (23) (30) (16) (14)
(#annotations) (175) (288) (416) (92) (144) (975) (46) (60) (16) (14)
human_acceptable 97.1 93.1 96.9 93.5 94.4 97.9 84.8 100.0 100.0 100.0
tts_acceptable 64.0 96.5 63.9 97.8 81.2 98.5 76.1 76.7 0.0 100.0
human_better 63.4 16.3 68.3 14.1 60.4 33.2 30.4 33.3 100.0 85.7
tts_better 12.6 52.4 5.8 27.2 15.3 11.8 30.4 6.7 0.0 0.0
(same) (24.0) (31.2) (26.0) (58.7) (24.3) (55.0) (39.1) (60.0) (0.0) (14.3)
Not teacher language Danish English Farsi French Icelandic Irish Italian Mandarin Spanish Swedish
(#raters) * (10) (20) (3) (3) (4) (5) * (2) (3)
(#items) * (18) (32) (23) (16) (39) (23) * (16) (14)
(#annotations) * (180) (640) (69) (48) (156) (115) * (32) (42)
human_acceptable * 85.6 98.3 88.4 100.0 94.9 90.4 * 100.0 97.6
tts_acceptable * 98.3 74.1 98.6 97.9 96.2 88.7 * 84.4 97.6
human_better * 16.1 59.8 1.4 35.4 16.0 32.2 * 90.6 45.2
tts_better * 39.4 5.9 15.9 14.6 16.7 23.5 * 0.0 7.1
(same) * (44.4) (34.2) (82.6) (50.0) (67.3) (44.3) * (9.4) (47.6)
Native/near-native language Danish English Farsi French Icelandic Irish Italian Mandarin Spanish Swedish
(#raters) (7) (13) (27) (3) (6) (6) (7) (2) (2) (3)
(#items) (25) (18) (32) (23) (16) (39) (23) (30) (16) (14)
(#annotations) (175) (234) (864) (69) (96) (234) (161) (60) (32) (42)
human_acceptable 97.1 95.3 97.3 85.5 93.8 97.9 88.8 100.0 100.0 97.6
tts_acceptable 64.0 94.9 72.5 97.1 81.2 99.6 85.1 76.7 40.6 97.6
human_better 63.4 23.1 61.0 5.8 53.1 47.0 31.7 33.3 100.0 40.5
tts_better 12.6 36.8 6.9 26.1 14.6 7.7 25.5 6.7 0.0 7.1
(same) (24.0) (40.2) (32.1) (68.1) (32.3) (45.3) (42.9) (60.0) (0.0) (52.4)
Not native/near-native language Danish English Farsi French Icelandic Irish Italian Mandarin Spanish Swedish
(#raters) * (13) (6) (4) (6) (23) * * (1) (1)
(#items) * (18) (32) (23) (16) (39) * * (16) (14)
(#annotations) * (234) (192) (92) (96) (897) * * (16) (14)
human_acceptable * 85.0 99.5 95.7 97.9 97.4 * * 100.0 100.0
tts_acceptable * 99.6 59.4 98.9 89.6 97.8 * * 87.5 100.0
human_better * 9.4 72.9 10.9 55.2 26.6 * * 81.2 100.0
tts_better * 58.1 1.0 19.6 15.6 13.7 * * 0.0 0.0
(same) * (32.5) (26.0) (69.6) (29.2) (59.6) * * (18.8) (0.0)
Only sentences language Danish English Farsi French Icelandic Irish Italian Mandarin Spanish Swedish
(#raters) (7) (26) (33) (7) (12) (29) (7) (2) (3) (4)
(#items) (14) (8) (20) (22) (15) (39) (23) (17) (16) (14)
(#annotations) (98) (208) (660) (154) (180) (1131) (161) (34) (48) (56)
human_acceptable 98.0 96.2 98.6 90.9 96.1 97.5 88.8 100.0 100.0 98.2
tts_acceptable 41.8 93.8 62.4 98.1 85.6 98.1 85.1 58.8 56.2 98.2
human_better 81.6 31.2 73.0 9.1 53.9 30.9 31.7 47.1 93.8 55.4
tts_better 3.1 28.4 5.0 23.4 15.6 12.5 25.5 2.9 0.0 5.4
(same) (15.3) (40.4) (22.0) (67.5) (30.6) (56.7) (42.9) (50.0) (6.2) (39.3)
Only words language Danish English Farsi French Icelandic Irish Italian Mandarin Spanish Swedish
(#raters) (7) (26) (33) (7) (12) * * (2) * *
(#items) (11) (10) (12) (1) (1) * * (13) * *
(#annotations) (77) (260) (396) (7) (12) * * (26) * *
human_acceptable 96.1 85.4 96.2 100.0 91.7 * * 100.0 * *
tts_acceptable 92.2 100.0 82.8 100.0 83.3 * * 100.0 * *
human_better 40.3 4.2 46.7 0.0 58.3 * * 15.4 * *
tts_better 24.7 62.7 7.3 0.0 8.3 * * 11.5 * *
(same) (35.1) (33.1) (46.0) (100.0) (33.3) * * (73.1) * *
Teacher + Only sentences language Danish English Farsi French Icelandic Irish Italian Mandarin Spanish Swedish
(#raters) (7) (16) (13) (4) (9) (25) (2) (2) (1) (1)
(#items) (14) (8) (20) (22) (15) (39) (23) (17) (16) (14)
(#annotations) (98) (128) (260) (88) (135) (975) (46) (34) (16) (14)
human_acceptable 98.0 100.0 97.3 93.2 94.8 97.9 84.8 100.0 100.0 100.0
tts_acceptable 41.8 92.2 55.0 97.7 81.5 98.5 76.1 58.8 0.0 100.0
human_better 81.6 31.2 78.1 14.8 60.0 33.2 30.4 47.1 100.0 85.7
tts_better 3.1 32.8 4.2 28.4 15.6 11.8 30.4 2.9 0.0 0.0
(same) (15.3) (35.9) (17.7) (56.8) (24.4) (55.0) (39.1) (50.0) (0.0) (14.3)
Teacher + Only words language Danish English Farsi French Icelandic Irish Italian Mandarin Spanish Swedish
(#raters) (7) (16) (13) (4) (9) * * (2) * *
(#items) (11) (10) (12) (1) (1) * * (13) * *
(#annotations) (77) (160) (156) (4) (9) * * (26) * *
human_acceptable 96.1 87.5 96.2 100.0 88.9 * * 100.0 * *
tts_acceptable 92.2 100.0 78.8 100.0 77.8 * * 100.0 * *
human_better 40.3 4.4 51.9 0.0 66.7 * * 15.4 * *
tts_better 24.7 68.1 8.3 0.0 11.1 * * 11.5 * *
(same) (35.1) (27.5) (39.7) (100.0) (22.2) * * (73.1) * *
Not teacher + Only sentences language Danish English Farsi French Icelandic Irish Italian Mandarin Spanish Swedish
(#raters) * (10) (20) (3) (3) (4) (5) * (2) (3)
(#items) * (8) (20) (22) (15) (39) (23) * (16) (14)
(#annotations) * (80) (400) (66) (45) (156) (115) * (32) (42)
human_acceptable * 90.0 99.5 87.9 100.0 94.9 90.4 * 100.0 97.6
tts_acceptable * 96.2 67.2 98.5 97.8 96.2 88.7 * 84.4 97.6
human_better * 31.2 69.8 1.5 35.6 16.0 32.2 * 90.6 45.2
tts_better * 21.2 5.5 16.7 15.6 16.7 23.5 * 0.0 7.1
(same) * (47.5) (24.8) (81.8) (48.9) (67.3) (44.3) * (9.4) (47.6)
Not teacher + Only words language Danish English Farsi French Icelandic Irish Italian Mandarin Spanish Swedish
(#raters) * (10) (20) (3) (3) * * * * *
(#items) * (10) (12) (1) (1) * * * * *
(#annotations) * (100) (240) (3) (3) * * * * *
human_acceptable * 82.0 96.2 100.0 100.0 * * * * *
tts_acceptable * 100.0 85.4 100.0 100.0 * * * * *
human_better * 4.0 43.3 0.0 33.3 * * * * *
tts_better * 54.0 6.7 0.0 0.0 * * * * *
(same) * (42.0) (50.0) (100.0) (66.7) * * * * *
Native/near-native + Only sentences language Danish English Farsi French Icelandic Irish Italian Mandarin Spanish Swedish
(#raters) (7) (13) (27) (3) (6) (6) (7) (2) (2) (3)
(#items) (14) (8) (20) (22) (15) (39) (23) (17) (16) (14)
(#annotations) (98) (104) (540) (66) (90) (234) (161) (34) (32) (42)
human_acceptable 98.0 100.0 98.3 84.8 94.4 97.9 88.8 100.0 100.0 97.6
tts_acceptable 41.8 88.5 66.5 97.0 81.1 99.6 85.1 58.8 40.6 97.6
human_better 81.6 43.3 70.9 6.1 53.3 47.0 31.7 47.1 100.0 40.5
tts_better 3.1 20.2 6.1 27.3 14.4 7.7 25.5 2.9 0.0 7.1
(same) (15.3) (36.5) (23.0) (66.7) (32.2) (45.3) (42.9) (50.0) (0.0) (52.4)
Native/near-native + Only words language Danish English Farsi French Icelandic Irish Italian Mandarin Spanish Swedish
(#raters) (7) (13) (27) (3) (6) * * (2) * *
(#items) (11) (10) (12) (1) (1) * * (13) * *
(#annotations) (77) (130) (324) (3) (6) * * (26) * *
human_acceptable 96.1 91.5 95.7 100.0 83.3 * * 100.0 * *
tts_acceptable 92.2 100.0 82.4 100.0 83.3 * * 100.0 * *
human_better 40.3 6.9 44.4 0.0 50.0 * * 15.4 * *
tts_better 24.7 50.0 8.3 0.0 16.7 * * 11.5 * *
(same) (35.1) (43.1) (47.2) (100.0) (33.3) * * (73.1) * *
Not native/near-native + Only sentences language Danish English Farsi French Icelandic Irish Italian Mandarin Spanish Swedish
(#raters) * (13) (6) (4) (6) (23) * * (1) (1)
(#items) * (8) (20) (22) (15) (39) * * (16) (14)
(#annotations) * (104) (120) (88) (90) (897) * * (16) (14)
human_acceptable * 92.3 100.0 95.5 97.8 97.4 * * 100.0 100.0
tts_acceptable * 99.0 44.2 98.9 90.0 97.8 * * 87.5 100.0
human_better * 19.2 82.5 11.4 54.4 26.6 * * 81.2 100.0
tts_better * 36.5 0.0 20.5 16.7 13.7 * * 0.0 0.0
(same) * (44.2) (17.5) (68.2) (28.9) (59.6) * * (18.8) (0.0)
Not native/near-native + Only words language Danish English Farsi French Icelandic Irish Italian Mandarin Spanish Swedish
(#raters) * (13) (6) (4) (6) * * * * *
(#items) * (10) (12) (1) (1) * * * * *
(#annotations) * (130) (72) (4) (6) * * * * *
human_acceptable * 79.2 98.6 100.0 100.0 * * * * *
tts_acceptable * 100.0 84.7 100.0 83.3 * * * * *
human_better * 1.5 56.9 0.0 66.7 * * * * *
tts_better * 75.4 2.8 0.0 0.0 * * * * *
(same) * (23.1) (40.3) (100.0) (33.3) * * * * *