Automatic evaluation methods of a speech translation system's capability

The main goal of the paper is to propose automatic schemes for the translation paired comparison method, which was proposed by the authors to evaluate precisely a speech translation system's capability. In the method, the outputs of the speech translation system are subjectively compared with the results of native Japanese taking the Test of English for International Communication (TOEIC), which is used as a measure of a person's speech translation capability. Experiments are conducted on TDMT, which is a subsystem of the Japanese-to-English speech translation system ATR-MATRIX developed at ATR Interpreting Telecommunications Research Laboratories. The winning rate of TDMT shows a good correlation with the TOEIC scores of the examinees. A regression analysis on the subjective results shows that the translation capability of TDMT matches a person scoring around 700 on the TOEIC. The automatic evaluation methods use DP-based similarity, which is calculated by DP distances between a translation output and multiple translation answers. The answers are collected by two methods: paraphrasing and query from a parallel corpus. In both types of collection, the similarity shows the same good correlation with the TOEIC scores of the examinees as the subjective winning rate. Regression analysis using similarity shows that the system's matched point is around 750. We also show effects of paraphrased data.