Evaluating speech recognition in the context of a spoken dialogue system: critical error rate

Evaluating a speech recognition system is a key issue towards understanding its deficiencies and focusing potential improvements on useful aspects. When a system is designed for a given application, it is particularly relevant to have an evaluation procedure that reflects the role of the system in this application. Evaluating continuous speech recognition through word error rate is not completely appropriate when the speech recognizer is used as spoken dialogue system input. Some errors are particularly harmful, when they concern content words for example, while some others do not have any impact on the following comprehension step. The attempt is not to evaluate natural language understanding but to propose a more appropriate evaluation of speech recognition, by making use of semantic information to define the notion of critical errors.