Automatic evaluation of end-to-end dialog systems with adequacy-fluency metrics

Abstract

End-to-End dialog systems are gaining interest due to the recent advances of deep neuralnetworks and the availability of large human-human dialog corpora. However, in spite ofbeing of fundamental importance to systematically improve the performance of this kind ofsystems, automatic evaluation of the generated dialog utterances is still an unsolved problem.Indeed, most of the proposed objective metrics shown low correlation with human evaluations.In this paper, we evaluate a two-dimensional evaluation metric that is designed to operateat sentence level, which considers the syntactic and semantic information carried along theanswers generated by an end-to-end dialog system with respect to a set of references. Theproposed metric, when applied to outputs generated by the systems participating in track 2of the DSTC-6 challenge, shows a higher correlation with human evaluations (up to 12.8%relative improvement at the system level) than the best of the alternative state-of-the-artautomatic metrics currently available.