The IWSLT 2021 BUT Speech Translation Systems

The paper describes BUTs English to Germanoffline speech translation (ST) systemsdeveloped for IWSLT2021. They are based onjointly trained Automatic Speech Recognition-Machine Translation models. Their performancesis evaluated on MustC-Common testset. In this work, we study their efficiencyfrom the perspective of having a large amountof separate ASR training data and MT trainingdata, and a smaller amount of speechtranslationtraining data. Large amounts ofASR and MT training data are utilized for pretrainingthe ASR and MT models. Speechtranslationdata is used to jointly optimizeASR-MT models by defining an end-to-enddifferentiable path from speech to translations.For this purpose, we use the internal continuousrepresentations from the ASR-decoder asthe input to MT module. We show that speechtranslation can be further improved by trainingthe ASR-decoder jointly with the MT-moduleusing large amount of text-only MT trainingdata. We also show significant improvementsby training an ASR module capable of generatingpunctuated text, rather than leaving thepunctuation task to the MT module.