Multilingual Sequence-to-Sequence Speech Recognition: Architecture, Transfer Learning, and Language Modeling

Sequence-to-sequence (seq2seq) approach for low-resourceASR is a relatively new direction in speech research. The approachbenefits by performing model training without using lexicon andalignments. However, this poses a new problem of requiring moredata compared to conventional DNN-HMM systems. In this work,we attempt to use data from 10 BABEL languages to build a multilingualseq2seq model as a prior model, and then port them towards4 other BABEL languages using transfer learning approach. We alsoexplore different architectures for improving the prior multilingualseq2seq model. The paper also discusses the effect of integrating arecurrent neural network language model (RNNLM) with a seq2seqmodel during decoding. Experimental results show that the transferlearning approach from the multilingual model shows substantialgains over monolingual models across all 4 BABEL languages.Incorporating an RNNLM also brings significant improvements interms of %WER, and achieves recognition performance comparableto the models trained with twice more training data.