Aligning pre-trained models via an interpretable latent space

The usage of large pre-trained models has become ubiquitous in several fields of Artificial Intelligence (AI). The recent developments and capabilities of large language models are a prime example. Similar trends are seen in areas such as speech technology, computer vision, and across disciplines related to medicine and healthcare. In speech and language processing, current state-of-the-art models are trained independent of each other, and a majority of them are uni-modal at their input. Whereas, a number of applications such as spoken language translation, task-oriented dialogue systems and atypical speech assessment either require or benefit from a careful combination of two or more models. A naive way of building a cascade pipeline results in error propagation and compounding, while joint-training causes catastrophic forgetting, where the benefits of pre-training diminish. Combined with these limitations, the black-box nature of the models make them hard to interpret; moreover, they propagate harmful biases acquired from the massive web-crawled training data. To overcome these limitations of the current state-of-the-art, this PhD topic aims to develop theoretically-motivated methods for aligning any arbitrary pre-trained models via an interpretable latent space. The alignment will enable to join the models without requiring to fine-tune them. The interpretable latent space will ease the study and identification of the linguistic, para-linguistic, and fairness attributes that are encoded in the pre-trained models. This will also allow the explainability of the models' output in human-centred applications related to medicine and healthcare such as atypical speech and language assessment. The shared latent space enables to use efficient data augmentation and bias mitigation methods that will enhance the robustness of speech and language applications.