Publication Details

Parameter-Efficient Transfer Learning of Pre-Trained Transformer Models for Speaker Verification Using Adapters

PENG, J.; STAFYLAKIS, T.; GU, R.; PLCHOT, O.; MOŠNER, L.; BURGET, L.; ČERNOCKÝ, J. Parameter-Efficient Transfer Learning of Pre-Trained Transformer Models for Speaker Verification Using Adapters. In ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings. Rhodes Island: IEEE Signal Processing Society, 2023. p. 1-5. ISBN: 978-1-7281-6327-7.
Czech title
Parametrově efektivní přenosové učení předtrénovaných modelů typu transformer pomocí adaptérů pro úlohu ověřování mluvčích
Type
conference paper
Language
English
Authors
Peng Junyi (DCGM)
Stafylakis Themos
GU, R.
Plchot Oldřich, Ing., Ph.D. (DCGM)
Mošner Ladislav, Ing. (DCGM)
Burget Lukáš, doc. Ing., Ph.D. (DCGM)
Černocký Jan, prof. Dr. Ing. (DCGM)
URL
Keywords

Speaker verification, pre-trained model, adapter, fine-tuning, transfer learning

Abstract

Recently, the pre-trained Transformer models have received a rising interest in the field of speech processing thanks to their great success in various downstream tasks. However, most fine-tuning approaches update all the parameters of the pre-trained model, which becomes prohibitive as the model size grows and sometimes results in over- fitting on small datasets. In this paper, we conduct a comprehensive analysis of applying parameter-efficient transfer learning (PETL) methods to reduce the required learnable parameters for adapting to speaker verification tasks. Specifically, during the fine-tuning process, the pre-trained models are frozen, and only lightweight modules inserted in each Transformer block are trainable (a method known as adapters). Moreover, to boost the performance in a cross- language low-resource scenario, the Transformer model is further tuned on a large intermediate dataset before directly fine-tuning it on a small dataset. With updating fewer than 4% of parameters, (our proposed) PETL-based methods achieve comparable performances with full fine-tuning methods (Vox1-O: 0.55%, Vox1-E: 0.82%, Vox1-H:1.73%).

Published
2023
Pages
1–5
Proceedings
ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings
ISBN
978-1-7281-6327-7
Publisher
IEEE Signal Processing Society
Place
Rhodes Island
DOI
10.1109/ICASSP49357.2023.10094795
EID Scopus
2-s2.0-85177568701
BibTeX 
@inproceedings{BUT185200,
  author="PENG, J. and STAFYLAKIS, T. and GU, R. and PLCHOT, O. and MOŠNER, L. and BURGET, L. and ČERNOCKÝ, J.",
  title="Parameter-Efficient Transfer Learning of Pre-Trained Transformer Models for Speaker Verification Using Adapters",
  booktitle="ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings",
  year="2023",
  pages="1--5",
  publisher="IEEE Signal Processing Society",
  address="Rhodes Island",
  doi="10.1109/ICASSP49357.2023.10094795",
  isbn="978-1-7281-6327-7",
  url="https://ieeexplore.ieee.org/document/10094795"
}
Back to top