Publication Details

Target Speaker ASR with Whisper

POLOK Alexander, KLEMENT Dominik, WIESNER Matthew, KHUDANPUR Sanjeev, ČERNOCKÝ Jan and BURGET Lukáš. Target Speaker ASR with Whisper. In: Proceedings of ICASSP 2025. Hyderabad: IEEE Biometric Council, 2025, pp. 1-5. ISBN 979-8-3503-6874-1. Available from: https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10887683
Czech title
ASR cílového mluvčího pomocí modelu Whisper
Type
conference paper
Language
english
Authors
Polok Alexander, Ing. (DCGM FIT BUT)
Klement Dominik, Bc. (FIT BUT)
Wiesner Matthew (JHU)
Khudanpur Sanjeev (JHU)
Černocký Jan, prof. Dr. Ing. (DCGM FIT BUT)
Burget Lukáš, doc. Ing., Ph.D. (DCGM FIT BUT)
URL
Keywords

target-speaker ASR, diarization conditioning, multi-speaker ASR, Whisper

Abstract

We propose a novel approach to enable the use of
large, single-speaker ASR models, such as Whisper, for target
speaker ASR. The key claim of this method is that it is much
easier to model relative differences among speakers by learning
to condition on frame-level diarization outputs than to learn
the space of all speaker embeddings. We find that adding even
a single bias term per diarization output type before the first
transformer block can transform single-speaker ASR models
into target-speaker ASR models. Our approach also supports
speaker-attributed ASR by sequentially generating transcripts
for each speaker in a diarization output. This simplified method
outperforms baseline speech separation and diarization cascade
by 12.9% absolute ORC-WER on the NOTSOFAR-1 dataset.

Published
2025
Pages
1-5
Proceedings
Proceedings of ICASSP 2025
Conference
ICASSP 2025, International Conference on Acoustics, Speech, and Signal Processing, Hyderabad, IN
ISBN
979-8-3503-6874-1
Publisher
IEEE Biometric Council
Place
Hyderabad, IN
DOI
10.1109/ICASSP49660.2025.10887683
BibTeX 
@INPROCEEDINGS{FITPUB13520,
   author = "Alexander Polok and Dominik Klement and Matthew Wiesner and Sanjeev Khudanpur and Jan \v{C}ernock\'{y} and Luk\'{a}\v{s} Burget",
   title = "Target Speaker ASR with Whisper",
   pages = "1--5",
   booktitle = "Proceedings of ICASSP 2025",
   year = 2025,
   location = "Hyderabad, IN",
   publisher = "IEEE Biometric Council",
   ISBN = "979-8-3503-6874-1",
   doi = "10.1109/ICASSP49660.2025.10887683",
   language = "english",
   url = "https://www.fit.vut.cz/research/publication/13520"
}
Back to top