Detail výsledku
Target Speaker ASR with Whisper
Klement Dominik, Ing., FIT (FIT), UPGM (FIT)
Wiesner Matthew, PhD., FIT (FIT)
Khudanpur Sanjeev
Černocký Jan, prof. Dr. Ing., UPGM (FIT)
Burget Lukáš, doc. Ing., Ph.D., UPGM (FIT)
We propose a novel approach to enable the use of large, single-speaker ASR
models, such as Whisper, for target speaker ASR. The key claim of this method is
that it is much easier to model relative differences among speakers by learning
to condition on frame-level diarization outputs than to learn the space of all
speaker embeddings. We find that adding even a single bias term per diarization
output type before the first transformer block can transform single-speaker ASR
models into target-speaker ASR models. Our approach also supports
speaker-attributed ASR by sequentially generating transcripts for each speaker in
a diarization output. This simplified method outperforms baseline speech
separation and diarization cascade by 12.9% absolute ORC-WER on the NOTSOFAR-1
dataset.
target-speaker ASR, diarization conditioning, multi-speaker ASR, Whisper
@inproceedings{BUT198049,
author="Alexander {Polok} and Dominik {Klement} and Matthew {Wiesner} and Sanjeev {Khudanpur} and Jan {Černocký} and Lukáš {Burget}",
title="Target Speaker ASR with Whisper",
booktitle="ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings",
year="2025",
pages="1--5",
publisher="IEEE Signal Processing Society",
address="Hyderabad",
doi="10.1109/ICASSP49660.2025.10887683",
isbn="979-8-3503-6874-1",
url="https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10887683"
}