Investigation of Speaker Representation for Target-Speaker Speech Processing

Target-speaker speech processing (TS) tasks, such as target-speaker
automatic speech recognition (TS-ASR), target speech extraction
(TSE), and personal voice activity detection (p-VAD), are important
for extracting information about a desired speaker's speech even
when it is corrupted by interfering speakers. While most studies
have focused on training schemes or system architectures for each
specific task, the auxiliary network for embedding target-speaker
cues has not been investigated comprehensively in a unified cross-
task evaluation. Therefore, this paper aims to address a fundamental
question: what is the preferred speaker embedding for TS tasks?
To this end, for the TS-ASR, TSE, and p-VAD tasks, we compare
pre-trained speaker encoders (i.e., self-supervised or speaker recog-
nition models) that compute speaker embeddings from pre-recorded
enrollment speech of the target speaker with ideal speaker embed-
dings derived directly from the target speaker's identity in the form
of a one-hot vector. To further understand the properties of ideal
speaker embedding, we optimize it using a gradient-based approach
to improve performance on the TS task. Our analysis reveals that
speaker verification performance is somewhat unrelated to TS task
performances, the one-hot vector outperforms enrollment-based
ones, and the optimal embedding depends on the input mixture.