This paper describes the multi-query multi-head attention (MQMHA)
pooling and inter-topK penalty methods which were first proposed in
our submitted system description for VoxCeleb speaker recognition
challenge (VoxSRC) 2021. Most multi-head attention pooling mechanisms either attend to the whole feature through multiple heads or
attend to several split parts of the whole feature. Our proposed
MQMHA combines both these two mechanisms and gain more
diversified information. The margin-based softmax loss functions
are commonly adopted to obtain discriminative speaker representations. To further enhance the inter-class discriminability, we propose
a method that adds an extra inter-topK penalty on some confused
speakers. By adopting both the MQMHA and inter-topK penalty, we
achieved state-of-the-art performance in all of the public VoxCeleb
test sets.