Sorry, you need to enable JavaScript to visit this website.

Unsupervised Audio-Caption Aligning Learns Correspondences between Individual Sound Events and Textual Phrases

Citation Author(s):
Huang Xie, Okko Rasanen, Konstantinos Drossos, Tuomas Virtanen
Submitted by:
Huang Xie
Last updated:
9 May 2022 - 7:47am
Document Type:
Presentation Slides
Document Year:
Huang Xie
Paper Code:


We investigate unsupervised learning of correspondences between sound events and textual phrases through aligning audio clips with textual captions describing the content of a whole audio clip. We align originally unaligned and unannotated audio clips and their captions by scoring the similarities between audio frames and words, as encoded by modality-specific encoders and using a ranking-loss criterion to optimize the model. After training, we obtain clip-caption similarity by averaging frame-word similarities and estimate event-phrase correspondences by calculating frame-phrase similarities. We evaluate the method with two cross-modal tasks: audio-caption retrieval, and phrase-based sound event detection (SED). Experimental results show that the proposed method can globally associate audio clips with captions as well as locally learn correspondences between individual sound events and textual phrases in an unsupervised manner.

1 user has voted: Huang Xie


Presentation slides