Sorry, you need to enable JavaScript to visit this website.

Listen to the Pixels

Primary tabs

Citation Author(s):
Sanjoy Chowdhury,Subhrajyoti Dasgupta, Sudip Das, Ujjwal Bhattacharya
Submitted by:
Sanjoy Chowdhury
Last updated:
26 September 2021 - 4:44am
Document Type:
Presentation Slides
Document Year:
Presenters Name:
Sanjoy Chowdhury
Paper Code:



Performing sound source separation and visual object segmentation jointly in naturally occurring videos is a notoriously difficult task, especially in the absence of annotated data. In this study, we leverage the concurrency between audio and visual modalities in an attempt to solve the joint audio-visual segmentation problem in a self-supervised manner. Human beings interact with the physical world through a few sensory systems such as vision, auditory, movement, etc. The usefulness of the interplay of such systems lies in the concept of degeneracy. It tells us that the cross-modal signals can educate each other without the presence of an external supervisor. In this work, we efficiently exploit this fact that learning from one modality inherently helps to find patterns in others by introducing a novel audio-visual fusion technique. Also, to the best of our knowledge, we are the first to address the partially occluded sound source segmentation task. Our study shows that the proposed model significantly outperforms existing state-of-the-art methods in both visual and audio source separation tasks.

0 users have voted:


The work talks about audio visual joint segmentation in a self-supervised manner.

Dataset Files

ICIP '21 Listen to the Pixels_slides.pdf