Sorry, you need to enable JavaScript to visit this website.

facebooktwittermailshare

Multimodal One-shot Learning of Speech and Images

Abstract: 

Image a robot is shown new concepts visually together with spoken tags, e.g. "milk", "eggs", "butter". After seeing one paired audiovisual example per class, it is shown a new set of unseen instances of these objects, and asked to pick the "milk". Without receiving any hard labels, could it learn to match the new continuous speech input to the correct visual instance? Although unimodal one-shot learning has been studied, where one labelled example in a single modality is given per class, this example motivates multimodal one-shot learning. Our main contribution is to formally define this task, and to propose several baseline and advanced models. We use a dataset of paired spoken and visual digits to specifically investigate recent advances in Siamese convolutional neural networks. Our best Siamese model achieves twice the accuracy of a nearest neighbour model using pixel-distance over images and dynamic time warping over speech in 11-way cross-modal matching.

up
0 users have voted:

Paper Details

Authors:
Ryan Eloff, Herman A. Engelbrecht, Herman Kamper
Submitted On:
10 May 2019 - 6:38pm
Short Link:
Type:
Poster
Event:
Presenter's Name:
Ryan Eloff
Paper Code:
ICASSP19005
Document Year:
2019
Cite

Document Files

icassp2019_poster_multimodal_oneshot

(35)

Subscribe

[1] Ryan Eloff, Herman A. Engelbrecht, Herman Kamper, "Multimodal One-shot Learning of Speech and Images", IEEE SigPort, 2019. [Online]. Available: http://sigport.org/4421. Accessed: Nov. 21, 2019.
@article{4421-19,
url = {http://sigport.org/4421},
author = {Ryan Eloff; Herman A. Engelbrecht; Herman Kamper },
publisher = {IEEE SigPort},
title = {Multimodal One-shot Learning of Speech and Images},
year = {2019} }
TY - EJOUR
T1 - Multimodal One-shot Learning of Speech and Images
AU - Ryan Eloff; Herman A. Engelbrecht; Herman Kamper
PY - 2019
PB - IEEE SigPort
UR - http://sigport.org/4421
ER -
Ryan Eloff, Herman A. Engelbrecht, Herman Kamper. (2019). Multimodal One-shot Learning of Speech and Images. IEEE SigPort. http://sigport.org/4421
Ryan Eloff, Herman A. Engelbrecht, Herman Kamper, 2019. Multimodal One-shot Learning of Speech and Images. Available at: http://sigport.org/4421.
Ryan Eloff, Herman A. Engelbrecht, Herman Kamper. (2019). "Multimodal One-shot Learning of Speech and Images." Web.
1. Ryan Eloff, Herman A. Engelbrecht, Herman Kamper. Multimodal One-shot Learning of Speech and Images [Internet]. IEEE SigPort; 2019. Available from : http://sigport.org/4421