Sorry, you need to enable JavaScript to visit this website.

facebooktwittermailshare

End-to-End Multimodal Speech Recognition

Abstract: 

Transcription or sub-titling of open-domain videos is still a chal- lenging domain for Automatic Speech Recognition (ASR) due to the data’s challenging acoustics, variable signal processing and the essentially unrestricted domain of the data. In previous work, we have shown that the visual channel – specifically object and scene features – can help to adapt the acoustic model (AM) and language model (LM) of a recognizer, and we are now expanding this work to end-to-end approaches. In the case of a Connectionist Tempo- ral Classification (CTC)-based approach, we retain the separation of AM and LM, while for a sequence-to-sequence (S2S) approach, both information sources are adapted together, in a single model. This paper also analyzes the behavior of CTC and S2S models on noisy video data (How-To corpus), and compares it to results on the clean Wall Street Journal (WSJ) corpus, providing insight into the robustness of both approaches.
Index Terms— Audiovisual Speech Recognition, Connectionist Temporal Classification, Sequence-to-Sequence Model, Adaptation

up
1 user has voted: Siddharth Dalmia

Paper Details

Authors:
Ramon Sanabria, Florian Metze
Submitted On:
12 April 2018 - 8:02pm
Short Link:
Type:
Poster
Event:
Presenter's Name:
Shruti Palaskar, Ramon Sanabria and Florian Metze
Paper Code:
4069
Document Year:
2018
Cite

Document Files

icassp-poster-end.pdf

(128)

Subscribe

[1] Ramon Sanabria, Florian Metze, "End-to-End Multimodal Speech Recognition", IEEE SigPort, 2018. [Online]. Available: http://sigport.org/2524. Accessed: Jun. 18, 2019.
@article{2524-18,
url = {http://sigport.org/2524},
author = {Ramon Sanabria; Florian Metze },
publisher = {IEEE SigPort},
title = {End-to-End Multimodal Speech Recognition},
year = {2018} }
TY - EJOUR
T1 - End-to-End Multimodal Speech Recognition
AU - Ramon Sanabria; Florian Metze
PY - 2018
PB - IEEE SigPort
UR - http://sigport.org/2524
ER -
Ramon Sanabria, Florian Metze. (2018). End-to-End Multimodal Speech Recognition. IEEE SigPort. http://sigport.org/2524
Ramon Sanabria, Florian Metze, 2018. End-to-End Multimodal Speech Recognition. Available at: http://sigport.org/2524.
Ramon Sanabria, Florian Metze. (2018). "End-to-End Multimodal Speech Recognition." Web.
1. Ramon Sanabria, Florian Metze. End-to-End Multimodal Speech Recognition [Internet]. IEEE SigPort; 2018. Available from : http://sigport.org/2524