Sorry, you need to enable JavaScript to visit this website.


End-to-End Multimodal Speech Recognition


Transcription or sub-titling of open-domain videos is still a chal- lenging domain for Automatic Speech Recognition (ASR) due to the data’s challenging acoustics, variable signal processing and the essentially unrestricted domain of the data. In previous work, we have shown that the visual channel – specifically object and scene features – can help to adapt the acoustic model (AM) and language model (LM) of a recognizer, and we are now expanding this work to end-to-end approaches. In the case of a Connectionist Tempo- ral Classification (CTC)-based approach, we retain the separation of AM and LM, while for a sequence-to-sequence (S2S) approach, both information sources are adapted together, in a single model. This paper also analyzes the behavior of CTC and S2S models on noisy video data (How-To corpus), and compares it to results on the clean Wall Street Journal (WSJ) corpus, providing insight into the robustness of both approaches.
Index Terms— Audiovisual Speech Recognition, Connectionist Temporal Classification, Sequence-to-Sequence Model, Adaptation

1 user has voted: Siddharth Dalmia

Paper Details

Ramon Sanabria, Florian Metze
Submitted On:
12 April 2018 - 8:02pm
Short Link:
Presenter's Name:
Shruti Palaskar, Ramon Sanabria and Florian Metze
Paper Code:
Document Year:

Document Files




[1] Ramon Sanabria, Florian Metze, "End-to-End Multimodal Speech Recognition", IEEE SigPort, 2018. [Online]. Available: Accessed: Jun. 02, 2020.
url = {},
author = {Ramon Sanabria; Florian Metze },
publisher = {IEEE SigPort},
title = {End-to-End Multimodal Speech Recognition},
year = {2018} }
T1 - End-to-End Multimodal Speech Recognition
AU - Ramon Sanabria; Florian Metze
PY - 2018
PB - IEEE SigPort
UR -
ER -
Ramon Sanabria, Florian Metze. (2018). End-to-End Multimodal Speech Recognition. IEEE SigPort.
Ramon Sanabria, Florian Metze, 2018. End-to-End Multimodal Speech Recognition. Available at:
Ramon Sanabria, Florian Metze. (2018). "End-to-End Multimodal Speech Recognition." Web.
1. Ramon Sanabria, Florian Metze. End-to-End Multimodal Speech Recognition [Internet]. IEEE SigPort; 2018. Available from :