Sorry, you need to enable JavaScript to visit this website.

facebooktwittermailshare

Can DNNs Learn to Lipread Full Sentences ?

Abstract: 

Finding visual features and suitable models for lipreading tasks that are more complex than a well-constrained vocabulary has proven challenging. This paper explores state-of-the-art Deep Neural Network architectures for lipreading based on a Sequence to Sequence Recurrent Neural Network. We report results for both hand-crafted and 2D/3D Convolutional Neural Network visual front-ends, online monotonic attention, and a joint Connectionist Temporal Classification-Sequence-to-Sequence loss. The system is evaluated on the publicly available TCD-TIMIT dataset, with 59 speakers and a vocabulary of over 6000 words. Results show a major improvement on a Hidden Markov Model framework. A fuller analysis of performance across visemes demonstrates that the network is not only learning the language model, but actually learning to lipread.

up
0 users have voted:

Paper Details

Authors:
George Sterpu, Christian Saam, Naomi Harte
Submitted On:
8 October 2018 - 1:50am
Short Link:
Type:
Presentation Slides
Event:
Presenter's Name:
George Sterpu
Paper Code:
MA.L1.4
Document Year:
2018
Cite

Document Files

slides.pdf

(46)

Subscribe

[1] George Sterpu, Christian Saam, Naomi Harte, "Can DNNs Learn to Lipread Full Sentences ?", IEEE SigPort, 2018. [Online]. Available: http://sigport.org/3608. Accessed: Jun. 15, 2019.
@article{3608-18,
url = {http://sigport.org/3608},
author = {George Sterpu; Christian Saam; Naomi Harte },
publisher = {IEEE SigPort},
title = {Can DNNs Learn to Lipread Full Sentences ?},
year = {2018} }
TY - EJOUR
T1 - Can DNNs Learn to Lipread Full Sentences ?
AU - George Sterpu; Christian Saam; Naomi Harte
PY - 2018
PB - IEEE SigPort
UR - http://sigport.org/3608
ER -
George Sterpu, Christian Saam, Naomi Harte. (2018). Can DNNs Learn to Lipread Full Sentences ?. IEEE SigPort. http://sigport.org/3608
George Sterpu, Christian Saam, Naomi Harte, 2018. Can DNNs Learn to Lipread Full Sentences ?. Available at: http://sigport.org/3608.
George Sterpu, Christian Saam, Naomi Harte. (2018). "Can DNNs Learn to Lipread Full Sentences ?." Web.
1. George Sterpu, Christian Saam, Naomi Harte. Can DNNs Learn to Lipread Full Sentences ? [Internet]. IEEE SigPort; 2018. Available from : http://sigport.org/3608