Sorry, you need to enable JavaScript to visit this website.

facebooktwittermailshare

VID2SPEECH: SPEECH RECONSTRUCTION FROM SILENT VIDEO

Abstract: 

Speechreading is a notoriously difficult task for humans to perform. In this paper we present an end-to-end model based on a convolutional neural network (CNN) for generating an intelligible acoustic speech signal from silent video frames of a speaking person. The proposed CNN generates sound features for each frame based on its neighboring frames. Waveforms are then synthesized from the learned speech features to produce intelligible speech. We show that by leveraging the automatic feature learning capabilities of a CNN, we can obtain state-of-the-art word intelligibility on the GRID dataset, and show promising results for learning out-of-vocabulary (OOV) words.

up
0 users have voted:

Paper Details

Authors:
Ariel Ephrat, Shmuel Peleg
Submitted On:
27 February 2017 - 3:05pm
Short Link:
Type:
Poster
Event:
Presenter's Name:
Ariel Ephrat
Paper Code:
1239
Document Year:
2017
Cite

Document Files

vid2speech_poster

(92 downloads)

Subscribe

[1] Ariel Ephrat, Shmuel Peleg, "VID2SPEECH: SPEECH RECONSTRUCTION FROM SILENT VIDEO", IEEE SigPort, 2017. [Online]. Available: http://sigport.org/1448. Accessed: Oct. 18, 2017.
@article{1448-17,
url = {http://sigport.org/1448},
author = {Ariel Ephrat; Shmuel Peleg },
publisher = {IEEE SigPort},
title = {VID2SPEECH: SPEECH RECONSTRUCTION FROM SILENT VIDEO},
year = {2017} }
TY - EJOUR
T1 - VID2SPEECH: SPEECH RECONSTRUCTION FROM SILENT VIDEO
AU - Ariel Ephrat; Shmuel Peleg
PY - 2017
PB - IEEE SigPort
UR - http://sigport.org/1448
ER -
Ariel Ephrat, Shmuel Peleg. (2017). VID2SPEECH: SPEECH RECONSTRUCTION FROM SILENT VIDEO. IEEE SigPort. http://sigport.org/1448
Ariel Ephrat, Shmuel Peleg, 2017. VID2SPEECH: SPEECH RECONSTRUCTION FROM SILENT VIDEO. Available at: http://sigport.org/1448.
Ariel Ephrat, Shmuel Peleg. (2017). "VID2SPEECH: SPEECH RECONSTRUCTION FROM SILENT VIDEO." Web.
1. Ariel Ephrat, Shmuel Peleg. VID2SPEECH: SPEECH RECONSTRUCTION FROM SILENT VIDEO [Internet]. IEEE SigPort; 2017. Available from : http://sigport.org/1448