Sorry, you need to enable JavaScript to visit this website.

facebooktwittermailshare

MEMORY VISUALIZATION FOR GATED RECURRENT NEURAL NETWORKS IN SPEECH RECOGNITION

Abstract: 

Recurrent neural networks (RNNs) have shown clear superiority in sequence modeling, particularly the ones with gated units, such as long short-term memory (LSTM) and gated recurrent unit (GRU). However, the dynamic properties behind the remarkable performance remain unclear in many applications, e.g., automatic speech recognition (ASR). This paper employs visualization techniques to study the behavior of LSTM and GRU when performing speech recognition tasks. Our experiments show some interesting patterns in the gated memory, and some of them have inspired simple yet effective modifications on the network structure. We report two of such modifications: (1) lazy cell update in LSTM, and (2) shortcut connections for residual learning. Both modifications lead to more comprehensible and powerful networks.

up
1 user has voted: Zhiyuan Tang

Paper Details

Authors:
Zhiyuan Tang, Ying Shi, Dong Wang, Yang Feng, Shiyue Zhang
Submitted On:
4 March 2017 - 3:11am
Short Link:
Type:
Poster
Event:
Presenter's Name:
Zhiyuan Tang
Paper Code:
4020
Document Year:
2017
Cite

Document Files

icassp17_visual.pdf

(70 downloads)

Subscribe

[1] Zhiyuan Tang, Ying Shi, Dong Wang, Yang Feng, Shiyue Zhang, "MEMORY VISUALIZATION FOR GATED RECURRENT NEURAL NETWORKS IN SPEECH RECOGNITION", IEEE SigPort, 2017. [Online]. Available: http://sigport.org/1463. Accessed: Jul. 27, 2017.
@article{1463-17,
url = {http://sigport.org/1463},
author = {Zhiyuan Tang; Ying Shi; Dong Wang; Yang Feng; Shiyue Zhang },
publisher = {IEEE SigPort},
title = {MEMORY VISUALIZATION FOR GATED RECURRENT NEURAL NETWORKS IN SPEECH RECOGNITION},
year = {2017} }
TY - EJOUR
T1 - MEMORY VISUALIZATION FOR GATED RECURRENT NEURAL NETWORKS IN SPEECH RECOGNITION
AU - Zhiyuan Tang; Ying Shi; Dong Wang; Yang Feng; Shiyue Zhang
PY - 2017
PB - IEEE SigPort
UR - http://sigport.org/1463
ER -
Zhiyuan Tang, Ying Shi, Dong Wang, Yang Feng, Shiyue Zhang. (2017). MEMORY VISUALIZATION FOR GATED RECURRENT NEURAL NETWORKS IN SPEECH RECOGNITION. IEEE SigPort. http://sigport.org/1463
Zhiyuan Tang, Ying Shi, Dong Wang, Yang Feng, Shiyue Zhang, 2017. MEMORY VISUALIZATION FOR GATED RECURRENT NEURAL NETWORKS IN SPEECH RECOGNITION. Available at: http://sigport.org/1463.
Zhiyuan Tang, Ying Shi, Dong Wang, Yang Feng, Shiyue Zhang. (2017). "MEMORY VISUALIZATION FOR GATED RECURRENT NEURAL NETWORKS IN SPEECH RECOGNITION." Web.
1. Zhiyuan Tang, Ying Shi, Dong Wang, Yang Feng, Shiyue Zhang. MEMORY VISUALIZATION FOR GATED RECURRENT NEURAL NETWORKS IN SPEECH RECOGNITION [Internet]. IEEE SigPort; 2017. Available from : http://sigport.org/1463