Sorry, you need to enable JavaScript to visit this website.

facebooktwittermailshare

High Order Recurrent Neural Networks for Acoustic Modelling

Abstract: 

Vanishing long-term gradients are a major issue in training standard recurrent neural networks (RNNs), which can be alleviated by long short-term memory (LSTM) models with memory cells. However, the extra parameters associated with the memory cells mean an LSTM layer has four times as many parameters as an RNN with the same hidden vector size. This paper addresses the vanishing gradient problem using a high order RNN (HORNN) which has additional connections from multiple previous time steps. Speech recognition experiments using British English multi-genre broadcast (MGB3) data showed that the proposed HORNN architectures for rectified linear unit and sigmoid activation functions reduced word error rates (WER) by 4.2% and 6.3% over the corresponding RNNs, and gave similar WERs to a (projected) LSTM while using only 20%--50% of the recurrent layer parameters and computation.

up
0 users have voted:

Paper Details

Authors:
Chao Zhang, Phil Woodland
Submitted On:
12 April 2018 - 12:16pm
Short Link:
Type:
Poster
Event:
Presenter's Name:
Phil Woodland
Paper Code:
3291
Document Year:
2018
Cite

Document Files

cz277-ICASSP18-Poster-v3.pdf

(80 downloads)

Subscribe

[1] Chao Zhang, Phil Woodland, "High Order Recurrent Neural Networks for Acoustic Modelling", IEEE SigPort, 2018. [Online]. Available: http://sigport.org/2429. Accessed: Oct. 16, 2018.
@article{2429-18,
url = {http://sigport.org/2429},
author = {Chao Zhang; Phil Woodland },
publisher = {IEEE SigPort},
title = {High Order Recurrent Neural Networks for Acoustic Modelling},
year = {2018} }
TY - EJOUR
T1 - High Order Recurrent Neural Networks for Acoustic Modelling
AU - Chao Zhang; Phil Woodland
PY - 2018
PB - IEEE SigPort
UR - http://sigport.org/2429
ER -
Chao Zhang, Phil Woodland. (2018). High Order Recurrent Neural Networks for Acoustic Modelling. IEEE SigPort. http://sigport.org/2429
Chao Zhang, Phil Woodland, 2018. High Order Recurrent Neural Networks for Acoustic Modelling. Available at: http://sigport.org/2429.
Chao Zhang, Phil Woodland. (2018). "High Order Recurrent Neural Networks for Acoustic Modelling." Web.
1. Chao Zhang, Phil Woodland. High Order Recurrent Neural Networks for Acoustic Modelling [Internet]. IEEE SigPort; 2018. Available from : http://sigport.org/2429