Sorry, you need to enable JavaScript to visit this website.

facebooktwittermailshare

CHARACTER-LEVEL LANGUAGE MODELING WITH HIERARCHICAL RECURRENT NEURAL NETWORKS

Abstract: 

Recurrent neural network (RNN) based character-level language models (CLMs) are extremely useful for modeling out-of-vocabulary words by nature. However, their performance is generally much worse than the word-level language models (WLMs), since CLMs need to consider longer history of tokens to properly predict the next one. We address this problem by proposing hierarchical RNN architectures, which consist of multiple modules with different timescales. Despite the multi-timescale structures, the input and output layers operate with the character-level clock, which allows the existing RNN CLM training approaches to be directly applicable without any modifications. Our CLM models show better perplexity than Kneser-Ney (KN) 5-gram WLMs on the One Billion Word Benchmark with only 2% of parameters. Also, we present real-time character-level end-to-end speech recognition examples on the Wall Street Journal (WSJ) corpus, where replacing traditional mono-clock RNN CLMs with the proposed models results in better recognition accuracies even though the number of parameters are reduced to 30%.

poster.pdf

PDF icon poster.pdf (334 downloads)
up
0 users have voted:

Paper Details

Authors:
Kyuyeon Hwang, Wonyong Sung
Submitted On:
6 March 2017 - 3:05am
Short Link:
Type:
Poster
Event:
Presenter's Name:
Wonyong Sung
Paper Code:
HLT-P1.4
Document Year:
2017
Cite

Document Files

poster.pdf

(334 downloads)

Subscribe

[1] Kyuyeon Hwang, Wonyong Sung, "CHARACTER-LEVEL LANGUAGE MODELING WITH HIERARCHICAL RECURRENT NEURAL NETWORKS", IEEE SigPort, 2017. [Online]. Available: http://sigport.org/1645. Accessed: Dec. 16, 2017.
@article{1645-17,
url = {http://sigport.org/1645},
author = {Kyuyeon Hwang; Wonyong Sung },
publisher = {IEEE SigPort},
title = {CHARACTER-LEVEL LANGUAGE MODELING WITH HIERARCHICAL RECURRENT NEURAL NETWORKS},
year = {2017} }
TY - EJOUR
T1 - CHARACTER-LEVEL LANGUAGE MODELING WITH HIERARCHICAL RECURRENT NEURAL NETWORKS
AU - Kyuyeon Hwang; Wonyong Sung
PY - 2017
PB - IEEE SigPort
UR - http://sigport.org/1645
ER -
Kyuyeon Hwang, Wonyong Sung. (2017). CHARACTER-LEVEL LANGUAGE MODELING WITH HIERARCHICAL RECURRENT NEURAL NETWORKS. IEEE SigPort. http://sigport.org/1645
Kyuyeon Hwang, Wonyong Sung, 2017. CHARACTER-LEVEL LANGUAGE MODELING WITH HIERARCHICAL RECURRENT NEURAL NETWORKS. Available at: http://sigport.org/1645.
Kyuyeon Hwang, Wonyong Sung. (2017). "CHARACTER-LEVEL LANGUAGE MODELING WITH HIERARCHICAL RECURRENT NEURAL NETWORKS." Web.
1. Kyuyeon Hwang, Wonyong Sung. CHARACTER-LEVEL LANGUAGE MODELING WITH HIERARCHICAL RECURRENT NEURAL NETWORKS [Internet]. IEEE SigPort; 2017. Available from : http://sigport.org/1645