Sorry, you need to enable JavaScript to visit this website.

facebooktwittermailshare

Verifying the Long-range Dependency of RNN Language Models

Abstract: 

It has been argued that recurrent neural network language models are better in capturing long-range dependency than n-gram language models. In this paper, we attempt to verify this claim by investigating the prediction accuracy and the perplexity of these language models as a function of word position, i.e., the position of a word in a sentence. It is expected that as word position increases, the advantage of using recurrent neural network language models over n-gram language models will become more and more evident. On the text corpus of Penn Tree Bank (PTB), a recurrent neural network language model outperforms a trigram language model in both perplexity and word prediction. However, on the AMI meeting corpus, a trigram outperforms a recurrent neural network language model.

up
0 users have voted:

Paper Details

Authors:
Tzu-Hsuan Tseng, Tzu-Hsuan Yang, Chia-Ping Chen
Submitted On:
21 November 2016 - 10:24am
Short Link:
Type:
Presentation Slides
Event:
Presenter's Name:
Tzu-Hsuan, Tseng
Paper Code:
41
Document Year:
2016
Cite

Document Files

41_ngram_rnn

(291)

Subscribe

[1] Tzu-Hsuan Tseng, Tzu-Hsuan Yang, Chia-Ping Chen, "Verifying the Long-range Dependency of RNN Language Models", IEEE SigPort, 2016. [Online]. Available: http://sigport.org/1284. Accessed: Apr. 25, 2019.
@article{1284-16,
url = {http://sigport.org/1284},
author = {Tzu-Hsuan Tseng; Tzu-Hsuan Yang; Chia-Ping Chen },
publisher = {IEEE SigPort},
title = {Verifying the Long-range Dependency of RNN Language Models},
year = {2016} }
TY - EJOUR
T1 - Verifying the Long-range Dependency of RNN Language Models
AU - Tzu-Hsuan Tseng; Tzu-Hsuan Yang; Chia-Ping Chen
PY - 2016
PB - IEEE SigPort
UR - http://sigport.org/1284
ER -
Tzu-Hsuan Tseng, Tzu-Hsuan Yang, Chia-Ping Chen. (2016). Verifying the Long-range Dependency of RNN Language Models. IEEE SigPort. http://sigport.org/1284
Tzu-Hsuan Tseng, Tzu-Hsuan Yang, Chia-Ping Chen, 2016. Verifying the Long-range Dependency of RNN Language Models. Available at: http://sigport.org/1284.
Tzu-Hsuan Tseng, Tzu-Hsuan Yang, Chia-Ping Chen. (2016). "Verifying the Long-range Dependency of RNN Language Models." Web.
1. Tzu-Hsuan Tseng, Tzu-Hsuan Yang, Chia-Ping Chen. Verifying the Long-range Dependency of RNN Language Models [Internet]. IEEE SigPort; 2016. Available from : http://sigport.org/1284