Sorry, you need to enable JavaScript to visit this website.

facebooktwittermailshare

On Training Bi-directional Neural Network Language Model with Noise Contrastive Estimation

Abstract: 

Although uni-directional recurrent neural network language
model(RNNLM) has been very successful, it’s hard to train a
bi-directional RNNLM properly due to the generative nature of
language model. In this work, we propose to train bi-directional
RNNLM with noise contrastive estimation(NCE), since the
properities of NCE training will help the model to acheieve
sentence-level normalization. Experiments are conducted on
two hand-crafted tasks on the PTB data set: a rescore task and
a sanity test. Although(regretfully), the model trained by NCE
did not out-perform the baseline uni-directional NNLM, it is
shown that NCE-trained bi-directional NNLM behaves well in
the sanity test and outperformed the one trained by conventional
maximum likelihood training on the rescore task.

up
0 users have voted:

Paper Details

Authors:
Tianxing He, Yu Zhang, Jasha Droppo, Kai Yu
Submitted On:
16 October 2016 - 11:45am
Short Link:
Type:
Poster
Event:
Presenter's Name:
Tianxing He
Paper Code:
93
Document Year:
2016
Cite

Document Files

iscslp2016_poster_v2.pdf

(103 downloads)

Subscribe

[1] Tianxing He, Yu Zhang, Jasha Droppo, Kai Yu, "On Training Bi-directional Neural Network Language Model with Noise Contrastive Estimation", IEEE SigPort, 2016. [Online]. Available: http://sigport.org/1255. Accessed: Jul. 22, 2017.
@article{1255-16,
url = {http://sigport.org/1255},
author = {Tianxing He; Yu Zhang; Jasha Droppo; Kai Yu },
publisher = {IEEE SigPort},
title = {On Training Bi-directional Neural Network Language Model with Noise Contrastive Estimation},
year = {2016} }
TY - EJOUR
T1 - On Training Bi-directional Neural Network Language Model with Noise Contrastive Estimation
AU - Tianxing He; Yu Zhang; Jasha Droppo; Kai Yu
PY - 2016
PB - IEEE SigPort
UR - http://sigport.org/1255
ER -
Tianxing He, Yu Zhang, Jasha Droppo, Kai Yu. (2016). On Training Bi-directional Neural Network Language Model with Noise Contrastive Estimation. IEEE SigPort. http://sigport.org/1255
Tianxing He, Yu Zhang, Jasha Droppo, Kai Yu, 2016. On Training Bi-directional Neural Network Language Model with Noise Contrastive Estimation. Available at: http://sigport.org/1255.
Tianxing He, Yu Zhang, Jasha Droppo, Kai Yu. (2016). "On Training Bi-directional Neural Network Language Model with Noise Contrastive Estimation." Web.
1. Tianxing He, Yu Zhang, Jasha Droppo, Kai Yu. On Training Bi-directional Neural Network Language Model with Noise Contrastive Estimation [Internet]. IEEE SigPort; 2016. Available from : http://sigport.org/1255