Sorry, you need to enable JavaScript to visit this website.

On Training Bi-directional Neural Network Language Model with Noise Contrastive Estimation

Citation Author(s):
Tianxing He, Yu Zhang, Jasha Droppo, Kai Yu
Submitted by:
Tianxing He
Last updated:
16 October 2016 - 11:45am
Document Type:
Document Year:
Presenters Name:
Tianxing He
Paper Code:



Although uni-directional recurrent neural network language
model(RNNLM) has been very successful, it’s hard to train a
bi-directional RNNLM properly due to the generative nature of
language model. In this work, we propose to train bi-directional
RNNLM with noise contrastive estimation(NCE), since the
properities of NCE training will help the model to acheieve
sentence-level normalization. Experiments are conducted on
two hand-crafted tasks on the PTB data set: a rescore task and
a sanity test. Although(regretfully), the model trained by NCE
did not out-perform the baseline uni-directional NNLM, it is
shown that NCE-trained bi-directional NNLM behaves well in
the sanity test and outperformed the one trained by conventional
maximum likelihood training on the rescore task.

0 users have voted:

Dataset Files