Although uni-directional recurrent neural network language
model(RNNLM) has been very successful, it’s hard to train a
bi-directional RNNLM properly due to the generative nature of
language model. In this work, we propose to train bi-directional
RNNLM with noise contrastive estimation(NCE), since the
properities of NCE training will help the model to acheieve
sentence-level normalization. Experiments are conducted on
two hand-crafted tasks on the PTB data set: a rescore task and
a sanity test. Although(regretfully), the model trained by NCE
did not out-perform the baseline uni-directional NNLM, it is
shown that NCE-trained bi-directional NNLM behaves well in
the sanity test and outperformed the one trained by conventional
maximum likelihood training on the rescore task.