A simple but powerful language model called fixed-size
ordinally-forgetting encoding (FOFE) based feedforward neural
network language models (FNN-LMs) has been proposed recently.
Experimental results have shown that FOFE based FNNLMs
can outperform not only the standard FNN-LMs but also
the popular recurrent neural network language models (RNNLMs).
In this paper, we extend FOFE based FNN-LMs from
several aspects. Firstly, we have proposed a new method to
further improve the performance of FOFE based FNN-LMs by
adding transitions of part-of-speech (POS) tags as additional
features. Secondly, we have investigated how to speedup the
learning of FOFE based FNN-LMs by using noise contrastive
estimation (NCE). As a result, we can dramatically speedup
the learning of FOFE based FNN-LMs while we still achieve
very competitive experimental results on Large Text Compression
Benchmark (LTCB).