Sorry, you need to enable JavaScript to visit this website.

facebooktwittermailshare

Whole Sentence Neural Language Model

Abstract: 

Recurrent neural networks have become increasingly popular for the task of language modeling achieving impressive gains in state-of-the-art speech recognition and natural language processing (NLP) tasks. Recurrent models exploit word dependencies over a much longer context window (as retained by the history states) than what is feasible with n-gram language models. However the training criterion of choice for recurrent language models continues to be the local conditional likelihood of generating the current word given the (possibly long) word context, thus making local decisions at each word. This locally-conditional design fundamentally limits the ability of the model in exploiting whole sentence structures. In this paper, we present our initial results at whole sentence neural language models which assign a probability to the entire word sequence. We extend the previous work on whole sentence maximum entropy models to recurrent language models while using Noise Contrastive Estimation (NCE) for training, as these sentence models are fundamentally un-normalizable. We present results on a range of tasks: from sequence identification tasks such as, palindrome detection to large vocabulary automatic speech recognition (LVCSR) and demonstrate the modeling power of this approach.

up
0 users have voted:

Paper Details

Authors:
Abhinav Sethy, Kartik Audhkhasi, Bhuvana Ramabhadran
Submitted On:
20 April 2018 - 10:30pm
Short Link:
Type:
Presentation Slides
Event:
Presenter's Name:
Yinghui Huang
Paper Code:
HLT-L2.06
Document Year:
2018
Cite

Document Files

whole sentence neural language model

(226 downloads)

Subscribe

[1] Abhinav Sethy, Kartik Audhkhasi, Bhuvana Ramabhadran, "Whole Sentence Neural Language Model ", IEEE SigPort, 2018. [Online]. Available: http://sigport.org/3118. Accessed: Dec. 12, 2018.
@article{3118-18,
url = {http://sigport.org/3118},
author = {Abhinav Sethy; Kartik Audhkhasi; Bhuvana Ramabhadran },
publisher = {IEEE SigPort},
title = {Whole Sentence Neural Language Model },
year = {2018} }
TY - EJOUR
T1 - Whole Sentence Neural Language Model
AU - Abhinav Sethy; Kartik Audhkhasi; Bhuvana Ramabhadran
PY - 2018
PB - IEEE SigPort
UR - http://sigport.org/3118
ER -
Abhinav Sethy, Kartik Audhkhasi, Bhuvana Ramabhadran. (2018). Whole Sentence Neural Language Model . IEEE SigPort. http://sigport.org/3118
Abhinav Sethy, Kartik Audhkhasi, Bhuvana Ramabhadran, 2018. Whole Sentence Neural Language Model . Available at: http://sigport.org/3118.
Abhinav Sethy, Kartik Audhkhasi, Bhuvana Ramabhadran. (2018). "Whole Sentence Neural Language Model ." Web.
1. Abhinav Sethy, Kartik Audhkhasi, Bhuvana Ramabhadran. Whole Sentence Neural Language Model [Internet]. IEEE SigPort; 2018. Available from : http://sigport.org/3118