Sorry, you need to enable JavaScript to visit this website.

facebooktwittermailshare

Knowledge Distillation for Small-footprint Highway Networks

Abstract: 

Deep learning has significantly advanced state-of-the-art of speech
recognition in the past few years. However, compared to conventional
Gaussian mixture acoustic models, neural network models are
usually much larger, and are therefore not very deployable in embedded
devices. Previously, we investigated a compact highway deep
neural network (HDNN) for acoustic modelling, which is a type
of depth-gated feedforward neural network. We have shown that
HDNN-based acoustic models can achieve comparable recognition
accuracy with much smaller number of model parameters compared
to plain deep neural network (DNN) acoustic models. In this paper,
we push the boundary further by leveraging on the knowledge
distillation technique that is also known as teacher-student training,
i.e., we train the compact HDNN model with the supervision of a
high accuracy cumbersome model. Furthermore, we also investigate
sequence training and adaptation in the context of teacher-student
training. Our experiments were performed on the AMI meeting
speech recognition corpus. With this technique, we significantly improved
the recognition accuracy of the HDNN acoustic model with
less than 0.8 million parameters, and narrowed the gap between this
model and the plain DNN with 30 million parameters.

up
0 users have voted:

Paper Details

Authors:
Liang Lu, Michelle Guo, Steve Renals
Submitted On:
3 March 2017 - 5:15pm
Short Link:
Type:
Presentation Slides
Event:
Presenter's Name:
Liang Lu
Paper Code:
SP-L1.4
Document Year:
2017
Cite

Document Files

Slides for ICASSP 2017

(265)

Subscribe

[1] Liang Lu, Michelle Guo, Steve Renals, "Knowledge Distillation for Small-footprint Highway Networks", IEEE SigPort, 2017. [Online]. Available: http://sigport.org/1619. Accessed: Jul. 09, 2020.
@article{1619-17,
url = {http://sigport.org/1619},
author = {Liang Lu; Michelle Guo; Steve Renals },
publisher = {IEEE SigPort},
title = {Knowledge Distillation for Small-footprint Highway Networks},
year = {2017} }
TY - EJOUR
T1 - Knowledge Distillation for Small-footprint Highway Networks
AU - Liang Lu; Michelle Guo; Steve Renals
PY - 2017
PB - IEEE SigPort
UR - http://sigport.org/1619
ER -
Liang Lu, Michelle Guo, Steve Renals. (2017). Knowledge Distillation for Small-footprint Highway Networks. IEEE SigPort. http://sigport.org/1619
Liang Lu, Michelle Guo, Steve Renals, 2017. Knowledge Distillation for Small-footprint Highway Networks. Available at: http://sigport.org/1619.
Liang Lu, Michelle Guo, Steve Renals. (2017). "Knowledge Distillation for Small-footprint Highway Networks." Web.
1. Liang Lu, Michelle Guo, Steve Renals. Knowledge Distillation for Small-footprint Highway Networks [Internet]. IEEE SigPort; 2017. Available from : http://sigport.org/1619