Sorry, you need to enable JavaScript to visit this website.

facebooktwittermailshare

Knowledge Distillation Using Output Errors for Self-Attention ASR Models

Abstract: 

Most automatic speech recognition (ASR) neural network models are not suitable for mobile devices due to their large model sizes. Therefore, it is required to reduce the model size to meet the limited hardware resources. In this study, we investigate sequence-level knowledge distillation techniques of self-attention ASR models for model compression. In order to overcome the performance degradation of compressed models, our proposed method adds an exponential weight to the sequence-level knowledge distillation loss function, which reflects the word error rate of the output of the teacher model based on the ground-truth word sequences. Evaluated on LibriSpeech dataset, the proposed knowledge distillation method achieves significant improvements over the student baseline model.

up
0 users have voted:

Paper Details

Authors:
Hwidong Na, Hoshik Lee, Jihyun Lee, Tae Gyoon Kang, Min-Joong Lee, Young Sang Choi
Submitted On:
8 May 2019 - 10:02pm
Short Link:
Type:
Poster
Event:
Presenter's Name:
Hwidong Na
Document Year:
2019
Cite

Document Files

icassp-2019-poster_v1.1.pptx

(151)

Subscribe

[1] Hwidong Na, Hoshik Lee, Jihyun Lee, Tae Gyoon Kang, Min-Joong Lee, Young Sang Choi, "Knowledge Distillation Using Output Errors for Self-Attention ASR Models", IEEE SigPort, 2019. [Online]. Available: http://sigport.org/4140. Accessed: Jul. 09, 2020.
@article{4140-19,
url = {http://sigport.org/4140},
author = {Hwidong Na; Hoshik Lee; Jihyun Lee; Tae Gyoon Kang; Min-Joong Lee; Young Sang Choi },
publisher = {IEEE SigPort},
title = {Knowledge Distillation Using Output Errors for Self-Attention ASR Models},
year = {2019} }
TY - EJOUR
T1 - Knowledge Distillation Using Output Errors for Self-Attention ASR Models
AU - Hwidong Na; Hoshik Lee; Jihyun Lee; Tae Gyoon Kang; Min-Joong Lee; Young Sang Choi
PY - 2019
PB - IEEE SigPort
UR - http://sigport.org/4140
ER -
Hwidong Na, Hoshik Lee, Jihyun Lee, Tae Gyoon Kang, Min-Joong Lee, Young Sang Choi. (2019). Knowledge Distillation Using Output Errors for Self-Attention ASR Models. IEEE SigPort. http://sigport.org/4140
Hwidong Na, Hoshik Lee, Jihyun Lee, Tae Gyoon Kang, Min-Joong Lee, Young Sang Choi, 2019. Knowledge Distillation Using Output Errors for Self-Attention ASR Models. Available at: http://sigport.org/4140.
Hwidong Na, Hoshik Lee, Jihyun Lee, Tae Gyoon Kang, Min-Joong Lee, Young Sang Choi. (2019). "Knowledge Distillation Using Output Errors for Self-Attention ASR Models." Web.
1. Hwidong Na, Hoshik Lee, Jihyun Lee, Tae Gyoon Kang, Min-Joong Lee, Young Sang Choi. Knowledge Distillation Using Output Errors for Self-Attention ASR Models [Internet]. IEEE SigPort; 2019. Available from : http://sigport.org/4140