Sorry, you need to enable JavaScript to visit this website.

facebooktwittermailshare

Robust End-To-End Keyword Spotting And Voice Command Recognition For Mobile Game

Abstract: 

We present an effective method to solve a small-footprint keyword spotting (KWS) and voice command based user interface for mobile game. For KWS task, our goal is to design and implement a computationally very light deep neural network model into mobile device, in the same time to improve the accuracy in various noisy environments. We propose a simple yet effective convolutional neural network (CNN) with Google’s tensorflow-lite for android and Apple’s core ML for iOS deployment. Tensorflow provides post training integer quantization and with tensorflow-lite we could deploy 8-bit quantization model into android device. Meanwhile, we choose Apple’s core ML for it’s better performance than tensorflow-lite for iOS device. The size of our CNN model is 0.2 MB with 7 MB memory usage totally, and the CPU usage is around 1% (test phone: Galaxy S8 and iPhone 8). To improve the overall accuracy of KWS in noisy environments, we design a hybrid thresholding method, which make use of both average inference score and volume of incoming speech signal. We also propose a voice command SDK which running game command recognition on-the-fly inside the mobile device as well. We design a convolutional recurrent neural network transducer (CNN-RNN-T) as our automatic speech recognition (ASR) model. Text-to-speech (TTS) was applied to generate game command voices with various data augmentation techniques. Our CNN-RNN-T’s character error rate (CER) is 17%, the model size is 7 MB and the processing time is less than 1 second for 3 seconds incoming speech signal, which provides real-time voice command recognition for further game actions. To the best of our knowledge, this is the first work try to resolve both KWS and voice command recognition running inside device for mobile game. We will perform live demonstration of KWS and game command based user interface integrated into a full-sized, production-quality mobile game A3: Still Alive, which is one of the major games from Netmarble this year and will be available on market soon.

up
0 users have voted:

Paper Details

Authors:
Hu Xu*, Youshin Lim*, Shounan An, Hyemin Cho, Yoonseok Hong, Insoo Oh
Submitted On:
13 May 2020 - 11:20pm
Short Link:
Type:
Presentation Slides
Event:
Presenter's Name:
Shounan An
Paper Code:
S&T-P4
Document Year:
2020
Cite

Document Files

[S&T]ROBUST END-TO-END KEYWORD SPOTTING AND VOICE COMMAND RECOGNITION FOR MOBILE GAME_icassp20'.pdf

(163)

Subscribe

[1] Hu Xu*, Youshin Lim*, Shounan An, Hyemin Cho, Yoonseok Hong, Insoo Oh, "Robust End-To-End Keyword Spotting And Voice Command Recognition For Mobile Game", IEEE SigPort, 2020. [Online]. Available: http://sigport.org/5213. Accessed: Dec. 02, 2020.
@article{5213-20,
url = {http://sigport.org/5213},
author = {Hu Xu*; Youshin Lim*; Shounan An; Hyemin Cho; Yoonseok Hong; Insoo Oh },
publisher = {IEEE SigPort},
title = {Robust End-To-End Keyword Spotting And Voice Command Recognition For Mobile Game},
year = {2020} }
TY - EJOUR
T1 - Robust End-To-End Keyword Spotting And Voice Command Recognition For Mobile Game
AU - Hu Xu*; Youshin Lim*; Shounan An; Hyemin Cho; Yoonseok Hong; Insoo Oh
PY - 2020
PB - IEEE SigPort
UR - http://sigport.org/5213
ER -
Hu Xu*, Youshin Lim*, Shounan An, Hyemin Cho, Yoonseok Hong, Insoo Oh. (2020). Robust End-To-End Keyword Spotting And Voice Command Recognition For Mobile Game. IEEE SigPort. http://sigport.org/5213
Hu Xu*, Youshin Lim*, Shounan An, Hyemin Cho, Yoonseok Hong, Insoo Oh, 2020. Robust End-To-End Keyword Spotting And Voice Command Recognition For Mobile Game. Available at: http://sigport.org/5213.
Hu Xu*, Youshin Lim*, Shounan An, Hyemin Cho, Yoonseok Hong, Insoo Oh. (2020). "Robust End-To-End Keyword Spotting And Voice Command Recognition For Mobile Game." Web.
1. Hu Xu*, Youshin Lim*, Shounan An, Hyemin Cho, Yoonseok Hong, Insoo Oh. Robust End-To-End Keyword Spotting And Voice Command Recognition For Mobile Game [Internet]. IEEE SigPort; 2020. Available from : http://sigport.org/5213