Sorry, you need to enable JavaScript to visit this website.

facebooktwittermailshare

Emotion Classification: How Does an Automated System Compare to Naive Human Coders?

Abstract: 

The fact that emotions play a vital role in social interactions, along with the demand for novel human-computer interaction applications, have led to the development of a number of automatic emotion classification systems. However, it is still debatable whether the performance of such systems can compare with human coders. To address this issue, in this study, we present a comprehensive comparison in a speech-based emotion classification task between 138 Amazon Mechanical Turk workers (Turkers) and a state-of-the-art automatic computer system. The comparison includes classifying speech utterances into six emotions (happy, neutral, sad, anger, disgust and fear), into three arousal classes (active, passive, and neutral), and into three valence classes (positive, negative, and neutral). The results show that the computer system outperforms the naive Turkers in almost all cases. Furthermore, the computer system can increase the classification accuracy by rejecting to classify utterances for which it is not confident, while the Turkers do not show a significantly higher classification accuracy on their confident utterances versus unconfident ones.

up
0 users have voted:

Paper Details

Authors:
Kenneth Imade, Na Yang, Melissa Sturge-Apple, Zhiyao Duan, Wendi Heinzelman
Submitted On:
17 March 2016 - 3:26pm
Short Link:
Type:
Presentation Slides
Event:
Presenter's Name:
Wendi Heinzelman
Document Year:
2016
Cite

Document Files

EmotionICASSP16.pdf

(582)

Subscribe

[1] Kenneth Imade, Na Yang, Melissa Sturge-Apple, Zhiyao Duan, Wendi Heinzelman, "Emotion Classification: How Does an Automated System Compare to Naive Human Coders?", IEEE SigPort, 2016. [Online]. Available: http://sigport.org/749. Accessed: Sep. 16, 2019.
@article{749-16,
url = {http://sigport.org/749},
author = {Kenneth Imade; Na Yang; Melissa Sturge-Apple; Zhiyao Duan; Wendi Heinzelman },
publisher = {IEEE SigPort},
title = {Emotion Classification: How Does an Automated System Compare to Naive Human Coders?},
year = {2016} }
TY - EJOUR
T1 - Emotion Classification: How Does an Automated System Compare to Naive Human Coders?
AU - Kenneth Imade; Na Yang; Melissa Sturge-Apple; Zhiyao Duan; Wendi Heinzelman
PY - 2016
PB - IEEE SigPort
UR - http://sigport.org/749
ER -
Kenneth Imade, Na Yang, Melissa Sturge-Apple, Zhiyao Duan, Wendi Heinzelman. (2016). Emotion Classification: How Does an Automated System Compare to Naive Human Coders?. IEEE SigPort. http://sigport.org/749
Kenneth Imade, Na Yang, Melissa Sturge-Apple, Zhiyao Duan, Wendi Heinzelman, 2016. Emotion Classification: How Does an Automated System Compare to Naive Human Coders?. Available at: http://sigport.org/749.
Kenneth Imade, Na Yang, Melissa Sturge-Apple, Zhiyao Duan, Wendi Heinzelman. (2016). "Emotion Classification: How Does an Automated System Compare to Naive Human Coders?." Web.
1. Kenneth Imade, Na Yang, Melissa Sturge-Apple, Zhiyao Duan, Wendi Heinzelman. Emotion Classification: How Does an Automated System Compare to Naive Human Coders? [Internet]. IEEE SigPort; 2016. Available from : http://sigport.org/749