Version 1
: Received: 27 August 2020 / Approved: 28 August 2020 / Online: 28 August 2020 (15:05:37 CEST)
How to cite:
Boigne, J.; Liyanage, B.; Östrem, T. Recognizing More Emotions with Less Data Using Self-supervised Transfer Learning. Preprints2020, 2020080645. https://doi.org/10.20944/preprints202008.0645.v1
Boigne, J.; Liyanage, B.; Östrem, T. Recognizing More Emotions with Less Data Using Self-supervised Transfer Learning. Preprints 2020, 2020080645. https://doi.org/10.20944/preprints202008.0645.v1
Boigne, J.; Liyanage, B.; Östrem, T. Recognizing More Emotions with Less Data Using Self-supervised Transfer Learning. Preprints2020, 2020080645. https://doi.org/10.20944/preprints202008.0645.v1
APA Style
Boigne, J., Liyanage, B., & Östrem, T. (2020). Recognizing More Emotions with Less Data Using Self-supervised Transfer Learning. Preprints. https://doi.org/10.20944/preprints202008.0645.v1
Chicago/Turabian Style
Boigne, J., Biman Liyanage and Ted Östrem. 2020 "Recognizing More Emotions with Less Data Using Self-supervised Transfer Learning" Preprints. https://doi.org/10.20944/preprints202008.0645.v1
Abstract
We propose a novel transfer learning method for speech emotion recognition allowing us to obtain promising results when only few training data is available. With as low as 125 examples per emotion class, we were able to reach a higher accuracy than a strong baseline trained on 8 times more data. Our method leverages knowledge contained in pre-trained speech representations extracted from models trained on a more general self-supervised task which doesn’t require human annotations, such as the wav2vec model. We provide detailed insights on the benefits of our approach by varying the training data size, which can help labeling teams to work more efficiently. We compare performance with other popular methods on the IEMOCAP dataset, a well-benchmarked dataset among the Speech Emotion Recognition (SER) research community. Furthermore, we demonstrate that results can be greatly improved by combining acoustic and linguistic knowledge from transfer learning. We align acoustic pre-trained representations with semantic representations from the BERT model through an attention-based recurrent neural network. Performance improves significantly when combining both modalities and scales with the amount of data. When trained on the full IEMOCAP dataset, we reach a new state-of-the-art of 73.9% unweighted accuracy (UA).
Computer Science and Mathematics, Artificial Intelligence and Machine Learning
Copyright:
This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
