Mixed Emotion Recognition Through Facial Expression using Transformer-Based Model

Abstract

Basic facial expressions such as Angry, Disgust, Surprised, Happy, Scared, and Sadness can express emotions. However, in conversations, compound emotions can form Mixed Facial emotions, a combination of basic emotions that is much more complex Mixed emotion recognition is a recent study that has not been researched enough, even though datasets already contain mixed emotions This research aims to implement and fine-tune Transformer-based models such as Vision Transformer, Swin Transformer v2, and ConvNet-based model such as ConvNeXt architecture to identify and recognize mixed emotions through human faces using the IMED Dataset Various configurations with fine-tuned hyperparameters are tested and vary between each model The result shows that Vision Transformer architecture outperform other models in Mixed Emotion Recognition from Facial expressions and reach up to 79.37% Testing accuracy compared to Swin Transformer v2 model with 65.36% Testing accuracy and ConvNext with 74.77% Testing accuracy.