XAI for Facial Emotion Recognition: Quantitative Analysis via Grad-CAM and SHAP

Abstract

Automated facial emotion recognition (FER) forms the basis of affective computing and mental healthmeasurement, as well as human-computer interaction. However, there is one critical and understudied issue: deep learningmodels are the state of the art, and they are very accurate, but do not offer any idea regarding what facial cues influence theirchoice, thus highly constraining its clinical and educational use. We close the gap by developing an integrated ExplainableFER framework that one can achieve by fine-tuning a ResNet-50 convolutional neural network using a fine-tuned ResNet-50convolutional neural network, and applies two additional complementary attribing methods to interpretability to GradientweightedClass Activation Mapping (Grad-CAM) and Integrated Gradients (IG) to provide high predictive performance andexplanations that can be easily interpreted by clinicians. The model was trained and tested on a subject-independent protocolon the Extended Cohn-Kanade (CK+) dataset [18], which consisted of 22400 images generated by 123 individuals in eightratioed emotion pluses with controlled offline augmented subject isolation. The model has been optimized using Adam andStepLR and achieved test accuracy of 99.955% (2,239/2,240 correct) and macro F1 of 0.9996- the best reported on CK+under XAI-integrated works. Grad-CAM visualization showed semantically meaningful spatial attention consistent withaction units in Facial Action Coding System (FACS) and IG made possible a new quantitative Facial Region ImportanceMatrix of seven anatomical regions in eight emotion categories. Each component contribution is proved by an ablation study.It may be deployed in a single open-source Google Colab notebook.