Evaluating significant features in context‐aware multimodal emotion recognition with XAI methods

Aaishwarya Khalane, Rikesh Makwana, Talal Shaikh*, Abrar Ullah

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

3 Citations (Scopus)
57 Downloads (Pure)


Expert systems are being extensively used to make critical decisions involving emotional analysis in affective computing. The evolution of deep learning algorithms has improved the potential for extracting value from multimodal emotional data. However, these black‐box algorithms do not often explain the heuristics behind processing the input features for achieving certain outputs. This study focuses on the risks of using black‐box deep learning models for critical tasks, such as emotion recognition, and describes how human understandable interpretations of the workings of these models are extremely important. This study utilizes one of the largest multimodal datasets available–CMU‐MOSEI. Many researchers have used the pre‐extracted features provided by the CMU Multimodal SDK with black‐box deep learning models making it difficult to interpret the contribution of its individual features. This study describes the implications of significant features from various modalities (audio, video, text) identified using XAI in Multimodal Emotion Recognition. It describes the process of curating reduced feature models by using the Gradient SHAP XAI method. These reduced models with highly contributing features achieve comparable and at times even better results compared to their corresponding all‐feature models as well as the baseline model GraphMFN. This study reveals that carefully selecting significant features for a model can help filter out irrelevant features, and attenuate the noise or bias caused by them, leading to an improved performance efficiency of the expert systems by making them transparent, easily interpretable, and trustworthy.
Original languageEnglish
Article numbere13403
JournalExpert Systems
Early online date8 Aug 2023
Publication statusE-pub ahead of print - 8 Aug 2023


  • BERT
  • XAI
  • explainable artificial intelligence
  • interpretability
  • multimodal emotion recognition

ASJC Scopus subject areas

  • Control and Systems Engineering
  • Theoretical Computer Science
  • Computational Theory and Mathematics
  • Artificial Intelligence


Dive into the research topics of 'Evaluating significant features in context‐aware multimodal emotion recognition with XAI methods'. Together they form a unique fingerprint.

Cite this