Needle in a Haystack: Spotting and recognising micro-expressions “in the wild”

Y. S. Gan, John See, Huai-Qian Khor, Kun Hong Liu, Sze-Teng Liong*

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

Abstract

Computational research on facial micro-expressions has long focused on videos captured under constrained laboratory conditions due to the challenging elicitation process and limited samples that are publicly available. Moreover, processing micro-expressions is extremely challenging under unconstrained scenarios. This paper introduces, for the first time, a completely automatic micro-expression “spot-and-recognize” framework that is performed on in-the-wild videos, such as in poker games and political interviews. The proposed method first spots the apex frame from a video by handling head movements and unconscious actions which are typically larger in motion intensity, with alignment employed to enforce a canonical face pose. Optical flow guided features play a central role in our method: they can robustly identify the location of the apex frame, and are used to learn a shallow neural network model for emotion classification. Experimental results demonstrate the feasibility of the proposed methodology, establishing good baselines for both spotting and recognition tasks – ASR of 0.33 and F1-score of 0.6758 respectively on the MEVIEW micro-expression database. In addition, we present comprehensive qualitative and quantitative analyses to further show the effectiveness of the proposed framework, with new suggestion for an appropriate evaluation protocol. In a nutshell, this paper provides a new benchmark for apex spotting and emotion recognition in an in-the-wild setting.

Original languageEnglish
Pages (from-to)283-298
Number of pages16
JournalNeurocomputing
Volume503
Early online date30 Jun 2022
DOIs
Publication statusPublished - 7 Sep 2022

Keywords

  • Apex frame
  • Face alignment
  • In-the-wild
  • Micro-expression recognition
  • Micro-expression spotting

ASJC Scopus subject areas

  • Computer Science Applications
  • Cognitive Neuroscience
  • Artificial Intelligence

Fingerprint

Dive into the research topics of 'Needle in a Haystack: Spotting and recognising micro-expressions “in the wild”'. Together they form a unique fingerprint.

Cite this