SFAMNet: A scene flow attention-based micro-expression network

Gen-Bing Liong, Sze-Teng Liong, Chee Seng Chan, John See

Research output: Contribution to journalArticlepeer-review

Abstract

Tremendous progress has been made in facial Micro-Expression (ME) spotting and recognition; however, most works have focused on either spotting or recognition tasks on the 2D videos. Until recently, the estimation of the 3D motion field (a.k.a scene flow) for the ME has only become possible after the release of the multi-modal ME dataset. In this paper, we propose the first Scene Flow Attention-based Micro-expression Network, namely SFAMNet. It takes the scene flow computed using the RGB-D flow algorithm as the input and predicts the spotting confidence score and emotion labels. Specifically, SFAMNet is an attention-based end-to-end multi-stream multi-task network devised to spot and recognize the ME. Besides that, we present a data augmentation strategy to alleviate the small sample size problem during network learning. Extensive experiments are performed on three tasks: (i) ME spotting; (ii) ME recognition; and (iii) ME analysis on the multi-modal CAS(ME)dataset. Empirical results indicate that depth is vital in capturing the ME information and the effectiveness of the proposed approach. Our source code is publicly available at https://github.com/genbing99/SFAMNet.
Original languageEnglish
Article number126998
JournalNeurocomputing
Volume566
Early online date4 Nov 2023
DOIs
Publication statusPublished - 21 Jan 2024

Keywords

  • Analysis
  • Attention
  • Facial micro-expression
  • Recognition
  • Scene flow
  • Spotting

ASJC Scopus subject areas

  • Artificial Intelligence
  • Cognitive Neuroscience
  • Computer Science Applications

Fingerprint

Dive into the research topics of 'SFAMNet: A scene flow attention-based micro-expression network'. Together they form a unique fingerprint.

Cite this