MTSN: A Multi-Temporal Stream Network for Spotting Facial Macro- and Micro-Expression with Hard and Soft Pseudo-labels

Gen Bing Liong, Sze-Teng Liong, John See, Chee-Seng Chan

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

This paper considers the challenge of spotting facial macro- and micro-expression from long videos.We propose the multi-temporal stream network (MTSN) model that takes two distinct inputs by considering the different temporal information in the facial movement. We also introduce a hard and soft pseudo-labeling technique to enable the network to distinguish expression frames from nonexpression frames via the learning of salient features in the expression peak frame. Consequently, we demonstrate how a single output from the MTSN model can be post-processed to predict both macro- and micro-expression intervals. Our results outperform the MEGC 2022 baseline method significantly by achieving an overall F1-score of 0.2586 and also did remarkably well on the MEGC 2021 benchmark with an overall F1-score of 0.3620 and 0.2867 on CAS(ME)2and SAMM Long Videos, respectively.

Original languageEnglish
Title of host publicationFME '22: Proceedings of the 2nd Workshop on Facial Micro-Expression
Subtitle of host publicationAdvanced Techniques for Multi-Modal Facial Expression
PublisherAssociation for Computing Machinery
Pages3-10
Number of pages8
ISBN (Electronic)9781450394956
DOIs
Publication statusPublished - 10 Oct 2022
Event2nd Workshop on Facial Micro-Expression: Advanced Techniques for Multi-Modal Facial Expression Analysis 2022 - Lisboa, Portugal
Duration: 14 Oct 2022 → …

Conference

Conference2nd Workshop on Facial Micro-Expression: Advanced Techniques for Multi-Modal Facial Expression Analysis 2022
Abbreviated titleFME 2022
Country/TerritoryPortugal
CityLisboa
Period14/10/22 → …

Keywords

  • emotion analysis
  • macro-expression
  • Micro-expression
  • spotting

ASJC Scopus subject areas

  • Computational Theory and Mathematics
  • Computer Science Applications
  • Computer Vision and Pattern Recognition
  • Software

Fingerprint

Dive into the research topics of 'MTSN: A Multi-Temporal Stream Network for Spotting Facial Macro- and Micro-Expression with Hard and Soft Pseudo-labels'. Together they form a unique fingerprint.

Cite this