MTSN: A Multi-Temporal Stream Network for Spotting Facial Macro- and Micro-Expression with Hard and Soft Pseudo-labels

Gen Bing Liong; Sze-Teng Liong; John See; Chee-Seng Chan

doi:10.1145/3552465.3555040

MTSN: A Multi-Temporal Stream Network for Spotting Facial Macro- and Micro-Expression with Hard and Soft Pseudo-labels

Gen Bing Liong, Sze-Teng Liong, John See, Chee-Seng Chan

School of Mathematical & Computer Sciences

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

16 Citations (Scopus)

Abstract

This paper considers the challenge of spotting facial macro- and micro-expression from long videos.We propose the multi-temporal stream network (MTSN) model that takes two distinct inputs by considering the different temporal information in the facial movement. We also introduce a hard and soft pseudo-labeling technique to enable the network to distinguish expression frames from nonexpression frames via the learning of salient features in the expression peak frame. Consequently, we demonstrate how a single output from the MTSN model can be post-processed to predict both macro- and micro-expression intervals. Our results outperform the MEGC 2022 baseline method significantly by achieving an overall F1-score of 0.2586 and also did remarkably well on the MEGC 2021 benchmark with an overall F1-score of 0.3620 and 0.2867 on CAS(ME)²and SAMM Long Videos, respectively.

Original language	English
Title of host publication	FME '22: Proceedings of the 2nd Workshop on Facial Micro-Expression
Subtitle of host publication	Advanced Techniques for Multi-Modal Facial Expression
Publisher	Association for Computing Machinery
Pages	3-10
Number of pages	8
ISBN (Electronic)	9781450394956
DOIs	https://doi.org/10.1145/3552465.3555040
Publication status	Published - 10 Oct 2022
Event	2nd Workshop on Facial Micro-Expression: Advanced Techniques for Multi-Modal Facial Expression Analysis 2022 - Lisboa, Portugal Duration: 14 Oct 2022 → …

Conference

Conference	2nd Workshop on Facial Micro-Expression: Advanced Techniques for Multi-Modal Facial Expression Analysis 2022
Abbreviated title	FME 2022
Country/Territory	Portugal
City	Lisboa
Period	14/10/22 → …

Keywords

emotion analysis
macro-expression
Micro-expression
spotting

ASJC Scopus subject areas

Computational Theory and Mathematics
Computer Science Applications
Computer Vision and Pattern Recognition
Software

Access to Document

10.1145/3552465.3555040

Cite this

Liong, G. B., Liong, S.-T., See, J., & Chan, C.-S. (2022). MTSN: A Multi-Temporal Stream Network for Spotting Facial Macro- and Micro-Expression with Hard and Soft Pseudo-labels. In FME '22: Proceedings of the 2nd Workshop on Facial Micro-Expression: Advanced Techniques for Multi-Modal Facial Expression (pp. 3-10). Association for Computing Machinery. https://doi.org/10.1145/3552465.3555040

@inproceedings{c23d470104d74e1db59e512e3c97e145,

title = "MTSN: A Multi-Temporal Stream Network for Spotting Facial Macro- and Micro-Expression with Hard and Soft Pseudo-labels",

abstract = "This paper considers the challenge of spotting facial macro- and micro-expression from long videos.We propose the multi-temporal stream network (MTSN) model that takes two distinct inputs by considering the different temporal information in the facial movement. We also introduce a hard and soft pseudo-labeling technique to enable the network to distinguish expression frames from nonexpression frames via the learning of salient features in the expression peak frame. Consequently, we demonstrate how a single output from the MTSN model can be post-processed to predict both macro- and micro-expression intervals. Our results outperform the MEGC 2022 baseline method significantly by achieving an overall F1-score of 0.2586 and also did remarkably well on the MEGC 2021 benchmark with an overall F1-score of 0.3620 and 0.2867 on CAS(ME)2and SAMM Long Videos, respectively.",

keywords = "emotion analysis, macro-expression, Micro-expression, spotting",

author = "Liong, {Gen Bing} and Sze-Teng Liong and John See and Chee-Seng Chan",

note = "Funding Information: Acknowledgments. This work is supported in part by HWUM JWS2022/01 funding, Heriot-Watt University Malaysia. Publisher Copyright: {\textcopyright} 2022 IEEE. All rights reserved.; 2nd Workshop on Facial Micro-Expression: Advanced Techniques for Multi-Modal Facial Expression Analysis 2022, FME 2022 ; Conference date: 14-10-2022",

year = "2022",

month = oct,

day = "10",

doi = "10.1145/3552465.3555040",

language = "English",

pages = "3--10",

booktitle = "FME '22: Proceedings of the 2nd Workshop on Facial Micro-Expression",

publisher = "Association for Computing Machinery",

address = "United States",

}

Liong, GB, Liong, S-T, See, J & Chan, C-S 2022, MTSN: A Multi-Temporal Stream Network for Spotting Facial Macro- and Micro-Expression with Hard and Soft Pseudo-labels. in FME '22: Proceedings of the 2nd Workshop on Facial Micro-Expression: Advanced Techniques for Multi-Modal Facial Expression. Association for Computing Machinery, pp. 3-10, 2nd Workshop on Facial Micro-Expression: Advanced Techniques for Multi-Modal Facial Expression Analysis 2022, Lisboa, Portugal, 14/10/22. https://doi.org/10.1145/3552465.3555040

MTSN: A Multi-Temporal Stream Network for Spotting Facial Macro- and Micro-Expression with Hard and Soft Pseudo-labels. / Liong, Gen Bing; Liong, Sze-Teng; See, John et al.
FME '22: Proceedings of the 2nd Workshop on Facial Micro-Expression: Advanced Techniques for Multi-Modal Facial Expression. Association for Computing Machinery, 2022. p. 3-10.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - MTSN

T2 - 2nd Workshop on Facial Micro-Expression: Advanced Techniques for Multi-Modal Facial Expression Analysis 2022

AU - Liong, Gen Bing

AU - Liong, Sze-Teng

AU - See, John

AU - Chan, Chee-Seng

PY - 2022/10/10

Y1 - 2022/10/10

N2 - This paper considers the challenge of spotting facial macro- and micro-expression from long videos.We propose the multi-temporal stream network (MTSN) model that takes two distinct inputs by considering the different temporal information in the facial movement. We also introduce a hard and soft pseudo-labeling technique to enable the network to distinguish expression frames from nonexpression frames via the learning of salient features in the expression peak frame. Consequently, we demonstrate how a single output from the MTSN model can be post-processed to predict both macro- and micro-expression intervals. Our results outperform the MEGC 2022 baseline method significantly by achieving an overall F1-score of 0.2586 and also did remarkably well on the MEGC 2021 benchmark with an overall F1-score of 0.3620 and 0.2867 on CAS(ME)2and SAMM Long Videos, respectively.

AB - This paper considers the challenge of spotting facial macro- and micro-expression from long videos.We propose the multi-temporal stream network (MTSN) model that takes two distinct inputs by considering the different temporal information in the facial movement. We also introduce a hard and soft pseudo-labeling technique to enable the network to distinguish expression frames from nonexpression frames via the learning of salient features in the expression peak frame. Consequently, we demonstrate how a single output from the MTSN model can be post-processed to predict both macro- and micro-expression intervals. Our results outperform the MEGC 2022 baseline method significantly by achieving an overall F1-score of 0.2586 and also did remarkably well on the MEGC 2021 benchmark with an overall F1-score of 0.3620 and 0.2867 on CAS(ME)2and SAMM Long Videos, respectively.

KW - emotion analysis

KW - macro-expression

KW - Micro-expression

KW - spotting

UR - http://www.scopus.com/inward/record.url?scp=85141627554&partnerID=8YFLogxK

U2 - 10.1145/3552465.3555040

DO - 10.1145/3552465.3555040

M3 - Conference contribution

AN - SCOPUS:85141627554

SP - 3

EP - 10

BT - FME '22: Proceedings of the 2nd Workshop on Facial Micro-Expression

PB - Association for Computing Machinery

Y2 - 14 October 2022

ER -

MTSN: A Multi-Temporal Stream Network for Spotting Facial Macro- and Micro-Expression with Hard and Soft Pseudo-labels

Abstract

Conference

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this