Multi-scale Spatiotemporal Information Fusion Network for Video Action Recognition

Yutong Cai; Weiyao Lin; John See; Ming-Ming Cheng; Guangcan Liu; Hongkai Xiong

doi:10.1109/VCIP.2018.8698676

Multi-scale Spatiotemporal Information Fusion Network for Video Action Recognition

Yutong Cai, Weiyao Lin^*, John See, Ming-Ming Cheng, Guangcan Liu, Hongkai Xiong

^*Corresponding author for this work

School of Mathematical & Computer Sciences

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

4 Citations (Scopus)

Abstract

Two-stream convolutional networks have shown excellent performance in video action recognition in recent years. However, it remains unclear how to model the correlation between the temporal and spatial streams more effectively. First, the spatial stream and temporal stream pay attention to different aspects, which can lead to different recognition results. Second, the variety in the length of optical flow fields tends to have a great impact on the classification results. In this paper, we propose a novel multi-scale spatiotemporal information fusion network to fuse the spatial and temporal features. Specifically, our network takes advantage of multi-scale temporal information to better utilize the motion cues. Considering the complementary relationship between the spatial and temporal features, we take the hierarchical fusion strategies and asynchronous fusion method to fuse the two-stream features. Experimental results on two benchmark datasets (UCF101 and HMDB51) show that the proposed network achieves competitive performance.

Original language	English
Title of host publication	2018 IEEE Visual Communications and Image Processing (VCIP)
Publisher	IEEE
ISBN (Electronic)	9781538644584
DOIs	https://doi.org/10.1109/VCIP.2018.8698676
Publication status	Published - 25 Apr 2019
Event	33rd IEEE International Conference on Visual Communications and Image Processing 2018 - Taichung, Taiwan, Province of China Duration: 9 Dec 2018 → 12 Dec 2018

Conference

Conference	33rd IEEE International Conference on Visual Communications and Image Processing 2018
Abbreviated title	VCIP 2018
Country/Territory	Taiwan, Province of China
City	Taichung
Period	9/12/18 → 12/12/18

Keywords

Action recognition
Asynchronous fusion
Convolutional network
Hierarchical fusion strategies
Multi-scale spatiotemporal information

ASJC Scopus subject areas

Computer Networks and Communications
Signal Processing

Access to Document

10.1109/VCIP.2018.8698676

Cite this

@inproceedings{559bb0b67387429da501a837eabe6038,

title = "Multi-scale Spatiotemporal Information Fusion Network for Video Action Recognition",

abstract = "Two-stream convolutional networks have shown excellent performance in video action recognition in recent years. However, it remains unclear how to model the correlation between the temporal and spatial streams more effectively. First, the spatial stream and temporal stream pay attention to different aspects, which can lead to different recognition results. Second, the variety in the length of optical flow fields tends to have a great impact on the classification results. In this paper, we propose a novel multi-scale spatiotemporal information fusion network to fuse the spatial and temporal features. Specifically, our network takes advantage of multi-scale temporal information to better utilize the motion cues. Considering the complementary relationship between the spatial and temporal features, we take the hierarchical fusion strategies and asynchronous fusion method to fuse the two-stream features. Experimental results on two benchmark datasets (UCF101 and HMDB51) show that the proposed network achieves competitive performance.",

keywords = "Action recognition, Asynchronous fusion, Convolutional network, Hierarchical fusion strategies, Multi-scale spatiotemporal information",

author = "Yutong Cai and Weiyao Lin and John See and Ming-Ming Cheng and Guangcan Liu and Hongkai Xiong",

note = "Funding Information: ACKNOWLEDGMENT This work is supported by NSFC (61471235, 61622305, 61502238), Shanghai {\textquoteright}The Belt and Road{\textquoteright} Young Scholar Exchange Grant (17510740100), and NSFJPC BK20141003. Publisher Copyright: {\textcopyright} 2018 IEEE. Copyright: Copyright 2019 Elsevier B.V., All rights reserved.; 33rd IEEE International Conference on Visual Communications and Image Processing 2018, VCIP 2018 ; Conference date: 09-12-2018 Through 12-12-2018",

year = "2019",

month = apr,

day = "25",

doi = "10.1109/VCIP.2018.8698676",

language = "English",

booktitle = "2018 IEEE Visual Communications and Image Processing (VCIP)",

publisher = "IEEE",

address = "United States",

}

Cai, Y, Lin, W, See, J, Cheng, M-M, Liu, G & Xiong, H 2019, Multi-scale Spatiotemporal Information Fusion Network for Video Action Recognition. in 2018 IEEE Visual Communications and Image Processing (VCIP)., 8698676, IEEE, 33rd IEEE International Conference on Visual Communications and Image Processing 2018, Taichung, Taiwan, Province of China, 9/12/18. https://doi.org/10.1109/VCIP.2018.8698676

TY - GEN

T1 - Multi-scale Spatiotemporal Information Fusion Network for Video Action Recognition

AU - Cai, Yutong

AU - Lin, Weiyao

AU - See, John

AU - Cheng, Ming-Ming

AU - Liu, Guangcan

AU - Xiong, Hongkai

N1 - Funding Information: ACKNOWLEDGMENT This work is supported by NSFC (61471235, 61622305, 61502238), Shanghai ’The Belt and Road’ Young Scholar Exchange Grant (17510740100), and NSFJPC BK20141003. Publisher Copyright: © 2018 IEEE. Copyright: Copyright 2019 Elsevier B.V., All rights reserved.

PY - 2019/4/25

Y1 - 2019/4/25

N2 - Two-stream convolutional networks have shown excellent performance in video action recognition in recent years. However, it remains unclear how to model the correlation between the temporal and spatial streams more effectively. First, the spatial stream and temporal stream pay attention to different aspects, which can lead to different recognition results. Second, the variety in the length of optical flow fields tends to have a great impact on the classification results. In this paper, we propose a novel multi-scale spatiotemporal information fusion network to fuse the spatial and temporal features. Specifically, our network takes advantage of multi-scale temporal information to better utilize the motion cues. Considering the complementary relationship between the spatial and temporal features, we take the hierarchical fusion strategies and asynchronous fusion method to fuse the two-stream features. Experimental results on two benchmark datasets (UCF101 and HMDB51) show that the proposed network achieves competitive performance.

AB - Two-stream convolutional networks have shown excellent performance in video action recognition in recent years. However, it remains unclear how to model the correlation between the temporal and spatial streams more effectively. First, the spatial stream and temporal stream pay attention to different aspects, which can lead to different recognition results. Second, the variety in the length of optical flow fields tends to have a great impact on the classification results. In this paper, we propose a novel multi-scale spatiotemporal information fusion network to fuse the spatial and temporal features. Specifically, our network takes advantage of multi-scale temporal information to better utilize the motion cues. Considering the complementary relationship between the spatial and temporal features, we take the hierarchical fusion strategies and asynchronous fusion method to fuse the two-stream features. Experimental results on two benchmark datasets (UCF101 and HMDB51) show that the proposed network achieves competitive performance.

KW - Action recognition

KW - Asynchronous fusion

KW - Convolutional network

KW - Hierarchical fusion strategies

KW - Multi-scale spatiotemporal information

UR - http://www.scopus.com/inward/record.url?scp=85065428409&partnerID=8YFLogxK

U2 - 10.1109/VCIP.2018.8698676

DO - 10.1109/VCIP.2018.8698676

M3 - Conference contribution

AN - SCOPUS:85065428409

BT - 2018 IEEE Visual Communications and Image Processing (VCIP)

PB - IEEE

T2 - 33rd IEEE International Conference on Visual Communications and Image Processing 2018

Y2 - 9 December 2018 through 12 December 2018

ER -

Multi-scale Spatiotemporal Information Fusion Network for Video Action Recognition

Abstract

Conference

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this