Uncovering Human Multimodal Activity Recognition with a Deep Learning Approach

Caetano M. Ranieri, Patricia A. Vargas, Roseli A. F. Romero

Research output: Chapter in Book/Report/Conference proceedingConference contribution

9 Citations (Scopus)
91 Downloads (Pure)


Recent breakthroughs on deep learning and computer vision have encouraged the use of multimodal human activity recognition aiming at applications in human-robot-interaction. The wide availability of videos at online platforms has made this modality one of the most promising for this task, whereas some researchers have tried to enhance the video data with wearable sensors attached to human subjects. However, temporal information on both video and inertial sensors are still under investigation. Most of the current work focusing on daily activities do not present comparative studies considering different temporal approaches. In this paper, we are proposing a new model build upon a Two-Stream ConvNet for action recognition, enhanced with Long Short-Term Memory (LSTM) and a Temporal Convolution Networks (TCN) to investigate the temporal information on videos and inertial sensors. A feature-level fusion approach prior to temporal modelling is also proposed and evaluated. Experiments have been conducted on the egocentric multimodal dataset and on the UTD-MHAD. LSTM and TCN showed competitive results, with the TCN performing slightly better for most applications. The feature-level fusion approach also performed well on the UTD-MHAD with some overfitting on the egocentric multimodal dataset. Overall the proposed model presented promising results on both datasets compatible with the state-of-the-art, providing insights on the use of deep learning for human-robot-interaction applications.
Original languageEnglish
Title of host publication2020 International Joint Conference on Neural Networks (IJCNN)
ISBN (Electronic)9781728169262
Publication statusPublished - 28 Sept 2020
EventIEEE World Congress on Computational Intelligence 2020 - Glasgow, United Kingdom
Duration: 19 Jul 202024 Jul 2020

Publication series

NameInternational Joint Conference on Neural Networks
ISSN (Electronic)2161-4407


ConferenceIEEE World Congress on Computational Intelligence 2020
Abbreviated titleIEEE WCCI 2020
Country/TerritoryUnited Kingdom
Internet address


  • CNN
  • Deep learning
  • LSTM
  • RNN
  • TCN
  • human activity recognition
  • human-robot-interaction

ASJC Scopus subject areas

  • Software
  • Artificial Intelligence


Dive into the research topics of 'Uncovering Human Multimodal Activity Recognition with a Deep Learning Approach'. Together they form a unique fingerprint.

Cite this