Deep Multi-Modal Network Based Automated Depression Severity Estimation

Md Azher Uddin, Joolekha Bibi Joolee, Kyung Ah Sohn

Research output: Contribution to journalArticlepeer-review

12 Citations (Scopus)
569 Downloads (Pure)


Depression is a severe mental illness that impairs a person's capacity to function normally in personal and professional life. The assessment of depression usually requires a comprehensive examination by an expert professional. Recently, machine learning-based automatic depression assessment has received considerable attention for a reliable and efficient depression diagnosis. Various techniques for automated depression detection were developed; however, certain concerns still need to be investigated. In this work, we propose a novel deep multi-modal framework that effectively utilizes facial and verbal cues for an automated depression assessment. Specifically, we first partition the audio and video data into fixed-length segments. Then, these segments are fed into the Spatio-Temporal Networks as input, which captures both spatial and temporal features as well as assigns higher weights to the features that contribute most. In addition, Volume Local Directional Structural Pattern (VLDSP) based dynamic feature descriptor is introduced to extract the facial dynamics by encoding the structural aspects. Afterwards, we employ the Temporal Attentive Pooling (TAP) approach to summarize the segment-level features for audio and video data. Finally, the multi-modal factorized bilinear pooling (MFB) strategy is applied to fuse the multi-modal features effectively. An extensive experimental study reveals that the proposed method outperforms state-of-the-art approaches.
Original languageEnglish
Pages (from-to)2153-2167
Number of pages15
JournalIEEE Transactions on Affective Computing
Issue number3
Early online date1 Jun 2022
Publication statusPublished - Jul 2023


  • Convolutional neural networks
  • Depression
  • Encoding
  • Feature extraction
  • Long short term memory
  • multi-modal factorized bilinear pooling
  • Optical flow
  • spatio-temporal networks
  • temporal attentive pooling
  • Three-dimensional displays
  • volume local directional structural pattern

ASJC Scopus subject areas

  • Software
  • Human-Computer Interaction


Dive into the research topics of 'Deep Multi-Modal Network Based Automated Depression Severity Estimation'. Together they form a unique fingerprint.

Cite this