Sequential fusion of facial appearance and dynamics for depression recognition

Chen, Qian, Chaturvedi, Iti, Ji, Shaoxiong, and Cambria, Erik (2021) Sequential fusion of facial appearance and dynamics for depression recognition. Pattern Recognition Letters, 150. pp. 115-121.

[img] PDF (Published Version) - Published Version
Restricted to Repository staff only

View at Publisher Website:


In mental health assessment, it is validated that nonverbal cues like facial expressions can be indicative of depressive disorders. Recently, the multimodal fusion of facial appearance and dynamics based on convolutional neural networks has demonstrated encouraging performance in depression analysis. However, correlation and complementarity between different visual modalities have not been well studied in prior methods. In this paper, we propose a sequential fusion method for facial depression recognition. For mining the correlated and complementary depression patterns in multimodal learning, a chained-fusion mechanism is introduced to jointly learn facial appearance and dynamics in a unified framework. We show that such sequential fusion can provide a probabilistic perspective of the model correlation and complementarity between two different data modalities for improved depression recognition. Results on a benchmark dataset show the superiority of our method against several state-of-the-art alternatives.

Item ID: 68789
Item Type: Article (Research - C1)
ISSN: 1872-7344
Keywords: Depression recognition, Facial representation, Convolutional neural network, Multimodal learning,sequential fusion
Copyright Information: © 2021 Elsevier B.V. All rights reserved.
Date Deposited: 04 Aug 2021 01:59
FoR Codes: 46 INFORMATION AND COMPUTING SCIENCES > 4603 Computer vision and multimedia computation > 460306 Image processing @ 80%
35 COMMERCE, MANAGEMENT, TOURISM AND SERVICES > 3508 Tourism > 350806 Tourist behaviour and visitor experience @ 20%
SEO Codes: 28 EXPANDING KNOWLEDGE > 2801 Expanding knowledge > 280115 Expanding knowledge in the information and computing sciences @ 80%
11 COMMERCIAL SERVICES AND TOURISM > 1199 Other commercial services and tourism > 119999 Other commercial services and tourism not elsewhere classified @ 20%
Downloads: Total: 9
More Statistics

Actions (Repository Staff Only)

Item Control Page Item Control Page