Li, Junhuai, Lin, Chuang, Wang, Huaijun, Zhi, Yuxing, Chen, Jing, and Huang, Tao (2025) Adversarial Training and Cross-modal Feature Fusion in Multimodal Sentiment Analysis. In: Proceedings of the IEEE International Conference on Acoustics Speech and Signal Processing. From: ICASSP 2025: IEEE International Conference on Acoustics, Speech and Signal Processing, 6-11 April 2025, Hyderabad, India.

PDF (Published Version) - Published Version Restricted to Repository staff only

 

1

Abstract

Multimodal sentiment analysis recognizes emotions through text, audio, and visual modalities, but data incompleteness is a major challenge. Existing methods often focus on specific types of deficiencies and perform poorly when multiple types of noise are present simultaneously. To address this issue, we propose a noise-prompted adversarial training framework with a multimodal interaction model to enhance the model's robustness to missing modalities. The model first extracts common and unique features from each modality using a BERT text encoder and a shared-private encoder. Correlation measurements are then used to calculate the similarity between modalities, and a weighting mechanism is applied to the shared features. These features are deeply fused using a Transformer, and adversarial training combined with semantic reconstruction supervision helps the model learn a unified representation of noisy and clean data. Experimental results show that this method significantly improves the performance of multimodal sentiment analysis.

Actions (Repository Staff Only)

Item Control Page