Naseem, Usman, Kim, Jinman, Khushi, Matloob, and Dunn, Adam G. (2023) A Multimodal Framework for the Identification of Vaccine Critical Memes on Twitter. In: Proceedings of the 16th ACM International Conference on Web Search and Data Mining. pp. 706-714. From: WSDM '23: 16th ACM International Conference on Web Search and Data Mining, 27 February - 3 March 2023, Singapore.

PDF (Published Version) - Published Version Restricted to Repository staff only

Abstract

Memes can be a useful way to spread information because they are funny, easy to share, and can spread quickly and reach further than other forms. With increased interest in COVID-19 vaccines, vaccination-related memes have grown in number and reach. Memes analysis can be difficult because they use sarcasm and often require contextual understanding. Previous research has shown promising results but could be improved by capturing global and local representations within memes to model contextual information. Further, the limited public availability of annotated vaccine critical memes datasets limit our ability to design computational methods to help design targeted interventions and boost vaccine uptake. To address these gaps, we present VaxMeme, which consists of 10,244 manually labelled memes. With VaxMeme, we propose a new multimodal framework designed to improve the memes’ representation by learning the global and local representations of memes. The improved memes’ representations are then fed to an attentive representation learning module to capture contextual information for classification using an optimised loss function. Experimental results show that our framework outperformed state-of-the-art methods with an F1-Score of 84.2%. We further analyse the transferability and generalisability of our framework and show that understanding both modalities is important to identify vaccine critical memes on Twitter. Finally, we discuss how understanding memes can be useful in designing shareable vaccination promotion, myth debunking memes and monitoring their uptake on social media platforms.

Item ID:	79219
Item Type:	Conference Item (Research - E1)
ISBN:	978-1-4503-9407-9
Copyright Information:	Copyright © 2023 by the Association for Computing Machinery, Inc. (ACM).
Date Deposited:	06 Sep 2023 03:07
FoR Codes:	46 INFORMATION AND COMPUTING SCIENCES > 4602 Artificial intelligence > 460208 Natural language processing @ 100%
SEO Codes:	22 INFORMATION AND COMMUNICATION SERVICES > 2204 Information systems, technologies and services > 220403 Artificial intelligence @ 100%
	More Statistics