Ji, Junhui, Ren, Wei, and Naseem, Usman (2023) Identifying Creative Harmful Memes via Prompt based Approach. In: Proceedings of the ACM Web Conference 2023. pp. 3868-3872. From: WWW '23: The ACM Web Conference 2023, April 30 - May 4 2023, Austin, TX, USA.

PDF (Published Version) - Published Version Restricted to Repository staff only

Abstract

The creative nature of memes has made it possible for harmful content to spread quickly and widely on the internet. Harmful memes can range from spreading hate speech promoting violence, and causing emotional distress to individuals or communities. These memes are often designed to be misleading, manipulative, and controversial, making it challenging to detect and remove them from online platforms. Previous studies focused on how to fuse visual and language modalities to capture contextual information. However, meme analysis still severely suffers from data deficiency, resulting in insufficient learning of fusion modules. Further, using conventional pretrained encoders for text and images exhibits a greater semantic gap in feature spaces and leads to low performance. To address these gaps, this paper reformulates a harmful meme analysis as an auto-filling and presents a prompt-based approach to identify harmful memes. Specifically, we first transform multimodal data to a single (i.e., textual) modality by generating the captions and attributes of the visual data and then prepend the textual data in the prompt-based pre-trained language model. Experimental results on two benchmark harmful memes datasets demonstrate that our method outperformed state-of-the-art methods. We conclude with the transferability and robustness of our approach to identify creative harmful memes.

Item ID:	79222
Item Type:	Conference Item (Research - E1)
ISBN:	978-1-4503-9416-1
Copyright Information:	© 2023 Copyright held by the owner/author(s).
Date Deposited:	08 Aug 2023 01:12
FoR Codes:	46 INFORMATION AND COMPUTING SCIENCES > 4602 Artificial intelligence > 460208 Natural language processing @ 100%
SEO Codes:	22 INFORMATION AND COMMUNICATION SERVICES > 2204 Information systems, technologies and services > 220403 Artificial intelligence @ 100%
	More Statistics