Huang, Ziyu, Peng, Yige, Xu, Mengdie, Jung, Younhyun, Ahn, Euijoon, Kim, Jinman, and Bi, Lei (2027) SA-LanSeg: Bridging semi-automatic interaction and language guidance for medical image segmentation. Pattern Recognition, 181. 114665.

PDF (Published Version) - Published Version Restricted to Repository staff only

Abstract

Medical image segmentation plays an essential role in clinical workflows, including diagnosis, surgical planning and treatment response assessment. However, achieving reliable segmentation in challenging cases, where lesions exhibit ambiguous localization or visually similar patterns, remains difficult. Semi-automatic segmentation has become a widely adopted approach to handle these cases by utilizing user-guided hints for improved segmentation. Recently, language-guided segmentation methods have emerged as another important paradigm for medical image segmentation by leveraging clinical textual reports to provide semantic guidance, such as lesion attributes and locations. Although both strategies outperform image-based methods, they continue to remain limited. Language-guided methods mainly address semantic ambiguity (what the target is) but struggle to capture precise spatial context, while semi-automatic methods reduce spatial ambiguity (where the target is) through user-guided hints but lack sufficient semantic understanding from clinical context. To overcome these limitations, in this study, we propose a unified Semi-Automatic Language-Guided Segmentation (SA-LanSeg) framework, which synergistically integrates semantic guidance from clinical reports with location cues obtained from user-guided hints. SA-LanSeg employs a multi-branch architecture to separately model image features, clinical report semantics, and user-guided hints, and introduces a cosine-similarity fusion mechanism to adaptively integrate these complementary feature streams for accurate segmentation. We comprehensively evaluated our SA-LanSeg on four public benchmark datasets including QaTa-COV19, MosMedData, Kvasir-SEG, and Bkai-polyp, totaling 13,987 images. Our framework consistently outperforms existing image-only, language-guided, and semi-automatic segmentation methods across all benchmark datasets, and is especially effective in segmenting challenging cases that require both accurate spatial and semantic contexts.

Item ID:	94493
Item Type:	Article (Research - C1)
ISSN:	1873-5142
Copyright Information:	© 2026 Elsevier Ltd. All rights are reserved, including those for text and data mining, AI training, and similar technologies.
Date Deposited:	26 Oct 2026 03:52
FoR Codes:	40 ENGINEERING > 4003 Biomedical engineering > 400304 Biomedical imaging @ 20% 46 INFORMATION AND COMPUTING SCIENCES > 4603 Computer vision and multimedia computation > 460308 Pattern recognition @ 40% 46 INFORMATION AND COMPUTING SCIENCES > 4611 Machine learning > 461103 Deep learning @ 40%
SEO Codes:	22 INFORMATION AND COMMUNICATION SERVICES > 2204 Information systems, technologies and services > 220403 Artificial intelligence @ 50% 20 HEALTH > 2002 Evaluation of health and support services > 200206 Health system performance (incl. effectiveness of programs) @ 50%
	More Statistics