This study presents AFLoc, a multimodal vision–language model designed to localize pathological regions in medical images without requiring manual annotations. Traditional pathology and radiology AI systems depend on expert-labeled regions to learn where disease is present, a process that is expensive, subjective, and difficult to scale. AFLoc addresses this limitation by learning to associate visual patterns directly with clinical language.
The technical foundation of AFLoc lies in a multilevel semantic alignment framework. Rather than treating an image and its report as a single global pair, the model aligns clinical text and image features at multiple levels of granularity. Word-level descriptions are matched with fine-grained local image features, sentence-level descriptions with deeper spatial features, and report-level summaries with global image representations. This design allows the model to interpret variable, unstructured clinical reports that lack explicit localization cues—reflecting how medical documentation is written in practice.
This alignment strategy enables annotation-free localization that is both accurate and generalizable. AFLoc demonstrated strong zero-shot performance across three distinct imaging modalities—chest X-rays, retinal fundus images, and histopathology—outperforming or matching state-of-the-art vision–language models such as BioViL, MedKLIP, PLIP, and CONCH on multiple localization benchmarks. Importantly, the model also surpassed human benchmarks for several pathology categories in chest X-ray localization tasks.
Beyond technical benchmarks, the study provides evidence of clinical utility and trustworthiness. In controlled reader studies with board-certified radiologists, AFLoc reduced average reading time by 20.5% while improving diagnostic accuracy scores. This indicates that the model does not merely highlight visually salient regions, but provides clinically meaningful cues that reduce cognitive load and support expert decision-making.
The impact of this work is twofold. Scientifically, it demonstrates that semantic grounding between language and vision is sufficient for reliable medical image localization, even in the absence of explicit supervision. Practically, it offers a scalable pathway for deploying localization-capable AI systems across institutions and disease domains where annotated data are scarce, while maintaining transparency in how predictions are generated and why they can be trusted.
H. Yang et al., “A multimodal vision–language model for generalizable annotation-free pathology localization,” Nature Biomedical Engineering, vol. 9, p. 231, Mar. 2025, doi: 10.1038/s41551-025-01574-7
