Simple Retrieval Augmented Classification (RAD-SRAC)

Motivation

Vision-Language Models (VLMs) have achieved remarkable results in image understanding, but in radiology their static training data often limits accuracy and adaptability. We explored whether adding image-based (rather than text-based) retrieval could improve diagnostic performance without retraining models, with the goal of building a training-free retrieval-augmented classification framework that can enhance any VLM with minimal resources.

Methodology

RAD-SRAC is a lightweight, few-shot framework for classification across radiological modalities (CT, MRI, X-ray). All dataset images are encoded with the domain-specific MedImageInsight encoder and stored in Qdrant, an open-source vector database. For every query image, the top-k (1–10) most similar samples are retrieved by cosine similarity and appended to the model's prompt as few-shot examples, either labeled or unlabeled. A structured prompt defines the modality and classes, injects the retrieved references, and requests a JSON output with the predicted class and brief reasoning.

Datasets

The framework was evaluated on KITS23 (CT, 5 tumor subtypes), Coronahack (X-ray, normal/bacterial/viral pneumonia), and Brain Tumor Classification (MRI, four classes). Each dataset was split into a database split for vector storage and a stratified test split.

Results

SRAC substantially improved F1 scores across all datasets and models. On Coronahack, GPT-4o jumped from 41% to 76% F1 (+35%) and Claude 3.5 Sonnet from 46% to 76% (+30%); on KITS23, gains of +6% to +8% were observed. The approach was effective for both large state-of-the-art VLMs (Claude 3.5 Sonnet, GPT-4o, Gemini 1.5 Pro, Qwen2-VL 72B) and smaller deployable models (Gemini 1.5 Flash-8B, Pixtral-12B).