Creating AI Tools for Healthcare — Workshop Recap

Our recent workshop, "Building AI Tools for Healthcare," offered a technical deep-dive into the methodologies and models shaping the future of medical AI. This post summarizes the key concepts, challenges, and solutions discussed across the three main sessions, each paired with a hands-on Kaggle notebook.

Part 1: Medical Imaging Models

The opening session addressed the primary bottleneck in healthcare AI: data. We highlighted systemic issues including the scarcity of public datasets, the lack of storage standards (NIfTI, DICOM, XML, CSV), and inconsistent labeling that often relies on mental shortcuts or regionalisms.

To combat this, we explored the Universal Medical Image Encoding (UMIE) initiative — a large-scale, unified dataset of over 1 million images (CT, MRI, X-ray) drawn from more than 20 sources. UMIE standardizes diverse data through modular, reusable pipelines inspired by sklearn.pipeline, handling everything from DICOM conversion to mask extraction and label unification. A crucial part of this standardization is adopting a unified ontology, such as RadLex, to map disparate labels to a common vocabulary.

The hands-on portion focused on fine-tuning BioMedCLIP, a multimodal foundation model pre-trained on millions of image-text pairs from biomedical literature, adapting it for tumor classification. We also surveyed alternative medical vision encoders like MedImageInsights, BioMedParse, and RadDino, each with unique architectures for tasks from report generation to semantic segmentation.

Notebook: Fine-tuning BioMedCLIP for Medical Imaging.

Part 2: Language Models

The second session broke LLM training into three stages: pre-training (predicting the next word on a massive corpus), supervised fine-tuning (teaching the model to follow instructions), and preference optimization (teaching it to respond in a specific style, e.g. RLHF).

A key challenge discussed was catastrophic forgetting, where a model loses general knowledge during fine-tuning on a specialized task. While fine-tuning can introduce new knowledge, it also increases hallucination risk if that information is not well-grounded. Continual pre-training — exemplified by the Me-LLaMA case study, a LLaMA 2 model continually trained on 129 billion medical tokens — is one solution. For more targeted adaptation, we focused on Parameter-Efficient Fine-Tuning, specifically Low-Rank Adaptation (LoRA), which freezes pre-trained weights and injects small trainable low-rank matrices, preserving the model's original knowledge base.

The practical lab applied these concepts by fine-tuning Gemma-3 for generating radiology reports.

Notebook: Fine-tuning Gemma-3 for Radiology Report Generation.

Part 3: Sentence Encoders & Retrieval Systems

The final session focused on converting text into meaningful numerical representations using sentence encoders. Typically built on transformer encoders like BERT, these models map sentences to high-dimensional vectors where semantic similarity is measured by cosine distance. Training is often done via contrastive learning with a Multiple Negative Ranking Loss, improved further by negative mining — selecting "hard" negatives that are superficially similar but semantically different.

We discussed wide-ranging medical applications: knowledge and evidence retrieval (RAG) to ground LLM answers in clinical guidelines, EHR mining to identify patient cohorts from unstructured notes, and coding against ontologies like ICD or SNOMED CT. The session detailed a Retrieval-Augmented Generation architecture, where documents are indexed in a vector database and relevant context is retrieved at inference time to augment the LLM's knowledge.

The workshop concluded with a practical notebook on predicting ICD codes from clinical notes — a direct application of sentence encoders for automated medical coding.

Notebook: ICD Codes Prediction using Sentence Encoders.