When a model makes a clinical prediction, someone usually has to ask why. There are good tools for answering that question for classical models, and good tools for prompting LLMs to explain themselves, but nothing that puts both in one place so you can see where they agree.

I built a framework that does. It generates explanations from logistic regression, XGBoost, and three locally hosted LLMs (Llama 3.1, Qwen 2.5, Gemma 3), normalises them into a common format, and displays them side by side in a Streamlit dashboard. An adapter-based design means adding a new model or explanation method doesn't require touching the rest of the system. LLM inference runs locally through Ollama on a single consumer GPU, with results cached so the dashboard stays responsive.

I tested it on two healthcare datasets chosen to contrast with each other: one structured, with fixed clinical feature names, and one free-text, with features extracted via spaCy

That contrast produced the most useful finding. On structured data, explanations from classical models and LLMs aligned completely. On free-text data, alignment dropped to around ten per cent, and almost none of it was disagreement about what mattered. The models were naming the same things differently: an LLM saying "feverish" where the extracted vocabulary held "fever". Comparing explanations across model types is mostly a feature-naming problem, not a modelling one.

Nine people tested the dashboard across two rounds of task-based usability testing, and their feedback drove a redesign between rounds.