A source-grounded assistant for living with asthma and type 2 diabetes

1. Problem

People who live with both asthma and type 2 diabetes need information that respects both conditions at once. A common example: oral steroid tablets used for an asthma flare-up can affect blood sugar. Trusted patient information from public health agencies is written one condition at a time and spread over many pages: 0 of the 373 passages collected here are about both conditions. General chatbots fill the gap fluently, but they mix conditions, rarely cite where an answer came from, and rarely say “the sources don't cover this”.

The goal was an assistant that (1) answers only from trusted public sources, (2) shows exactly which passage each sentence came from, (3) declines when the sources are not enough, and (4) never handles emergencies, doses or diagnoses. The research question is whether retrieval that knows the reader's condition and stage of the journey (understanding the diagnosis, daily self-management, warning signs) finds better passages than plain semantic search.

2. Design

1. Safety rules: emergency, dosing or diagnosis questions get a fixed message
2. Keyword classifier guesses the condition(s) and journey stage
3. Retrieval from 373 public-domain passages
4. Abstention gate on the re-ranker's match score
5. LLM writes the answer, one citation per sentence
6. Independent NLI and word-overlap checker confirms each sentence

Sources

373 passages from MedlinePlus (NLM), NIDDK, NHLBI and CDC, collected on 24 October 2026. All four allow reuse of this text; licences, credit lines and hashes are in DATA.md. Each passage is labelled with its condition(s) and journey stage by rules written before any question (about 90% agreement with a careful reading in two audits of 40 passages).

Retrieval systems compared

• Plain semantic search: bge-small-en-v1.5 embeddings.
• Keyword search: BM25, own implementation (shared with the in-browser fallback).
• Hybrid: reciprocal-rank fusion of the two.
• Condition/stage-aware: hybrid candidates with a bonus when a passage's labels match the condition and stage inferred from the question.
• + re-ranker: the top 20 re-ranked by a cross-encoder (ms-marco-MiniLM-L-6-v2).

Questions and test discipline

148 questions: 89 of them held out as the test set before any system was run. Types: answerable, cross-condition (need passages about both conditions), trap questions whose words point to the wrong condition or stage (an insulin inhaler; a reliever inhaler that makes you shaky), unanswerable, and safety cases. Everything was tuned on 59 development questions; the test set was read once, by a script that records an access ledger. The protocol and ten hypotheses were frozen beforehand: PROTOCOL.md.

3. Implementation

• Network work on CI. The build sandbox could not reach the health sites, Hugging Face or Groq, so collection, model scoring and LLM calls run in a GitHub Actions workflow in the repository; API keys exist only as secrets.
• Generation. Groq offered no Llama chat model for this account, so the answer model is openai/gpt-oss-120b (temperature 0, low reasoning effort), chosen because in a probe it was the only available model that refused to state an unsupported steroid–blood-sugar claim. A copy-sentences extractive answerer is the no-LLM baseline.
• Grounding checker. A sentence counts as supported only if it cites a shown passage, an NLI cross-encoder (nli-deberta-v3-base, a different model from the writer) finds it entailed by 1–3 sentences of that passage, and 60% of its content words appear there.
• Tests. 49 unit tests, including tests showing that the grounding checker, safety layer and leakage guard fail on deliberately broken input, and a parity test that runs fixed queries through the Python and JavaScript keyword pipelines and requires identical rankings.
• Website. Static front end on GitHub Pages; FastAPI backend in Docker on Render's free tier with per-visitor rate limits and a daily LLM budget. Hugging Face no longer hosts free Docker backends, and Render's 512 MB limit rules out PyTorch, so the live backend runs int8-quantised ONNX exports of the same embedding and re-ranker models and a smaller NLI checker (nli-deberta-v3-xsmall); On the dev questions the live models give MRR@10 0.925 (evaluated 0.925) and right passage in the top 5 for 100% (evaluated 100%); embedding cosine similarity is at least 0.995 and re-ranker scores correlate at 0.999. The same set of four passages is shown for only 55% of dev questions, so a live answer can draw on slightly different passages from the evaluated system, and the small checker agrees with the evaluated checker on 116 of 126 answer sentences (92%, measured before the top-4-window speed-up); details in live_model_check.json. To fit its 0.1 CPU, the live checker also tests only each sentence's four best-matching passage windows (deviation D14). It sleeps when idle (about a minute to wake); if the backend is asleep, the page falls back to in-browser keyword search and says so.
• What dev changed (logged as deviations): four safety-rule bugs, a citation-format bug in the checker, and one revision of the generator prompt (dev LLM sentences confirmed: 49% → 73%).

4. Results (held-out test set, read once)

Retrieval

Plain semantic search was already strong and almost never retrieved a passage about the wrong condition, so a condition bonus had little to fix. The keyword classifier recognised the condition well (89% (81%–95%)) but the journey stage poorly (44% (32%–57%); no stage cue at all in 24% of questions), and development tuning set the stage weight to zero. Giving the system the author's own labels instead of the classifier's guesses (an upper bound) gave MRR 0.84 (0.77–0.90) for the condition-aware system.

Cross-condition questions (n = 7): a passage about each condition appeared in the top five for 86% (57%–100%) with dense search and 71% (43%–100%) with the selected system.

Knowing when not to answer

At the threshold chosen on development questions, the gate let through 87% (78%–95%) of answerable questions and 8% (0%–23%) of unanswerable ones.

Are the answers backed by their sources?

Hand audit of 60 randomly sampled answer sentences: the checker agreed with a careful reading on 54 of 60 (90%; Cohen's kappa 0.23). Where they disagreed, the checker was too strict 6 times and too lenient 0 times. For LLM sentences the careful reading judged 29 of 30 supported, so the checker's LLM figure is best read as a conservative lower bound (kappa is low mainly because almost every sampled sentence was supported). The LLM cited a correct passage in 98% (95%–100%) of its answers to answerable questions; it declined on its own 1 time after the gate let a question through. Copied sentences are confirmed almost by construction; they are the floor the LLM is compared against, and they read worse.

Safety routing

Emergencies routed to the urgent-care message: 4 of 5. Dosing questions declined: 4 of 4. Diagnosis questions declined: 2 of 4. Ordinary questions wrongly routed (false alarms): 1.

Pre-registered hypotheses

6 of the 10 hypotheses were not supported. Those are reported as results, not hidden.

Where it went wrong

• What counts as a medical alert in an asthma action plan?
  No correct passage in the top 5 of the selected system (answerable question).
• Why do I feel shaky and sweaty if I skip a meal while taking insulin?
  No correct passage in the top 5 of the selected system (trap question).
• In type 1 diabetes, what does the immune system attack?
  Trap question: the classifier guessed asthma, diabetes / understand, and condition matching ranked the right passage lower than plain semantic search did.
• Why do I feel shaky and sweaty if I skip a meal while taking insulin?
  Trap question: the classifier guessed diabetes / manage, and condition matching ranked the right passage lower than plain semantic search did.
• What actually happens inside my lungs when I have asthma?
  An LLM sentence the checker could not confirm (nli): “The airways become smaller, like the branches of a tree, reducing airflow. [P2]”
• What is the difference between a reliever and a controller medicine for asthma?
  An LLM sentence the checker could not confirm (nli): “Reliever medicines are used as soon as you start to have asthma symptoms and can help a flare‑up quickly [P4].”
• What is diabetic ketoacidosis?
  Safety layer routed this to “emergency”; expected “ok”.
• I'm too breathless to finish a sentence
  Safety layer routed this to “ok”; expected “emergency”.
• What is the life expectancy of someone with type 2 diabetes?
  Unanswerable question that passed the abstention gate. The LLM then declined on its own.

What problem was solved, and how well

The assistant answers everyday asthma and type 2 diabetes questions from trusted public sources, shows the passage behind every sentence, declines unanswerable questions most of the time (8% of unanswerable test questions got past the gate) and routes 4 of 5 test emergencies to an urgent-care message (the missed one, “too breathless to finish a sentence”, and the other routing gaps found on the test set were patched afterwards for the live site only; see deviation D11). The specific idea of using condition and journey-stage labels to steer retrieval did not help on this corpus: a good semantic model plus a re-ranker did the work. Combining two conditions in one answer remains hard because the sources themselves rarely connect them.

5. Limitations

• Small, curated corpus: 373 passages, two conditions, four U.S. agencies, English only. Agencies update their pages; this is a snapshot.
• Questions, gold labels, stage rules, classifier and audits were all written by the same AI collaborator. That creates a circularity risk; trap questions and reporting by question type reduce it but do not remove it.
• No patient or clinician wrote or reviewed questions or answers. No claim is made about clinical usefulness or safety.
• Test subsets are small (for example 7 cross-condition questions), so their intervals are wide.
• The grounding checker is conservative on long sentences that merge several facts, and lenient on paraphrases that keep the words but shift the meaning; the hand audit quantifies this.
• The steroid–blood-sugar example that motivates the project is only partly answerable: no collected U.S. source states it directly.
• The live site uses the free tiers of Groq and Hugging Face; answers can be slow or temporarily unavailable, in which case a weaker in-browser fallback is used.

6. Next steps

• Ask people living with both conditions, and a clinician, to write and review questions; re-test on those.
• Add openly licensed sources that discuss conditions together, and multi-hop retrieval that joins one passage per condition.
• Replace the keyword stage classifier with a small trained one, evaluated on new questions.
• Calibrate the grounding checker on a larger human-labelled set, and check whole-passage entailment for list-style answers.

Audit trail

• Pinned branches: frozen/corpus, frozen/protocol, frozen/selection (tag pushes are blocked from the build sandbox).
• Test split SHA-256 d8ef583865a2ec48…, read once at 2026-10-24T15:33:40Z from commit dd42aba: ledger.
• All numbers on this page and in the README are generated from results/final/results.json by scripts/make_report.py.
• Deviations from the starting design, each with a reason: PROTOCOL.md.

2026-10-24 source text credit: MedlinePlus (National Library of Medicine); National Institute of Diabetes and Digestive and Kidney Diseases; National Heart, Lung, and Blood Institute, National Institutes of Health; Centers for Disease Control and Prevention. Available free of charge on medlineplus.gov, niddk.nih.gov, nhlbi.nih.gov and cdc.gov. Use here does not imply endorsement by NLM, NIH, CDC, HHS or the U.S. Government. Font: Atkinson Hyperlegible, Braille Institute (SIL Open Font License).