vadimbelsky/esi-triage-bert-v83p1

BERT v83+1 — Medical Triage Decision Support (ESI 1–5)

Multi-head BioBERT-based decision-support model for Emergency Severity Index (ESI) 1–5 triage prediction. Trained on a mixed corpus drawn from MIMIC-IV-ED, MC-MED, MIETIC (narrative-paraphrased), and Lukina v3 (Russian-physician narrative) eval/training mixes, plus engine-validated grounded-synthesis records.

This is the v83+1 checkpoint, succeeding v76. See Changelog for the head-to-head comparison.

Quick read

• val_esi_exact (in-line training val): 0.767
• 3-corpus +safety ESI 1 R (production stack): 94.1% (matches v76's 93.9% baseline)
• Per-corpus safety floors: MCMED 92.0% / MIETIC 96.7% / Lukina 97.1% / MIMIC 93.7% — all PASS
• Sym head: healthy (2.12 fires/record on MIETIC, similar to v76's 2.47)
• Major Lukina narrative WIN: BERT exact +4.5pp, ESI 1 R +5.7pp, ESI 3 R +8.1pp vs v76

Architecture

22-head BERT v44 architecture on BiomedBERT base. Heads include:

• esi_head — 5-class ESI 1–5 (primary inference output)
• symptom_head — 207 multi-label clinical concepts (display-critical)
• flag_head — altered_mentation, severe_pain_distress, active_hemorrhage
• airway_head, resus_head — safety binary heads (pos_weight=2500/200)
• resource_head (12) + resource_count_head (3-bucket)
• vitals_head (6-vital regression — known mean-collapsed; production uses text-regex)
• ner_head (21 BIO tags), arrival_head (5), pain_head, age_head, age_bucket_head (lifecycle), pain_bucket_head, gender_head (3), medrec_head (2), gestalt_head (5), disposition_head (collapsed), syndrome_head (15), history_visits/admits/last_dx_head (collapsed)

Working production-critical heads: esi_head, airway+resus, symptom_head (verified healthy 2.12 MIETIC fires/rec under Round 1411 SC#85-patched eval), resource_count_head, flag_head, arrival_head, ner_head, pain_head/age_head, medrec_head.

Known dead heads (low loss weights so harmless but waste capacity): vitals_head (regression mean-collapsed), disposition_head (99.6% UNKNOWN), history_visits/admits/last_dx (100% bucket 0).

Training data

• MIMIC-IV-ED (research-only license): bulk + ground-truth + pyxis. Most of corpus. Best signal on compact CCs.
• MIETIC + Lukina v3: narrative dialects. Critical for non-CC generalization.
• MC-MED: Stanford telegraphic; ICD-inferred resources fallback when pyxis empty.
• ER-REASON: UCSF discharge/progress notes (4.6%). Kept in training; removed from eval slice.
• Grounded synthesis (medgemma): sparse-concept coverage, ESI inherited from real parents. Cap ≤ 5% total.

Corpus totals: ~406,812 records (v82c base + 87 v83+1 staging deltas: 47 v82p1 narrative + 20 MCMED compact-CC + 20 MIETIC narrative ESI 3).

Intended use

Decision-support for ED triage nurses. The model produces:

• ESI prediction (1–5)
• Symptom concepts firing above per-class thresholds (displayed for trust)
• Resources predicted (workup planning)
• Safety flags (airway/resus risk)

NOT intended as autonomous triage. A licensed clinician must review every prediction. The model is calibrated to over-triage (false ESI 2 on gold ESI 3 is acceptable per direction-aware safety framework; false ESI 3 on gold ESI 1 is NOT).

Limitations

1. MIMIC-IV-ED license inheritance: weights derived from MIMIC are research-only. Commercial use requires upstream PhysioNet agreements.
2. No multi-rater clinician kappa: training labels are single-nurse triage (kappa 0.5–0.7 literature range). v83+1 has no clinician re-label validation step. Pending since Round 1.
3. No ED MD ontology sign-off: SYMPTOM_LABELS (207 concepts) reviewed by agent + engine handbook, not by attending clinician. Pending.
4. Dead heads: 5 heads are degenerate (vitals_head, disposition_head, 3 history heads). Production stack ignores them. v68 ARCH redesign deferred.
5. MIETIC ESI 3 R regression vs v76: -20pp (50% vs 70%). Most misses are direction-safe (gold=3 → pred=2 = over-triage = acceptable per CLAUDE.md). v83+1 recovers +6pp from v83's 44% but does not fully restore v76's 70%.
6. Pediatric coverage: Lukina peds is structurally underrepresented. Production stack does not use age-bucket head for pediatric vital thresholds (deferred).

Production stack

5-stage calibrated pipeline (recommended):

1. BERT esi_head argmax
2. • per-dialect temperature scaling (T ≥ 1.0 — smoothing only; sharpening hurts safety recall)
3. • per-dialect safety thresholds (Bayes-optimal; ESI 1 R floor 0.80) — see configs/v83p1_e3_per_dialect_safety.json
4. • safety_head OR rule (airway_p > 0.5 OR resus_p > 0.5 → ESI 1)
5. • dialect_aware engine override (engine wins on Step A/B1 only)

Do not use min(BERT, engine) — empirically devastates ESI 4 on narrative dialects via Step C/D over-prediction.

Eval scorecard

Patched 4-corpus Mac eval (Round 1411 SC#85 fix; model's symptom_labels_manifest.json loaded for index→label mapping).

BERT-only (no calibration):

Corpus	exact	adj	ESI 1 R	ESI 3 R	sym/rec
MIETIC (n=200)	79.0%	93.0%	80.0%	50.0%	2.12
Lukina (n=201)	61.7%	91.0%	97.1%	62.2%	1.27
MCMED (n=1000)	61.3%	96.5%	43.0%	74.2%	0.73
MIMIC (n=7917)	57.7%	95.1%	72.8%	49.8%	0.79

Production stack (per-corpus +safety ESI 1 R):

Corpus	Floor	v83+1	Status
MCMED	≥90%	92.0%	PASS
MIETIC	≥90%	96.7%	PASS
Lukina	≥95%	97.1%	PASS
MIMIC	≥88%	93.7%	PASS
3-corpus	≥94.5% (stretch)	94.1%	Match v76 baseline 93.9%

Changelog vs v76

Metric	v76 (predecessor)	v83+1	Δ
MIETIC exact	85.0%	79.0%	-6.0
MIETIC ESI 1 R	76.7%	80.0%	+3.3
MIETIC ESI 3 R	70.0%	50.0%	-20.0
Lukina exact	57.2%	61.7%	+4.5
Lukina ESI 1 R	91.4%	97.1%	+5.7
Lukina ESI 3 R	54.1%	62.2%	+8.1
MCMED exact	61.6%	61.3%	-0.3
MCMED ESI 1 R	39.0%	43.0%	+4.0
MCMED ESI 3 R	72.2%	74.2%	+2.0
MIMIC exact	57.1%	57.7%	+0.6
MIMIC ESI 1 R	74.3%	72.8%	-1.5
MIMIC ESI 3 R	48.3%	49.8%	+1.5
Sym fires (MIETIC)	2.47	2.12	healthy
3-corpus +safety ESI 1 R	93.9%	94.1%	+0.2

Wins (✅): Lukina (all 3), MCMED ESI 1 R, MIETIC ESI 1 R, MIMIC ESI 3 R, MIMIC exact, sym healthy, 3-corpus +safety. Losses (⚠): MIETIC ESI 3 R (partial recovery from v83's 44% to 50%; v76 was 70%). Most failures are direction-safe (over-triage to ESI 2).

Loading

import torch
from huggingface_hub import hf_hub_download
from transformers import AutoTokenizer
import sys
# Project layout requires importing V44MultiHeadBERT from the training code.
# Clone https://github.com/anthropics/... or use the embedded tokenizer.

model_path = hf_hub_download(repo_id="vadimbelsky/esi-triage-bert-v83p1", filename="model.pt")
manifest_path = hf_hub_download(repo_id="vadimbelsky/esi-triage-bert-v83p1", filename="symptom_labels_manifest.json")
tokenizer = AutoTokenizer.from_pretrained("vadimbelsky/esi-triage-bert-v83p1")

state = torch.load(model_path, map_location="cpu", weights_only=False)
# Resize model.symptom_head to match checkpoint's 207-dim head before load
# (Round 1411 SC#85 — current SYMPTOM_LABELS may have grown beyond checkpoint).

Citation

If you use this model in research, cite:

@misc{esi-triage-bert-v83p1-2026,
  author       = {Belsky, Vadim},
  title        = {BERT v83+1 — Medical Triage Decision Support (ESI 1-5)},
  year         = 2026,
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/vadimbelsky/esi-triage-bert-v83p1}},
}

License

Research-only. Inherits MIMIC-IV-ED license constraints (PhysioNet credentialed access, no commercial use). See https://physionet.org/content/mimic-iv-ed/2.2/ for the upstream license. Commercial use of these weights requires an independent MIMIC commercial agreement and may require retraining on commercially-licensed data.

---

Trained on Anthropic's Claude Code research-agent loop (Rounds 1316–1416, June 2026). Predecessor v76 trained Round 1112. See project methodology in CLAUDE.md of source repo.