Who Has to Check the AI in Your Medical Record?

Audio Companion

While your doctor looks at you, an AI may be writing the visit note. Another is scoring your risk of getting worse. A third is drafting the reply to the message you sent last night. None of it required your doctor — or you — to decide these tools had earned the job. They arrived the way office software arrives: bundled with the record system the hospital already bought, switched on as a pilot that never quite ended, sold as relief from after-hours paperwork.

In a 2024 national survey, most U.S. hospitals reported predictive AI built into or launched through their record systems — and AI that drafts the doctor's note is spreading fast too. So the interesting question is no longer "should we let AI into medicine?" It's in. The question is whether anyone checked that it works here, on these patients — and who has to act when it doesn't.

New drugs, new surgical techniques, the electronic record itself: plenty of medicine spread before the proof was complete. But medicine built machinery for catching up: committees that can pull a drug from the shelf, boards that can suspend a surgeon's privileges, conferences that pick apart complications, safety monitoring that keeps going after a product ships. Hospitals now name AI committees too. What's still thin is the part with teeth — a body with real authority to pause or pull an AI tool when it stops working well locally.

And the stakes moved. The chart used to be a filing cabinet. Now it drafts the story of your care, scores your risk, prompts the replies, routes the action. One vendor contract can set the defaults for thousands of doctors who never chose the product, the settings, or the terms.

When hospital AI fails, no alarm sounds. The screen looks the same as yesterday. Underneath, a model can drift out of tune with the patients in front of it — new software versions, new patient mix, new workflows. One review went looking for studies that had checked whether these models' risk predictions still matched what actually happened to patients. Every study it found reported the match had degraded. The tool can still look fine — still sorting sicker from healthier patients reasonably well — while the numbers underneath quietly go wrong.

The other failures are quiet too. A note-drafting tool can slip an invented detail into your chart; if a rushed doctor misses it, the next doctor may treat it as fact. An alert system can cry wolf so often that clicking past warnings becomes a reflex — and a reflex can't tell the hundredth false alarm from the one that matters.

Being installed proves nothing except that somebody said yes once. For a tool that can change a diagnosis, a treatment, or your permanent medical record, the evidence should match the stakes — and the case should be re-made at every activation, every big upgrade, every expansion, every contract renewal.

None of the usual substitutes clears that bar. Regulatory clearance to sell a product is not proof it keeps working in this hospital. A high score on a benchmark is not performance on these patients. "The doctors like it" is not safety. And saving time — often the real reason the tool was bought — answers a different question than whether it works.

A doctor's own experience isn't enough either, through no fault of the doctor. Daily use teaches real things — where the note-drafter drops context, which alerts come too late. But nobody can feel a model drifting statistically, or spot a failure that only hits one group of patients, or count errors scattered across thousands of visits. That takes an institution deliberately looking.

To be fair to the builders: most medical-AI research is still early-stage, and not every tool needs a full clinical trial. A note-drafter and an autonomous diagnostician don't deserve the same bar. Some narrow tools may already have earned their place for well-scoped jobs. And delay has real human costs — waits, burnout, too few specialists. All true. None of it removes the need for a standard. The bar is not the hard part. The hard part is finding anyone who can show you the bar being used.

Take Epic's sepsis-prediction tool. Epic is one of the big record-system vendors; sepsis is the runaway infection response that kills when it's caught late. Hospitals ran version 1 for years. Then outside researchers put it to the test. Using their own definition of sepsis, they found it performed far below the advertised range, missed most of the sepsis cases, and set off alerts on nearly one in five hospital stays.

Then, years after that public failure, more than a hundred organizations were still running the criticized version. That's the vendor's own count.

Nobody outside those hospitals can see what they privately checked. Maybe some re-tested it on their own patients, re-tuned it, made an informed call to continue. The contracts are confidential. The dashboards are internal. No rule anyone can point to requires showing your work. But that is exactly the problem: after a failure this public, a safety check no one can show you is — for the patient downstream — the same as no check at all.

The same story shows the pause button exists: when COVID scrambled patient patterns, the model's alerts surged — and one health system documented hitting pause. What the public record doesn't show is anyone required to press it.

It doesn't have to be this way. A newer version of the tool was tuned on four study sites' own patient data, then tested on the patients who came next. Results improved — though it still took dozens of alerts to find each true case. That is what checking looks like. It happened. Nothing in the public record shows it being required.

And this is not one vendor's story: the public paper trail on hospital AI purchasing shows no documented performance guarantees — and since the contracts are confidential, whatever bar a deal does set, no outsider can see it or enforce it. Where a tool gets renewed, upgraded, or expanded without anyone reopening the evidence, the institution isn't deciding. It's ratifying a default.

"Reliance" sounds like one thing. It's three.

The hospital decides. It buys, configures, renews. The default is structural: a contract or a pilot-that-never-ended carries the tool forward, year after year, with nobody asking whether it still earns the job.

The doctor signs. Doctors wanted relief from paperwork; they didn't choose the vendor or the terms. But professional responsibility follows the signature at the bottom of the note — not the purchase order.

You inherit. Whatever you think of AI, the deployment reaches you either way. In surveys, people report low trust that health systems will use AI responsibly — though that says more about how much they trust the health system in general than about what they know about AI. Many say they want to be told. In one randomized survey, telling patients their message was AI-drafted barely changed how they rated it. And outside a few states with new notice laws, no consent rule generally requires anyone to tell you. Telling patients is the cheapest form of accountability on the menu — an institution that doesn't spend even that is telling you something about the expensive forms.

Blur the three together and you've dressed three different things up as one feeling. Adoption statistics measure hospital decisions. Edit counts measure doctor behavior. Surveys measure what patients say they feel. None of them measures trust in the AI itself. What the numbers do support is narrower: reliance gets settled by default when using the tool becomes routine and no process with real authority exists to test it and change it — a process that is written down, sized to the stakes, and able to narrow or stop use. Whether anyone has actually come to trust the AI is a different question — feelings are invisible. The routine is sitting right there in the logs. And where routine is all there is, habit — not evidence — is the policy.

That is the diagnosis. The diagnosis can be proved wrong: make local testing before turn-on, and re-checks with standing power to narrow or pause, the norm instead of the exception.

One early study — not yet peer-reviewed — looked at nearly 24,000 AI-assisted notes. About one in six were signed with every AI-drafted section untouched; the rest were edited. Neither number tells you whether a note was actually reviewed: an unchanged note may have been read carefully, an edited one skimmed.

Hospitals stake safety on "the doctor will check it" — and no one has published a standard way of measuring the checking, the one act the note-drafting safety case rests on. If that's the plan, the honest version is sampling real notes and real outcomes. Edit counts aren't safety, and minutes-to-signature measures speed, not scrutiny. In one simulated test, note-drafting tools produced on the order of three errors per case, with potential for moderate-to-severe harm.

Alerts have their own trap. Doctors override half to nearly all of the old rule-based medication warnings — and a large share of those overrides are judged clinically reasonable. That's a baseline of alert noise, not a verdict on AI — and new model alerts arrive in workflows already shaped by it. Engagement is possible — one hospital system that deployed a sepsis alert with real training and feedback saw doctors engage with nearly nine in ten alerts — but that took deliberate work, at that site.

And the checking-up? Most AI-using hospitals say they do at least some evaluation after rollout; barely more than half say they do it for all or most of their models. "Evaluation" means whatever the person filling out the survey decides it means. And those surveys never ask who has the authority — and the budget — to pause anything.

Follow the risk when an untested default fails. The company that built the tool and keeps updating it? Record-system contracts have long been written to shield the vendor — hold-harmless clauses, liability caps — and because the contracts are confidential, nothing public shows that pattern stopping at AI. The doctor? The lawsuit risk lands on the signature at the bottom of the note — and the person who signs often has no power to pause the tool. You? You inherit the outcome.

That mismatch — control concentrated upstream, losses scattered across signatures, lawsuits, and patients — is what lets the quiet failures keep running.

A hospital can't validate what a vendor won't share, or pause a feature the contract makes inseparable. The off-switch has to be negotiated into the purchase — notice of version changes, audit access, a practical way to turn the thing off. If it isn't in the contract, don't count on it existing.

Even so, the hospital sits in the middle. It is the only player holding both the power to stop the tool — at least on paper — and a real stake in what happens. That's why the checking should live there: test before turning on and before big updates, watch afterward, keep the authority to stop, and give patients a straight answer about whether AI touched their care. Not because hospitals are villains — but because nobody else holds both the switch and the stake.

That doesn't let vendors off. They should stay on the hook for what they claim, what they change, what breaks, and the data hospitals need to do the checking — today, the fine print often runs the other way. None of this makes outside referees useless — device regulators, certification bodies, transparency rules all do real work. But their work is narrow, and it moves at rulemaking speed while workflow habits harden in months. The main federal transparency rules for these tools are, as this is written, proposed for partial repeal. Waiting for a universal outside referee is not a plan.

The checking should match the tool:

Notes signed unchanged that nobody samples. Alerts nobody audits for whether the response made sense. A model's accuracy nobody is named to re-check. A pause button no one is required to press.

The old question — "should we let AI into medicine?" — still matters out at the frontier, where autonomous systems are fighting for legal ground. For the tools already drafting notes and scoring charts, the useful questions are local: What is becoming the default here? What evidence could change it? Who acts when it should change?

The chart can make a tool feel ordinary. It cannot make it earned. If a hospital can't say what evidence would change its default — and who has the power to act — then habit, not evidence, is running the place.

How to read this issue: Explainer.

Confidence: Medium. Every study, survey, and rule cited was read directly, and the receipts below give each one's exact bounds. The essay's "no one can show you" claims are limited to the public record this research searched — they are not claims that nothing happened privately. Corrections are invited.

What would change our mind: Evidence that local validation before activation, and post-deployment review with standing authority to narrow or pause, have become the norm for these tool classes rather than the exception — or any public accounting that lets an outsider watch a hospital's own bar being applied.

What would prove this essay wrong

The receipts — every claim above, with its exact bounds.

"Most U.S. hospitals": 71% of surveyed U.S. non-federal acute-care hospitals reported predictive AI embedded in or launched through the EHR (ASTP/ONC Data Brief 80, published September 2025 on 2023–2024 survey data). Among hospitals using predictive AI, 80% used at least one model from their EHR developer — the reach of the vendor channel, not proof that vendors supplied or activated every tool.

"Epic is one of the big record-system vendors": the ambient-adoption study below counts 2,784 Epic-customer hospitals as of June 2025 — 42.4% of the 6,561 U.S. hospitals in its universe (Yang & Graetz, Am J Manag Care 2026). That is the basis for "one of the big"; it is not a market-share or ranking claim.

"Spreading fast too": by June 2025, 62.6% of hospitals identified as Epic users belonged to systems that had implemented or were implementing ambient AI documentation (Yang & Graetz 2026) — a large, nonrepresentative sample; system-level status assigned to member hospitals. The figure measures organizational rollout; it neither ranks ambient AI against other clinical tools nor measures individual clinician use.

"Hospitals now name AI committees": in the same national survey, 66% of AI-using hospitals named a committee as owning evaluation and 60% named division or department leaders (ASTP/ONC Data Brief 80). These are self-reported ownership answers; the survey does not ask whether any of those bodies holds authority to pause a tool — which is what "still thin" refers to.

"Every study it found": in one systematic review of performance under temporal dataset shift (Guo et al., Appl Clin Inform 2021), all 11 included studies that evaluated calibration — whether predicted risks match observed outcomes — reported deterioration, while only 3 of 12 that evaluated discrimination (the ability to separate higher- from lower-risk patients) did.

Invented details and downstream readers: the error types named (omissions, invented detail, misleading emphasis) characterize the counted draft-error base — see the ambient evaluation below (Mayo Clin Proc Digit Health 2025). On the downstream half: when AI suggestions were wrong, readers at every experience level were pulled toward them, though far less at the highest experience (Dratsch et al., Radiology 2023); and in a study evaluation, physicians’ finished replies stayed closer to the AI draft than to manually written ones (Chen et al., Lancet Digit Health 2024).

"Still early-stage": one 2026 scoping review classified 88.2% of medical-AI studies as preclinical and 97.6% as preclinical or early clinical evaluation; 2.4% were randomized trials (npj Digital Medicine 2026).

Epic sepsis v1: external validation, under the study’s sepsis definitions: AUC 0.63 versus the marketed 0.76–0.83, most sepsis cases missed, and an alert on about 18% of hospitalizations — 6,971 of 38,455 (Wong et al., JAMA Internal Medicine 2021). "More than a hundred organizations" still on v1 as of August 2025 is the vendor’s own count, relayed in a 2026 validation study (JAMA Network Open 2026). That is persistence, not proof that no site re-examined the model: private QA is rarely published, and switching costs, contract lock-in, upgrade backlogs, and deliberate site-specific judgment all remain possible.

"Alerts surged … hitting pause": across 24 U.S. hospitals, sepsis-model alerts rose 43% in the three weeks after each system’s first COVID-19 case even as census fell 35%; the University of Michigan hospital deactivated the model in April 2020 over false-positive alerting (Wong et al., JAMA Network Open 2021). "Anyone required to press it" is a public-record claim: surveys of hospital AI monitoring do not ask about pause authority or requirements.

The newer version: locally fine-tuned at four study sites and prospectively validated; better discrimination with still-low positive predictive value — on the order of dozens of alerts evaluated per true case at a common prediction horizon (JAMA Network Open 2026). It says nothing about how sites continuing on v1 applied their local bar. "Nothing in the public record shows it being required" is scoped to the public record assembled for this issue.

Contracts: published examinations of hospital AI procurement show no documented negotiated performance guarantees, and contracts are confidential. Hold-harmless terms are long-standing in EHR contracting (Koppel & Kreda, JAMA 2009; ONC’s 2016 EHR contracting guide), and nothing public shows the pattern stopping at AI. Clinical-AI malpractice law remains unsettled in the published reviews available for this issue.

The patient layer: 65.8% of surveyed adults reported low trust that health systems would use AI responsibly, with no association between AI knowledge and trust — general trust in the health system was the stronger predictor (Nong & Platt, JAMA Network Open 2025). 95.2% said notification matters, 62.7% "very" (Platt et al. 2024). In a randomized survey, disclosing that a portal message was AI-drafted moved satisfaction by about 0.13 points on a 5-point scale, with more than three-quarters satisfied either way (Cavalier et al. 2025) — that specific disclosure channel, not AI-in-care generally. On doctrine: Cohen’s reading is that in general liability will not lie for failing to inform patients about medical AI (Georgetown Law Journal, 2020) — a 2020 reading, and some states have since imposed AI-notice duties for specific uses. Consent to a microphone under recording law is not consent to rely on AI-mediated care.

"About one in six": a preprint — posted before peer review — from an academic health system: 15.6% of 23,760 AI-assisted notes signed with every AI-drafted section unchanged (3,698 of 23,760), 84.4% edited; clinician practice style was the dominant source of editing variation (medRxiv, version 2, January 2026). The unchanged-note rate is a ceiling on full no-edit signing, not a count of unreviewed notes. The review act itself has no published standard measurement as institutional practice.

"Three errors per case": one simulated evaluation of five ambient platforms — four commercial, one free — across 14 mock encounters found a mean of 3.0 errors per case with potential for moderate-to-severe harm (Mayo Clin Proc Digit Health 2025).

"Half to nearly all": clinicians override 49% to 96% of interruptive medication alerts (van der Sijs et al., JAMIA 2006); a later review of 23 studies found 46.2% to 96.2%, with a large share of overrides judged clinically appropriate (Poly et al., JMIR Med Inform 2020). Both describe older, rule-based decision support — baseline alert noise, not AI distrust.

"Nearly nine in ten": in the TREWS sepsis deployment, clinicians entered an evaluation for 89% of alerts and confirmed 38% of those they evaluated (Henry et al., Nature Medicine 2022) — examined response with training and feedback; at that site, a counterexample to unexamined tune-out.

"At least some evaluation": among AI-using hospitals, 79% report at least some post-implementation evaluation and 58% report it for all or most models (the 58% is nested within the 79%); 18% answered "don’t know". The figures are self-graded, and the survey does not ask about pause authority, budget, or monitoring for both over-acceptance and under-response (ASTP/ONC Data Brief 80).

Outside referees: FDA duties attach to regulated devices; the Joint Commission/CHAI responsible-use certification (announced June 2026) states that it "does not validate or certify individual AI products or tools." HTI-1 (final rule, January 2024) applies developer-transparency requirements only to health IT covered by the federal certification program, and HTI-5 proposes removing the source-attribute ("AI model card") and risk-management requirements for predictive decision support. HTI-5 was published as a proposed rule on 29 December 2025; its comment period closed 27 February 2026, and no final rule had issued as of this issue’s publication date — re-checked 18 July 2026.

Signal & Noise is written under the pen name Synthia Cipher. AI tools draft and critique; the human author owns the editorial judgment, final wording, and published claims. If something here is wrong, the fault is the author's, not the algorithm's.