Why Building a Medical AI Benchmark Feels Like Building an EHR

What do healthcare software and AI evaluation have in common?

The most dangerous mistake in software is to treat a difficult problem as if it were merely a storage problem.

That is the hidden lesson in medicine. It is also the hidden lesson in building intelligent systems. Both domains tempt engineers into believing that if they can just define the fields, standardize the labels, and add enough rules, the mess will become manageable. In practice, the mess pushes back. It pushes back through ambiguity, conflicting incentives, shifting standards, incomplete evidence, and users who need the system to be both precise and forgiving at the same time.

A hospital record system and an AI benchmark look like very different things on the surface. One handles patients, prescriptions, lab tests, identities, timestamps, and legal risk. The other handles questions, answers, scoring, and model comparisons. But at a deeper level, both are attempts to represent reality in a form that can be computed. And reality is never as clean as the schema or the benchmark would like.

That is why the two domains illuminate each other so well. An EHR exposes what happens when we pretend the world is neatly structured. An AI benchmark exposes what happens when we pretend intelligence can be judged by a small set of neat tasks. In both cases, the system fails not because data is missing, but because the underlying model of the world is too shallow.

The real problem is not data entry, it is world modeling

People often think the core challenge of an EHR is user interface design, integration, or compliance. Those are real problems, but they are downstream of a much harder one: how do you model a living, changing, uncertain world without flattening it into nonsense?

Medicine is full of relationships that are not simple facts. A clinician does not merely record that hypertension exists. They may need to capture that the hypertension likely caused heart failure, that the heart failure contributed to renal dysfunction, that the renal dysfunction may have been worsened by a drug, and that the drug was prescribed because of a different problem whose diagnosis is only tentatively established. The system must hold causality, uncertainty, severity, timing, and provenance all at once.

That is not just a database challenge. It is an ontology challenge. It is also a judgment challenge.

This is where many systems fail. They treat each observation as a static object. But clinical reality is not a list of objects. It is a chain of hypotheses, revisions, and consequences. A diagnosis is often not a verdict. It is a working theory under pressure.

The same is true of AI benchmarks. A benchmark question seems like a simple object: prompt in, answer out, score assigned. But the moment you ask what the benchmark is actually measuring, the simplicity collapses. Is it measuring memorization, reasoning, calibration, robustness to ambiguity, domain knowledge, or alignment to the evaluator’s preferences? Is the answer right because the model understood the task, or because the task matched patterns seen during training? Is the score stable under paraphrase, context shifts, or adversarial prompts?

A useful system does not merely store facts. It stores the structure of uncertainty around facts.

That is the shared lesson. In medicine, the record should preserve what was observed, what was inferred, what was suspected, and how confident anyone was at the time. In AI evaluation, the benchmark should preserve what was asked, what kind of competence was actually being tested, and how much of the score reflects task design rather than genuine capability.

The deeper problem is not representation. It is epistemology. What counts as knowing? What counts as evidence? What counts as a stable truth in a changing environment?

Why “standardization” often creates the illusion of understanding

Every large system promises order through standards. In healthcare, that means coding systems, test vocabularies, device identifiers, time formats, interoperability protocols, and security frameworks. In AI, it means benchmark suites, leaderboards, scoring rubrics, and normalized performance metrics. The promise is appealing: if everyone speaks the same language, complexity becomes manageable.

But standardization only helps if the standard maps cleanly onto reality. When it doesn’t, the standard becomes a theater set. It looks like structure from a distance while hiding incoherence underneath.

Consider clinical coding. A disease label in one code system may not map neatly to another. Drug nomenclature is even worse. Lab tests that seem identical may differ by panel, method, units, or reference range. A blood glucose result is not just a number, it is a number with context: fasting or random, whole blood or plasma, current calibration, timestamp precision, collection site, and whether the specimen was compromised. Without that context, the number is decorative rather than useful.

AI benchmarks have the same problem. A score can be precise while the underlying meaning is fuzzy. A model may improve on one benchmark because the benchmark favors a narrow style of answer, because the dataset leaked into training, or because the evaluation rewards form over substance. The leaderboard may be standardized, yet the thing being measured may not be what people think it is.

This creates a dangerous organizational habit. Leaders begin to manage the metric instead of the mission. In healthcare, the result is documentation optimized for billing or compliance rather than care. In AI, the result is systems optimized for benchmark performance rather than real-world utility.

The technical mistake is obvious in hindsight: the map is being treated as the territory. The cultural mistake is subtler. Once a standard acquires authority, people stop asking whether it still describes the thing that matters.

The hidden currency is not data, but provenance and trust

If there is one concept that should sit at the center of both healthcare software and AI evaluation, it is provenance.

In an EHR, provenance means knowing who recorded a fact, when, under what conditions, with what confidence, and with what possible conflicts. A blood pressure reading is not just a number. It came from a device with a calibration history, observed by a person in a particular context, on a patient whose identity must be resolved correctly across systems. A diagnosis is not just a label. It is a claim made by someone, based on evidence that may later be revised.

In AI benchmarking, provenance means knowing where the tasks came from, who curated them, what biases they embed, how they were scored, and what kinds of failures they fail to expose. A benchmark number without provenance is like a lab result without units and method. It may look authoritative while being nearly unusable.

This is why both fields are fundamentally hostile to shallow interoperability. You can export data, but can you export meaning? You can move a record, but can you move its uncertainty, its history, its caveats, and its context? You can compare two models, but can you compare the conditions under which those models were evaluated?

The most mature systems do not pretend to eliminate ambiguity. They preserve it in a structured way.

That is the real design principle:

Do not force the world into certainty. Build a system that can hold uncertainty without collapsing.

In medicine, that means supporting multiple hypotheses, severity levels, confidence levels, and temporal relationships. In AI, it means building evaluations that can reveal not just whether a model answered correctly, but whether it knew when it was uncertain, whether it remained stable under perturbation, and whether its competence transferred across settings.

A useful mental model: three layers of reality

One reason these systems become unmanageable is that people mix up three distinct layers of reality.

1. The clinical or task layer

This is the actual real world event. A patient feels pain. A lab value changes. A model answers a question.

2. The interpretive layer

This is the human judgment attached to the event. The pain may be ischemic. The lab value may indicate worsening renal function. The model’s answer may be correct, but only under a specific interpretation of the prompt.

3. The administrative layer

This is what the institution needs for billing, reporting, auditing, compliance, or leaderboard publication.

The failure mode appears when the administrative layer starts driving the other two. In hospitals, documentation can become a performance against the system rather than a reflection of the patient. In AI, benchmark optimization can become a performance against the evaluator rather than a reflection of intelligence.

A well designed system keeps these layers distinct but linked.

For example, imagine a clinician records, “possible pulmonary embolism, moderate confidence, pending CT angiography.” That single entry contains a fact, an interpretation, and a temporal state. Or imagine an AI benchmark item that records not just the model’s answer, but whether the task required retrieval, reasoning, multi step planning, or calibration under ambiguity. That benchmark would measure something more meaningful than raw correctness.

This is where the analogy becomes powerful. The best system is not the one that erases ambiguity. It is the one that makes ambiguity computable without making it invisible.

The best interfaces are not simpler, they are more honest

There is a myth in software that better design means less complexity on screen. Sometimes that is true, but in complex domains it is misleading. The goal is not to hide complexity. The goal is to reveal only the right complexity at the right time.

In an EHR, clinicians do not need all data at once. They need the right subset, with the right relationships, in the right sequence. Burden them with too little context and errors rise. Burden them with too much and cognition collapses. The interface becomes a factory for defensive behavior: endless clicking, copy forward habits, and documentation that satisfies the system while obscuring the patient.

AI evaluation has a parallel issue. A benchmark can be too thin to capture real capability, yet too bulky to be interpretable. If everything is measured, nothing is learned. If a benchmark suite is merely a giant pile of tasks, it becomes the equivalent of a cluttered chart. The metric exists, but the insight does not.

The answer is not minimalism. It is selective revelation.

A good interface or benchmark should answer three questions:

1. What is the current state?
2. What is uncertain?
3. What should the user do next?

In healthcare, that might mean showing the active problem list, the confidence behind each diagnosis, and the decisions that are time sensitive. In AI, that might mean showing not just average score, but performance by task type, robustness under variation, and failure modes by category.

When systems do not do this, people improvise their own shadow systems. Clinicians write notes for humans because the machine does not capture clinical reasoning well. Researchers create side spreadsheets because benchmarks do not reveal enough nuance. The institution then mistakes these workarounds for user resistance, when in fact they are signals of design failure.

Key Takeaways

• Treat uncertainty as a first class object. Record confidence, timing, provenance, and revision history, not just final outputs.
• Separate interpretation from administration. A diagnosis, a score, and a billing code are related but not identical. Do not collapse them into one layer.
• Ask what your standard actually measures. A code, benchmark, or metric is useful only if it maps to the real phenomenon you care about.
• Design for revision, not just capture. In complex domains, the truth changes. Good systems preserve the path of change.
• Prefer selective revelation over false simplicity. Show the right complexity at the right moment instead of flattening everything into one uniform view.

The deeper lesson: intelligence is not certainty, it is disciplined uncertainty

The most interesting connection between medical software and AI evaluation is that both reward a mature relationship with ambiguity. The novice wants a single clean answer. The expert wants a structure that can survive revision.

That is why the dream of perfect EHRs and perfect benchmarks is misguided. The problem is not that we have failed to build sufficiently elaborate systems. The problem is that we keep trying to turn living, moving, contested reality into static categories and then acting surprised when the categories break.

A better ambition is more humble and more powerful. Build systems that can keep track of what we know, what we think we know, what we do not know, and how those states change over time. That is what good clinical systems need. It is also what serious AI evaluation needs.

In the end, both fields are asking the same question: can we make a machine that helps humans think without pretending that thinking is simple?

The answer is yes, but only if we stop worshipping the clean model and start respecting the messy world it is supposed to represent.