The Missing Variable in AI: What a Human Health Navigator Reveals About Intelligent Systems

The intervention that changed the question

What if the most important measure of an intelligent system is not whether it predicts correctly, but whether it changes what people do next?

That question becomes visible in an unexpected place: health care navigation. A navigator does not cure a patient, prescribe medication, or replace a physician. The navigator helps a person find transportation, food, housing assistance, utility support, or a community clinic. The intervention seems modest. Yet evidence suggests it can reduce emergency department visits, even when its effects on many other health outcomes are difficult to detect.

This is more than a lesson about social services. It is a compact demonstration of a problem that increasingly defines artificial intelligence. A decision system does not merely observe a world. Once deployed, it becomes part of the world it is trying to understand. Its recommendations alter behavior, behavior alters demand, demand changes the system, and the resulting data can make the original model look better or worse for reasons that are not obvious.

The practical implication is profound. Organizations should stop treating AI as a collection of isolated prediction engines and start treating it as a collection of interventions inside living systems. The appropriate unit of design is not the model. It is the pathway connecting a person, a decision, an institution, and a consequence.

The future of reliable AI belongs to those who can model the environment around a prediction, not merely improve the prediction itself.

Why small interventions produce ambiguous results

Health care navigation helps expose the difference between access and outcomes. If a person with an urgent but noncritical problem can reach a primary care provider, obtain food, or avoid an interruption in electricity, that person may be less likely to use the emergency department. But a reduction in emergency visits does not automatically produce a measurable improvement in every clinical outcome. The patient may still face chronic illness, unstable housing, insufficient income, or limited trust in institutions.

This creates a familiar but underappreciated evaluation problem. An intervention can succeed at changing behavior without producing a dramatic change in the final metric selected by an organization. The causal chain is long:

1. A need is identified.
2. A person accepts assistance.
3. The person reaches a resource.
4. The resource is available and suitable.
5. The person changes a health related behavior.
6. The health system records the consequence.

Failure at any point can obscure success at another. A navigator may correctly identify a transportation barrier, yet the local clinic may have no appointments. A referral may be accepted, but the patient may not have a phone. A person may avoid the emergency department, but only because they delayed care elsewhere. A statistically weak result, therefore, does not always mean an ineffective intervention. It may mean that the organization lacks sufficient visibility into the pathway between action and outcome.

The same logic applies to AI. Suppose a hospital deploys a model that predicts which patients are likely to return to the emergency department. It then sends certain patients reminders, offers transportation, or assigns a care coordinator. The model is no longer being tested in the original population. Its predictions change the very events it was built to forecast.

This is called a feedback effect, but the phrase is too mild. The system has become reflexive. It acts on its own forecast, thereby changing the future that would have validated or falsified the forecast. A model that predicts high demand may trigger staffing increases, which prevent the predicted congestion. A fraud model may cause investigators to scrutinize a category of transactions, producing more detected fraud in that category. A customer service system that routes difficult cases to specialized agents may improve resolution while making the original channel appear unusually problematic.

The central mistake is to interpret the changing metric as a simple report card. In a reflexive system, the metric is also a trace of the intervention.

From isolated models to navigable worlds

As organizations deploy more models, they often imagine that complexity comes primarily from model size. In practice, complexity frequently comes from interaction. A demand forecast feeds a staffing tool. The staffing tool changes service capacity. Capacity affects customer behavior. Customer behavior changes the data used by the next forecast. Several models, business processes, human judgments, and external services form one system, even if different teams own each piece.

This is why the traditional model at a time approach breaks down. Each model can look accurate in isolation while the combined system behaves unpredictably. The problem resembles a city whose traffic lights were optimized independently. Every light may be well timed at its own intersection, yet the network can still generate gridlock because the lights influence one another.

Simulation offers a way to reason about this network. A simulation is not simply a forecast with more data. It is a reusable environment in which agents, rules, resources, and constraints interact. In a health care setting, a useful environment might represent patients with different needs, clinics with limited capacity, transportation options, emergency departments, navigators, insurers, and waiting times. One could then test questions that ordinary prediction cannot answer:

• What happens if navigation is offered only to people with the highest predicted risk?
• What happens if the intervention increases demand for primary care faster than clinics can absorb it?
• Which barriers cause a person to return to the emergency department even after a referral?
• Does prioritizing one population unintentionally lengthen access for another?
• How much does the result depend on a navigator, a text message, a community organization, or a change in scheduling capacity?

The point is not to create a perfect digital replica of society. That goal would be both impossible and dangerous. The point is to make assumptions explicit and expose interactions before they become expensive or harmful in production.

This reframes the purpose of AI platforms. Their highest value may not lie in generating more models. It may lie in preserving and reusing world models: shared representations of concepts, processes, resources, people, and causal relationships. A classifier can be replaced. A well constructed representation of how a domain works becomes institutional memory.

The distinction matters because organizations are accumulating technical debt in a form that is easy to underestimate. Technical debt is not only obsolete code or undocumented pipelines. It also includes incompatible definitions, one off integrations, unclear ownership, hidden assumptions, and models that cannot be observed once they interact with other systems. An organization may have thousands of useful components and still lack a coherent account of how those components affect one another.

Observability is a theory of consequences

Most monitoring asks whether a model is functioning according to a narrow specification. Is latency acceptable? Has accuracy declined? Is the input data complete? Those questions matter, but they are not enough for a system whose outputs influence human decisions.

Observability asks a deeper question: what state is the system in, and why did it arrive there? It connects logs, data changes, model outputs, application behavior, operational processes, and human actions. In a composite AI system, explainability cannot stop at which features influenced one model. It must also examine how multiple components combined to produce an outcome.

Consider the health navigation example. A fall in emergency department use could be beneficial, neutral, or harmful. To distinguish among them, an organization would need to observe more than the final visit count. It would need to track whether patients obtained appointments, whether wait times changed, whether unmet needs shifted, whether certain groups abandoned care, and whether navigators were concentrating on people easiest to reach rather than people with the greatest need.

This illustrates a useful framework for evaluating intelligent interventions. Every system should be examined across four layers:

1. Prediction

What does the system believe will happen? How confident is it? Which populations or conditions make the prediction unreliable?

2. Action

What does the system cause someone or something to do? Is the action a recommendation, a resource allocation, a message, a delay, or an escalation?

3. Adaptation

How do people, institutions, and other models respond to that action? Do they change their behavior, workload, or strategy?

4. Consequence

What happens to the outcome, and who bears the cost when the system is wrong or when its success creates new demand?

Most AI projects measure the first layer and sometimes the fourth. They neglect the middle. Yet action and adaptation are where the system acquires its social character.

A simulation can help connect the four layers. Observability can then compare the simulated pathway with the actual one. When the two diverge, the organization has a starting point for investigation. Perhaps the model misunderstood a domain concept. Perhaps a community resource became unavailable. Perhaps staff developed an informal workaround. Perhaps the system optimized an easy proxy while the real objective remained unmet.

This is also where foundation models introduce a special risk. Their broad capabilities make them reusable across many applications, but their errors and biases can travel across those applications as well. A general language model used for summarization, triage, translation, and customer communication becomes a shared dependency. An error in its interpretation of dialect, urgency, or cultural context can propagate through several downstream decisions.

The sensible response is not to reject general models or assume that larger models will solve every problem. It is to place them inside observable, bounded workflows. Start with tasks where errors can be reviewed. Document known weaknesses. Use the smallest model that meets the need. Make escalation to a person part of the architecture, not an emergency patch after deployment.

The new role of the AI team

Once AI is understood as a system of interventions, the composition of the team must change. A group trained only to optimize statistical performance cannot fully assess a system that includes institutional incentives, human cognition, resource scarcity, legal obligations, and unequal exposure to risk.

This is not an argument for adding experts as ceremonial reviewers. It is an argument for changing the design process itself. Software engineers understand reliability and interfaces. Domain experts understand what a category means in practice. Social scientists can examine behavior, power, and unintended effects. Operations specialists understand queues and bottlenecks. Philosophers and ethicists can clarify what the system is actually optimizing. Frontline workers can identify workarounds invisible in formal documentation.

Their collaboration requires shared environments, not merely more meetings. A common simulation, concept model, metadata layer, and observability practice allow different disciplines to examine the same system from different angles. Without that shared structure, each group produces a partial artifact that another group must reinterpret, increasing technical debt.

The most mature AI organization may therefore look less like a laboratory producing models and more like a science and systems consultancy embedded in the business. Its job is to construct environments for learning, test interventions, monitor behavior, and help the organization reason about uncertainty. Model creation becomes one capability among several, not the definition of intelligence work.

A practical organizational test follows: when a model fails, can the team explain only that its accuracy declined, or can it identify which change in data, process, resource availability, or human behavior caused the failure? The second team is not simply better at machine learning. It has a better theory of its own environment.

Key Takeaways

• Measure the pathway, not just the endpoint. For every AI intervention, map the sequence from prediction to action, adaptation, and consequence. Add measures for each stage.
• Build one reusable environment before building many more models. Represent the people, resources, rules, constraints, and processes that define the domain. Use it to test rare scenarios and interactions.
• Treat observability as causal investigation. Monitoring tells you that a metric changed. Observability should help explain why, across data, infrastructure, models, applications, and human workflows.
• Design for changing behavior. Assume that users, staff, customers, and other models will adapt to an AI system. Test what happens when they do.
• Make interdisciplinary work operational. Give domain experts, engineers, social scientists, and frontline workers shared concepts, shared data lineage, and shared simulation workflows.

The most important shift is conceptual. We have spent years asking whether an AI model is accurate. That question remains useful, but it is incomplete whenever the model enters a human institution. The better question is: accurate about what, for whom, under which intervention, and with what consequences for the surrounding system?

Health navigation makes the point with unusual clarity. A modest human intervention can redirect demand, change engagement, and improve one outcome while leaving others uncertain. That ambiguity is not merely a statistical nuisance. It is evidence that care is a dynamic system, and that outcomes emerge from relationships among people, resources, decisions, and institutions.

AI will inherit the same condition. The winners will not be the organizations that deploy the greatest number of models. They will be the organizations that can understand, simulate, and observe the worlds their models help create.

In the end, intelligence is not the ability to make a prediction in isolation. It is the ability to anticipate how a prediction will move through a world full of other minds, limited resources, and unintended responses.