The Hidden Politics of Making People Legible

What do a nineteenth century ledger recording an enslaved person’s value and a modern medical record generating an AI drafted message have in common?

At first, the answer seems obvious: nothing. One belongs to a system of racial violence and forced labor. The other promises to reduce administrative burden and help clinicians respond to patients. To equate them would be morally and historically careless.

Yet they illuminate the same underlying problem: whenever institutions convert human lives into manageable information, they gain power over what can be seen, decided, and ignored.

That connection matters because artificial intelligence is arriving not in an abstract digital world, but inside institutions already shaped by centuries of classification. The central question is not simply whether a language model can draft a patient message or suggest a useful data query. It is whether the systems that organize human beings can become more efficient without making human beings easier to flatten, misread, and control.

The common technology beneath very different systems

The transatlantic slave trade did not operate through violence alone. Violence was indispensable, but violence became scalable through administration. People were abducted, transported, priced, recorded, insured, exchanged, and governed through elaborate commercial and legal systems. A human being was transformed into an entry in a ledger, a unit of labor, a liability, a shipment, or an asset.

This did not mean enslaved people ceased to be human. It meant that institutions built procedures for treating their humanity as irrelevant to the decisions being made. The record did not merely describe reality. It helped produce a reality in which a person could be legally owned and economically optimized.

Modern health care records are created for a radically different purpose. They can preserve medical history, coordinate treatment, document consent, support research, and help clinicians see patterns across time. An electronic health record can protect a patient from being forgotten between appointments. It can also burden clinicians with documentation and force complex lives into standardized fields.

The important distinction is not between an evil ledger and a benevolent database. It is between the purposes, safeguards, and power relationships governing a system of classification. Both kinds of systems make people legible. The moral question is what happens after legibility is achieved.

A record can make someone visible to care. It can also make them visible to surveillance, exclusion, pricing, or suspicion. The same act of recording can protect a person in one context and expose them in another.

The danger is not that data is impersonal. The danger is that institutions may use impersonal representations to make personal decisions while forgetting what the representation leaves out.

Efficiency is never just efficiency

The appeal of generative AI in health care is easy to understand. A clinician receives dozens of patient messages, each requiring attention, context, tone, and documentation. A system that drafts a response could return time to the exam room. A tool that interprets a natural language request and recommends relevant metrics could make institutional data more accessible. Voice software that summarizes a conversation could reduce the friction of charting.

These are not trivial gains. Administrative overload contributes to exhaustion, delays, and the erosion of attention. If technology reduces clerical work, it may create more space for listening, diagnosis, and explanation.

But productivity is not a neutral measure. To say that a system increases productivity is to ask: productivity for whom, measured how, and at whose risk?

Suppose an AI system drafts a response to a patient who writes, “I have been feeling strange since starting the medication, but I do not want to be difficult.” The draft may be grammatically polished and medically plausible. It may also miss fear, understatement, cultural context, or a warning sign buried in an apparently ordinary sentence. If the clinician accepts it quickly because the system has made the response easy, the technology has not merely saved time. It has redistributed attention away from interpretation.

The same issue appears in data exploration. A provider asks a system to find patterns in a population. The model suggests metrics based on the structure of the database and the language of the request. But what is absent from the database may be precisely what matters: unstable housing, distrust of institutions, informal caregiving, language barriers, prior discrimination, or the fact that some groups are less likely to seek care in the first place.

The system can make a question easier to ask while quietly narrowing the universe of answers.

This is the legibility paradox: the more efficiently a system represents people, the easier it may become to act on them, but not necessarily to understand them. Better representation can increase both care and control. The outcome depends on who interprets the representation and who has the authority to challenge it.

From records to recommendations: where power moves

A useful way to evaluate AI in institutions is to distinguish four stages that are often collapsed into one:

1. Capture: What aspects of a person become data?
2. Compression: What complexity is discarded when the data is summarized?
3. Recommendation: What actions or interpretations does the system make more likely?
4. Accountability: Who can contest the result, and who bears the consequences if it is wrong?

This framework reveals why “human in the loop” is necessary but insufficient. A human can remain present while exercising almost no meaningful judgment. If the clinician is rushed, if the interface presents one fluent draft, or if institutional metrics reward speed, the person may function less as an independent reviewer than as a ceremonial approver.

Human oversight is real only when the human has the time, knowledge, authority, and incentive to disagree.

Consider a drafted patient message. At the capture stage, the system receives the patient’s words and relevant clinical context. At the compression stage, it produces a coherent interpretation. At the recommendation stage, it proposes a reply, perhaps one that reassures, advises monitoring, or recommends an appointment. At the accountability stage, the patient may never know that an AI system shaped the response, and the clinician may not be able to reconstruct why certain language was selected.

The polished output can hide the chain of judgment that produced it.

This is where the history of bureaucratic classification becomes relevant. Systems of domination often depended on making responsibility diffuse. A person could claim to be following a rule, maintaining a ledger, processing a shipment, or enforcing a contract. The cruelty was distributed across procedures, allowing each participant to see only a small part of the whole.

Modern health care is not a slave system, and the comparison must not erase that fact. The lesson is narrower and more useful: administrative distance can make consequential decisions feel less consequential. When a recommendation arrives as a field, score, draft, or workflow prompt, the institution may treat it as an object rather than as a judgment affecting a person.

Generative AI intensifies this problem because it produces language that sounds intentional. A spreadsheet cell does not pretend to understand a patient. A fluent draft can create that impression. Its danger is not only error. It is the possibility that fluency will be mistaken for comprehension.

The missing dimension: contestability

Most discussions of responsible AI emphasize accuracy, bias testing, privacy, and human review. These are essential, but they miss a deeper institutional requirement: contestability.

A contestable system allows the affected person and the responsible professional to ask:

• What information shaped this output?
• What assumptions did the system make?
• What relevant information was unavailable?
• Who benefits from this recommendation?
• How can the decision be corrected?
• What happens when the person involved disagrees?

Contestability changes the design objective. The goal is not to build an AI system that appears so reliable that nobody questions it. The goal is to build one that makes questioning possible and worthwhile.

For patient communication, this could mean showing the clinician which parts of the record informed a draft, flagging uncertainty, and making it easy to revise the response in a way that preserves the patient’s own language. It could mean recording whether a message was generated, heavily edited, or written from scratch. It could mean giving patients a clear route to request correction when an automated summary misrepresents them.

For data analysis, contestability could require the system to display the definition of each suggested metric, identify missing populations, and distinguish correlation from evidence that would justify intervention. A recommendation should come with an account of its boundaries, not merely a confident answer.

This suggests a practical principle:

The more an AI system influences a decision about a person, the more visible the system’s uncertainty, assumptions, and alternatives should become.

In low stakes settings, convenience may dominate. In high stakes settings, friction can be a safeguard. A second confirmation, an explanation of missing data, or a required review of alternative interpretations may look inefficient. But an institution that removes every pause may be optimizing for throughput when it should be optimizing for judgment.

Designing systems that preserve the person behind the record

How can health care organizations use generative AI without allowing efficiency to become the only value?

First, they should treat the record as a partial witness, not a complete portrait. Every patient chart is shaped by what the patient chose to disclose, what a clinician had time to ask, what the interface made easy to document, and what the institution considered relevant. AI systems should be designed to expose these limits rather than conceal them.

Second, organizations should evaluate systems by more than speed. Useful measures include whether patients understand the response, whether clinicians catch important concerns, whether disparities in follow up widen or narrow, and whether patients can correct errors. A faster workflow that produces more misunderstandings is not productive in any meaningful sense.

Third, deployment should include the people most exposed to failure. Patients, nurses, interpreters, schedulers, clinicians, and administrative staff experience the same system differently. A tool that saves a physician five minutes may create additional work for a scheduler or make a patient feel dismissed. Testing must examine the whole chain, not merely the user who clicks the final button.

Fourth, institutions should preserve a clear line of responsibility. “The model suggested it” cannot become an acceptable explanation for a harmful decision. The system may be a contributor to an outcome, but accountability must remain assigned to identifiable people and organizations with the power to repair the damage.

Finally, designers should ask a question that is rarely included in technical specifications: What kind of relationship does this tool encourage? Does it help a clinician listen more carefully, or does it make the interaction feel preprocessed? Does it give patients a clearer path into care, or does it turn their words into another administrative object?

Technology does not merely automate tasks. It trains institutions in what to notice and what to ignore.

Key Takeaways

• Separate legibility from understanding. A structured record or fluent summary is a representation, not the person. Ask what important context may have been excluded.
• Make human review substantive. Give reviewers time, authority, and enough information to disagree with an AI output. A required click is not meaningful oversight.
• Build for contestability. Show the evidence, assumptions, uncertainty, and alternatives behind recommendations. Create simple correction paths for patients and staff.
• Measure the effects that matter. Track comprehension, missed concerns, unequal outcomes, and patient trust, not only response time or documentation volume.
• Keep responsibility visible. No system should allow an institution to treat an AI recommendation as an ownerless event. Someone must be accountable for the consequence.

The deepest lesson from the long history of classification is not that records are inherently oppressive, nor that technology inevitably dehumanizes. It is that the act of representing a person is always connected to power. Whoever defines the categories, controls the workflow, and decides what counts as evidence gains influence over the person represented.

Generative AI therefore presents health care with a choice more fundamental than whether to adopt a new tool. Institutions can use it to make people more efficiently processable, or they can use it to create more time and capacity for genuinely human judgment.

The difference will not be determined by the model alone. It will be determined by whether organizations treat efficiency as the destination or as a means. The best system is not the one that produces the smoothest answer. It is the one that helps an institution remain answerable to the human being hidden inside the data.