The Hidden Requirement of AI: Knowing What to Ask For

What if the biggest risk in artificial intelligence is not that machines will misunderstand us, but that only some people will know how to make themselves understood?

A system can be accurate, fast, and technically impressive while still failing the people it was built to serve. It may satisfy every documented requirement and yet leave users unable to accomplish their real goals. The problem becomes more subtle with AI because the quality of the result depends partly on a skill that is rarely listed in the requirements: the ability to frame a task, provide context, evaluate an answer, and collaborate with the system through iteration.

This creates a new version of an old engineering distinction. Verification asks whether we built the thing right. Validation asks whether we built the right thing. In AI, there is a third question hiding inside both: who has the knowledge and confidence to turn the system into something useful?

My thesis is simple: AI systems should be evaluated not only by their capabilities, but by the distribution of capability they create among users. A tool that produces remarkable gains for highly educated people while confusing or marginally helping everyone else may be technically successful and socially underperforming. The central design task is therefore not merely to make AI more powerful. It is to make productive interaction with AI less dependent on prior privilege.

The difference between a working system and a useful one

Imagine a hospital introducing an AI assistant intended to help nurses prepare discharge instructions. The product team verifies that the assistant can generate documents, follows the approved template, cites the relevant clinical information, and passes a set of accuracy tests. Every requirement in the specification is checked off.

Then the system enters actual use. Experienced nurses learn to add patient history, specify the reading level, request warnings about ambiguity, and ask the assistant to identify missing information. Others enter a short command, receive a polished but incomplete document, and assume the result is sufficient. The system behaves exactly as designed. Yet its practical value varies dramatically according to the user’s ability to guide it.

This is a validation failure, but not in the familiar sense of a product that crashes or produces obviously wrong answers. It is a participation failure. The system is fit for purpose only for users who can discover the interaction patterns that unlock its value.

The distinction between verification and validation helps expose the problem. Verification is comparatively comfortable because it operates on explicit requirements. If the requirement says that the assistant must produce a document within ten seconds, the team can measure response time. If it says that the document must contain specified fields, the team can inspect those fields. A verification cross reference matrix can map each requirement to a test, an inspection, or another method of checking.

Validation is harder because it asks whether the requirements themselves describe what matters. Did the team ask whether nurses can identify when the output is incomplete? Did it test whether patients understand the instructions? Did it observe how the system is appropriated in a busy ward, where users adapt tools to local routines rather than follow the intended workflow?

AI makes these questions unavoidable because the system is partly completed by the user. A conventional calculator has a relatively stable interface between intention and result. An AI assistant has a more elastic interface. The user’s framing changes the task, the context changes the answer, and the evaluation of the answer changes the next prompt.

The user is not simply pressing a button. The user is helping specify the button’s meaning.

A requirement that omits the user’s ability to express, refine, and judge a task is not a complete requirement for an AI system.

Prompting is not a trick. It is an educational technology

People often talk about prompting as though it were a bag of clever phrases. That description misses the deeper issue. Effective prompting is a compact form of professional reasoning.

To use an AI system well, a person must often do at least five things:

1. Define the actual objective rather than the first visible request.
2. Supply relevant context and distinguish it from distracting information.
3. Specify constraints, standards, audience, and acceptable tradeoffs.
4. Inspect the output for omissions, errors, and unjustified confidence.
5. Revise the task when the initial result reveals a misunderstanding.

These are not merely interface skills. They resemble the skills taught through advanced education: decomposing a problem, making assumptions explicit, comparing alternatives, and giving precise feedback. This helps explain why AI tends to be used more, and appears to generate larger productivity gains, on tasks requiring higher education. The advantage may not arise only because educated workers have more valuable tasks. It may also arise because they possess more of the invisible knowledge required to collaborate with a probabilistic system.

Consider two users asking for help with a market analysis. The first writes, “Summarize this industry.” The second writes, “Compare the three largest competitors for a mid sized firm entering the European market. Separate verified facts from inference, identify which claims depend on data newer than twelve months, and end with five questions that require human research.”

The second request is better not because it contains magical wording. It expresses a clearer model of the work. It identifies scope, audience, evidence, uncertainty, and the next decision. The system’s output improves because the user has already performed part of the intellectual labor.

This produces a troubling feedback loop. People with more educational experience use AI more collaboratively. Collaborative use tends to produce better outputs and larger productivity gains. Those gains then give already advantaged users more time to learn, experiment, and refine their methods. Users who do not know how to structure a request may conclude that AI is unreliable, when the deeper problem is that the interface silently assumes a level of task literacy they have not been given.

The result is a form of interaction inequality. Access to the tool is widespread, but access to its best performance is not.

This is analogous to giving two people the same camera while providing only one of them with knowledge of lighting, composition, and lenses. The camera is technically available to both. Its expressive power is not.

The missing metric: capability lift, not capability alone

Most evaluations ask what an AI system can do under controlled conditions. That is a necessary starting point, but it is not enough for deployment. A system can demonstrate extraordinary capability in an expert’s hands and deliver little value to ordinary users.

A more useful framework separates four layers of performance:

1. System capability

Can the model produce a correct, useful, or creative result under favorable conditions? This is the layer most benchmark tests measure.

2. Interaction accessibility

Can a new user discover what information to provide, what kinds of requests are possible, and how to recover from a poor answer? This is an HCI question, not simply a model question.

3. Judgment support

Can the system help users determine whether its output is reliable? An answer that is easy to generate but hard to assess may increase the speed of error.

4. Capability lift

How much does the system improve outcomes for different groups of users compared with their starting point? This is the most important layer if the goal is broad productivity rather than impressive demonstrations.

Capability lift should be measured across a distribution of users, not just averaged across them. Suppose an AI tool improves expert performance by 35 percent, intermediate performance by 15 percent, and novice performance by 2 percent. A single average may make the product look successful. A distribution reveals that the tool is amplifying existing differences.

The opposite pattern is possible. A well designed system may provide experts with modest gains while giving novices templates, explanations, checklists, and guided questions that substantially improve their work. Such a system might appear less spectacular in an expert demonstration but create more public value in real deployment.

This suggests a practical evaluation question: What is the minimum interaction competence required to obtain meaningful value from the system, and who has been taught that competence?

The answer should become part of the product specification. If users must know how to request evidence, the interface should teach that. If they must recognize when context is missing, the system should ask for it. If a task requires several rounds of refinement, the product should make iteration visible rather than presenting one answer as final.

Design for appropriation, not obedience

People rarely use technologies exactly as their designers imagine. They adapt them to local habits, pressures, incentives, and workarounds. A note taking tool becomes a project tracker. A messaging platform becomes an informal approval system. An AI assistant may become a tutor, a ghostwriter, a critic, a search engine, or a source of plausible excuses.

This process of adaptation, often called appropriation, is not a defect to eliminate. It is evidence that a system has entered real human practice. But it creates a design responsibility: anticipate the ways people will reinterpret the tool, especially when those reinterpretations affect quality and safety.

For AI, appropriation can be productive when users learn to treat the system as a collaborator rather than an oracle. They may ask it to generate alternatives, challenge assumptions, translate specialist language, or simulate the objections of an audience. It can be harmful when users treat fluent output as completed work, delegate judgment, or use generic prompts for tasks that require local knowledge.

A useful mental model is the three stage collaboration loop:

Frame, generate, inspect. First, the human frames the objective and constraints. Then the system generates possibilities. Finally, the human inspects the result against evidence, purpose, and consequences. The loop repeats when the inspection reveals a gap.

Weak AI design hides this loop. It gives the user a blank text box and treats the first answer as the product. Strong design supports each stage. It offers examples of goals rather than only examples of commands. It asks targeted questions when the task is underspecified. It shows uncertainty and provenance where possible. It encourages comparison between alternatives. It makes revision feel normal rather than like a failure.

This is also where education and product design meet. We should not force every user to become an expert prompt engineer. That would be equivalent to requiring every driver to understand engine mechanics. The system should absorb routine complexity while teaching users the parts of reasoning that cannot safely be automated.

In other words, the goal is not to hide the intelligence of the system. It is to reveal the structure of the task.

What organizations should do now

Organizations adopting AI often begin with a tool selection question: Which model is most capable? They should add a more consequential question: Which model of the user does this tool assume?

A responsible deployment can begin with a capability and inclusion audit. Before launch, test the system with users who differ in education, domain expertise, confidence with technology, language background, and familiarity with the workflow. Give them realistic goals, not carefully written benchmark prompts. Observe not just the final answer but the path they take to reach it.

Look for four kinds of failure:

• Expression failure: users cannot translate their real goal into a request.
• Discovery failure: users do not know what the system can do or what context it needs.
• Judgment failure: users accept weak outputs because quality criteria are unclear.
• Transfer failure: users cannot apply a useful result to the actual setting where decisions are made.

Each failure calls for a different remedy. Expression failure may require guided task builders or examples organized by goal. Discovery failure may require contextual prompts and progressive disclosure. Judgment failure may require comparison tools, source inspection, or domain specific checklists. Transfer failure may require integration with existing workflows rather than another standalone chat window.

Training should follow the same logic. Do not teach people a collection of prompt formulas detached from their work. Teach them to specify purpose, audience, constraints, evidence, and uncertainty. Practice the full collaboration loop with real tasks. Reward users for detecting flaws, not merely for producing polished outputs.

Finally, measure outcomes by user segment. Ask who improved, who stagnated, and who became more dependent without becoming more capable. A deployment that raises the performance of experts while leaving novices behind may still be worthwhile, but it should not be described as universally empowering.

Key Takeaways

• Treat user interaction competence as a product requirement. Specify what users must know to get reliable value, then decide which parts the system should teach or automate.
• Evaluate capability lift by user group. Do not rely on an overall average that conceals unequal gains.
• Design for the full collaboration loop. Support framing, generation, inspection, and revision instead of presenting one answer as finished work.
• Test realistic appropriation. Observe how people adapt the system in their actual environments, including shortcuts, misunderstandings, and informal uses.
• Teach judgment, not prompt recipes. The durable skill is knowing how to define a goal, expose assumptions, evaluate evidence, and revise the task.

The most important AI divide may not be between people who have access to the technology and people who do not. It may be between people who can turn vague intentions into structured collaboration and people who are expected to benefit from a system that never teaches them how.

That changes the meaning of a successful AI product. Success is not proved when the model gives an expert an astonishing answer. It is proved when a diverse set of users can pursue meaningful goals, recognize the limits of the result, and become more capable through the interaction.

The future of AI will therefore be shaped by an unglamorous design decision: whether we treat prompting as a private talent or as a public capability to be cultivated. If it remains a private talent, AI will magnify the advantages of those already trained to formulate, question, and judge. If it becomes part of the system’s design, AI can do something more ambitious than automate tasks. It can distribute better ways of thinking.

The real test is not whether the machine can answer the question. It is whether more people can learn to ask a better one.