The Faster We Optimize, the More Carefully We Must Choose What Matters

The productivity question we are asking is too small

What if the greatest danger of artificial intelligence is not that it will replace human judgment, but that it will make bad judgment dramatically more efficient?

This possibility complicates the standard story about technological progress. Computers transformed offices, factories, and communication networks. A major wave of investment in computing and telecommunications helped produce a remarkable rise in productivity during the 1990s. The lesson seems straightforward: give organizations more powerful tools, and they will produce more with less.

But productivity is not the same thing as progress. An institution can process applications faster, answer customers more cheaply, and generate reports in minutes while becoming less humane, less creative, or less capable of recognizing what matters. Efficiency measures the relationship between inputs and outputs. It does not tell us whether the output was worth producing, whether the input was wisely chosen, or whether the system is pursuing the right objective.

This distinction becomes more important when the machine moves from physical work into cognitive work. Earlier technologies often automated repetitive actions. Generative AI can draft, classify, explain, persuade, imitate, and recommend across an enormous range of activities. It does not merely speed up the hands. It participates in the production of reasons, language, and decisions.

That creates a deeper question:

When machines become excellent at carrying out instructions, who is responsible for deciding which instructions deserve to exist?

The answer will determine whether AI becomes a genuine engine of progress or simply a faster way to preserve outdated systems.

The hidden conservative force inside every efficient machine

There is a paradox at the heart of computerization. A technology can be radically innovative in its capabilities while being conservative in its social effects. It may allow an organization to perform its existing routines with greater speed and precision without encouraging anyone to reconsider the routines themselves.

Imagine a city whose permitting office takes six weeks to approve a building project. The office introduces software that digitizes forms, routes them automatically, and flags missing information. Processing time falls to six days. This is an efficiency gain, and perhaps a meaningful one. Yet the city may still be asking for unnecessary forms, applying contradictory rules, and evaluating projects according to criteria designed decades earlier.

The software has improved the process without questioning the purpose of the process. It has made the old arrangement harder to challenge because the arrangement now appears modern and optimized.

This is the conservative tendency of technology: it often accepts the institution's definition of the problem as a fixed fact. Once a task has been formalized, a machine can help execute it. But formalization itself is a human act. Someone decided that a permit should be judged through those categories, that a customer service representative should maximize call volume, or that a student should be represented by a test score.

The most consequential decisions are therefore made before an algorithm begins calculating. They involve selecting the goal, defining success, choosing what counts as relevant evidence, and deciding which costs are acceptable. These are not merely technical choices. They are choices about values.

A computer can determine which applicant best matches a scoring rubric. It cannot, by calculation alone, establish whether the rubric captures the qualities a good employee needs. It can identify the shortest route to a destination. It cannot decide whether the destination is worth reaching. It can produce a persuasive explanation. It cannot guarantee that the belief being explained deserves belief.

This is why the arrival of more capable AI should make us more attentive to judgment, not less.

The fluency trap: when imitation looks like intelligence

Generative AI creates a special version of the old problem because it can imitate the surface of thought. It can produce a competent memo, a warm reply, a plausible lesson plan, or a strategic recommendation. Its fluency makes it easy to confuse successful performance with understanding.

Consider a game playing program that defeats novice players by using a simple strategy. It may appear intelligent because it produces winning moves, but its apparent intelligence can depend on exploiting predictable mistakes rather than understanding the game in a rich human sense. The distinction matters. A system can appear intelligent under a narrow test while lacking the judgment required in unfamiliar circumstances.

The same thing happens in organizations. A language model may generate thousands of customer responses that sound empathetic. But if the company has designed its support operation around reducing average handling time, the model may become a highly articulate mechanism for ending conversations quickly. It can make neglect sound considerate.

A hospital might use AI to summarize patient records. If the system is judged by speed and completeness, it may produce excellent summaries. But the most important clinical fact may be a patient's hesitation, a contradiction between their words and behavior, or a detail that does not fit the standard categories. An optimized summary can accidentally erase the very irregularity a physician needs to notice.

A school might use AI to produce individualized exercises. If the goal is to improve measurable performance, the system may successfully train students to answer familiar question types. Yet education may also require curiosity, intellectual courage, and the ability to formulate better questions. These qualities are difficult to capture because they involve changing the criteria of evaluation themselves.

The danger is not only that AI will make mistakes. Humans already make mistakes, and machines can sometimes reduce them. The deeper danger is that AI will make unexamined criteria scalable. Once a metric is embedded in a system, the organization can act on it continuously, cheaply, and at enormous volume.

A weak hiring rubric can now evaluate ten thousand candidates instead of one thousand. A narrow definition of productivity can govern every employee interaction. A simplistic risk model can influence decisions across an entire population. The machine does not need to be malicious to produce harmful results. It only needs to be competent at pursuing a goal that no one has revisited.

The more powerful the executor, the more dangerous it is to leave the objective unquestioned.

From task automation to judgment architecture

This suggests a useful framework for understanding AI adoption. Most organizations move through three levels, but they often mistake the first for the third.

1. Imitation: make the existing task faster

At the first level, AI imitates or accelerates a task already being performed. It drafts the email, summarizes the meeting, extracts information from a document, or answers a routine question. This is where immediate productivity gains are easiest to see.

The relevant question is: Can the same output be produced with less time, money, or effort?

These gains are real, but they are usually local. They improve an activity without changing the larger system around it.

2. Optimization: improve performance against a chosen metric

At the second level, AI coordinates many tasks around a target. It predicts demand, allocates staff, ranks leads, recommends prices, or identifies cases for review. The system is no longer just assisting a worker. It is shaping the flow of work.

The relevant question becomes: Can the organization perform better according to its existing measure of success?

This level can produce substantial gains, but it also increases the risk of metric capture. What is easy to count begins to define what matters. Response time replaces resolution. Engagement replaces learning. Throughput replaces care. The organization becomes more efficient at satisfying the proxy while drifting away from the purpose behind it.

3. Recomposition: redefine the work and the goal

At the third level, AI prompts an organization to reconsider the activity itself. Instead of asking how to process more applications, a city asks which regulations actually protect public interests. Instead of asking how to handle more support tickets, a company asks why customers encounter the same failure repeatedly. Instead of generating more reports, a leadership team asks which decisions truly require reporting.

The relevant question is: What should this system be trying to accomplish, and what is the best human and machine division of responsibility?

This is where the greatest gains may lie, but it is also where technology alone cannot lead. Recomposition requires judgment, institutional imagination, and the willingness to abandon familiar procedures. It is a creative act because it involves selecting new ends, not merely finding better means.

The three levels can be represented as a simple equation:

Automation improves execution. Optimization improves performance. Recomposition improves the definition of success.

Only the third reliably produces progress rather than acceleration.

What human judgment should do when machines can do more

If AI is a machine of the mind, the human role cannot be reduced to checking whether the machine followed instructions. Humans must remain involved earlier in the chain, where goals and criteria are formed.

This does not mean treating human intuition as automatically superior. Human judgment is biased, inconsistent, and often poorly calibrated. The point is not that people should make every decision unaided. The point is that decisions about values cannot be outsourced merely because they have been expressed in numbers.

A practical division of labor begins with distinguishing three kinds of questions.

Questions of fact ask what is happening. How many customers are waiting? Which treatment has the strongest evidence? What patterns appear in the data? Machines can be extremely useful here, especially when the relevant information is large and complex.

Questions of prediction ask what is likely to happen. Which supply route will be delayed? Which patients may need follow up? Which design is likely to fail? Machines can help with these questions, provided uncertainty is visible and the data is not mistaken for reality itself.

Questions of value ask what should happen. What level of privacy is acceptable? Which risks should a society tolerate? What does a fair opportunity look like? What kind of education is worth providing? These questions require public reasoning and judgment. They cannot be settled by accuracy alone.

The practical mistake is to let a strong answer to the first two questions silently decide the third. If a model predicts that a person is unlikely to repay a loan, that does not by itself determine whether the person should be denied credit. If an algorithm predicts that a student will perform poorly, that does not determine how much support the student deserves. Prediction can inform a decision, but it cannot define the moral meaning of the decision.

Organizations should therefore treat AI deployment as a form of judgment architecture. Before introducing a system, they should make explicit:

1. What human purpose is this process meant to serve?
2. Which outcomes are direct goals, and which are only convenient proxies?
3. What valuable information might disappear when the task is standardized?
4. Where should disagreement or ambiguity trigger human review?
5. Who has the authority to change the objective when circumstances change?

These questions are not bureaucratic obstacles to innovation. They are the conditions that make innovation worthwhile.

Key Takeaways

• Audit the goal before automating the task. Write down what the process is ultimately for, then identify whether the proposed AI system is optimizing that purpose or merely an easy to measure substitute.

• Separate facts, predictions, and values. Use AI aggressively for finding patterns and estimating outcomes, but do not let its confidence settle questions about fairness, dignity, acceptable risk, or social priorities.

• Look for the missing variable. Whenever a system becomes more efficient, ask what it may no longer notice: context, exceptions, long term effects, emotional cues, or the opportunity to question the premise.

• Design escalation points. Human involvement should occur where the case is unusual, the stakes are high, the evidence conflicts, or the decision could change a person's opportunities in a lasting way.

• Measure progress at the level of purpose. Do not stop at time saved or volume produced. Track whether the people affected are better served, whether recurring problems decline, and whether the organization is learning to choose better objectives.

The productivity boom associated with earlier computing came from large investments that transformed how businesses operated. The next wave may be even broader because the new machines can enter activities once thought inseparable from human cognition. But broader reach does not guarantee deeper progress.

AI can give institutions a second chance to rethink their work. It can also give them a way to avoid rethinking it forever. A company that uses intelligent systems only to process more tickets has not necessarily become more intelligent. A government that makes an outdated rule instantaneous has not necessarily become more just. A school that produces personalized instruction has not necessarily produced a better education.

The central competition will not be between humans and machines. It will be between organizations that use machines to preserve their assumptions and organizations that use machines to examine them.

The most valuable human contribution in an AI economy may therefore be neither speed nor information. It may be the courage to ask whether the current objective deserves to survive. Machines can calculate the best way to reach a destination. Progress begins when we become capable of choosing a better destination.