The Codebase Is Not Context: It Is the Missing Specification

What if an AI can solve a problem perfectly and still be fundamentally wrong?

That is the uncomfortable lesson hiding inside two seemingly unrelated facts. A language model can be trained to multiply enormous numbers, yet fail at much smaller examples. And an AI tool can be asked to understand a software repository, yet produce code that looks plausible while violating the system’s unwritten rules.

In both cases, the obstacle is not necessarily intelligence, compute, or optimization. The deeper problem is underspecification: the visible task permits many solutions, but only a few are useful in the real world.

A request such as “multiply these numbers” does not say which internal procedure should be learned. A request such as “add authentication to this application” does not say how the repository represents identity, where authorization belongs, which errors are safe to expose, or what historical compromise the current design is protecting.

The central challenge for useful AI is therefore not merely making models more capable. It is helping them discover which of the many possible answers is the one that belongs here.

The difference between solving a task and learning the right task

Suppose a model sees thousands of examples of arithmetic. It receives input strings and must produce output strings. The training objective rewards correct answers on the examples it has seen. But those examples can be explained by countless underlying rules.

A system might learn genuine multiplication. It might memorize common products. It might notice superficial patterns in the formatting of numbers. It might construct a shortcut that works on the training distribution and collapses when the number of digits changes.

All of these strategies can receive similar feedback during training. The objective says, “produce the expected output.” It does not necessarily say, “discover the compositional algorithm that remains valid outside the examples.”

This is why a dramatic improvement in generalization does not have to come from a better optimizer. A carefully designed training scheme can make the intended solution easier to identify. An intermediate reasoning trace, a particular architecture, or a useful constraint can eliminate many accidental solutions from consideration.

The important distinction is between optimization and specification. Optimization asks whether the system can find a low error solution. Specification asks whether the available evidence makes the desired solution distinguishable from all the other low error solutions.

A model cannot infer what the data never meaningfully distinguishes.

Software engineering has the same structure, but with a larger and more treacherous space of possible answers. A coding request is rarely just a function from input to output. It is a request to modify an existing organism without damaging its metabolism.

Consider the instruction: “Add a retry mechanism to the payment service.” A locally reasonable implementation might retry every failure three times. But perhaps the service already retries at a lower network layer. Perhaps payment creation is not idempotent. Perhaps the repository treats timeouts as uncertain outcomes and sends them to a reconciliation queue. Perhaps customer support relies on a specific error code to distinguish “payment declined” from “payment status unknown.”

The code that compiles is only one candidate among many. The code that passes a narrow test suite is still only one candidate. The useful code is the candidate that respects the repository’s explicit contracts, implicit conventions, operational assumptions, and historical boundaries.

The problem is not that the model lacks access to enough text. The problem is that the repository contains a distributed specification, and much of that specification is encoded indirectly.

A codebase is a society of constraints

A mature codebase is often described as a collection of files. That description is technically accurate and practically misleading. A living system is better understood as a society of constraints.

Its rules are distributed across several layers:

1. Structural constraints: module boundaries, interfaces, dependency direction, naming conventions, and directory organization.
2. Behavioral constraints: tests, validation logic, error handling, retries, permissions, and observable outputs.
3. Historical constraints: compatibility decisions, migrations, deprecated paths, and old bugs that became contractual behavior.
4. Operational constraints: deployment environments, monitoring, performance budgets, incident procedures, and data retention rules.
5. Cultural constraints: patterns that experienced engineers follow because they understand why the obvious alternative causes trouble.

A model that reads only the target file sees syntax. A model that reads the repository begins to see architecture. A model that reads the repository together with its tests, history, documentation, and surrounding call sites begins to see intent.

This distinction matters because the best implementation is often not the most elegant implementation in isolation. It is the one that minimizes violations across the entire network of constraints.

Imagine entering a city and being asked to build a bridge. A map helps, but it does not tell you which neighborhoods flood, which roads are politically sensitive, where utility tunnels run, or why an apparently inefficient intersection has remained unchanged for decades. The city’s true design is not contained in one document. It is latent in the relationships among structures, behavior, history, and use.

A codebase is similar. Its “expertise” is not a single file that can be retrieved. It is an organized pattern of evidence.

This gives a more precise meaning to the idea of creating a living expert on a codebase. The goal is not simply to attach a chatbot to a repository. It is to make the repository’s distributed constraints legible enough that an AI can use them when making decisions.

That requires more than context window size. More text can increase confusion if the system cannot distinguish governing evidence from incidental detail. A useful codebase expert must rank evidence, connect it to the requested change, and identify which rules are stable, local, or exceptional.

From autocomplete to institutional memory

Most coding assistants are optimized around a narrow interaction: predict the next token, generate a function, or answer a question about a selected region. These capabilities are valuable, but they frame software work as text completion.

Real maintenance work is closer to institutional memory.

An experienced engineer asked to modify an unfamiliar service does not begin by writing code. They ask questions. Where is the source of truth? Which component owns this decision? Is this pattern repeated elsewhere? What tests express the intended behavior? Was this strange branch introduced to address a production incident? What would break if this interface changed?

These questions are not ornamental. They are mechanisms for reducing underspecification.

We can model a coding task as having two parts:

Visible request plus hidden repository specification equals acceptable change.

The visible request may contain only a few sentences. The hidden specification may include thousands of relevant signals spread throughout the repository. The more consequential the change, the more dangerous it is to treat the visible request as complete.

This suggests a useful design principle for AI coding systems: before generating an implementation, generate a constraint map.

A constraint map does not need to be a long report. It might identify:

• the modules that own the behavior;
• the interfaces that must remain stable;
• similar implementations elsewhere;
• tests that define expected behavior;
• known edge cases and failure modes;
• migrations or compatibility requirements;
• assumptions that remain uncertain.

The point is not to make the model explain everything it read. The point is to make it distinguish “I found a pattern” from “I found the rule that governs this change.”

This also changes how we should evaluate AI generated code. A useful benchmark should not ask only whether the output compiles or passes visible tests. It should test whether the change generalizes across the repository’s local conventions and hidden constraints.

For example, an assistant might be evaluated on whether it:

• places new logic in the component that owns the relevant invariant;
• preserves error semantics used by downstream callers;
• follows existing dependency boundaries;
• updates tests at the right abstraction level;
• avoids duplicating a utility that already exists;
• preserves behavior that is not obvious from the feature request.

These are not merely style preferences. They are evidence that the system has identified the intended solution class rather than found a locally successful shortcut.

The repository as an inductive bias

In machine learning, an inductive bias is a preference that helps a system generalize. It narrows the space of possible explanations so that the model is more likely to choose a rule that works beyond the observed examples.

Software repositories can play the same role.

Repeated architectural patterns bias an assistant toward certain designs. Tests bias it toward particular behaviors. Type definitions bias it toward safe interfaces. Commit history can bias it away from changes that previously caused regressions. Build rules, linters, and deployment checks turn vague expectations into enforceable constraints.

A codebase is therefore not just something an AI should retrieve information from. It is part of the learning environment that shapes what counts as a good answer.

This yields a powerful mental model: context is passive information, while specification is active selection pressure.

Pasting twenty files into a prompt supplies context. Showing the relevant tests, naming the owning module, exposing similar changes, and requiring compatibility checks supplies selection pressure. The first may help the model talk about the system. The second helps it act within the system.

The distinction explains why larger context does not automatically produce better coding. If all repository material is treated as equally authoritative, the model faces a new form of underspecification. A stale comment, a one off workaround, and a widely repeated invariant may appear side by side as equivalent evidence.

A living codebase expert should help establish an evidence hierarchy:

1. Executable behavior and invariant tests usually outrank informal descriptions.
2. Repeated patterns usually outrank isolated examples.
3. Current owners and call paths usually outrank historical artifacts.
4. Recent architectural decisions may explain exceptions to older conventions.
5. Uncertainty should be surfaced rather than silently resolved through invention.

That last point is critical. When the repository does not determine a unique answer, the expert should not disguise ambiguity as confidence. It should ask a clarifying question, propose alternatives, or mark an assumption for review.

In this sense, uncertainty is not a failure of the system. It is a correct diagnosis of an underspecified task.

How to make an AI a better inhabitant of a codebase

The practical lesson is not that every engineer must document every line before using AI. That standard is impossible. The lesson is to invest in the forms of evidence that most effectively reduce ambiguity.

Start by identifying the repository’s high value invariants. These are statements such as “all writes pass through this service,” “this identifier is never exposed externally,” or “a timeout does not mean a failed transaction.” Invariants are more valuable than broad descriptions because they constrain implementation choices.

Next, connect tests to intent. A test named handles timeout is useful, but a test with a clear explanation of why timeout is treated as an unknown payment state is much more valuable. Tests should not only catch regressions. They should teach future maintainers, human or artificial, which behavior is deliberate.

Then make ownership visible. When a repository has several layers that appear capable of handling the same concern, an assistant can easily place logic in the wrong layer. Clear module boundaries, architectural decision records, and examples of canonical usage reduce this ambiguity.

Finally, design AI workflows around reviewable reasoning rather than raw generation. Ask the system to state which files and constraints it believes are relevant before it edits anything. Ask it to identify a similar existing pattern. Ask what could break if its proposed change is wrong. These prompts are valuable because they turn hidden assumptions into inspectable objects.

A compact workflow might look like this:

1. Map: identify owners, call paths, tests, and analogues.
2. Constrain: list invariants, compatibility requirements, and forbidden changes.
3. Propose: offer the smallest change that satisfies the constraints.
4. Challenge: search for counterexamples, edge cases, and conflicting conventions.
5. Verify: run tests and inspect the diff at the architectural level, not only the line level.

This workflow is useful even when no AI is involved. Its deeper purpose is to improve the specification of the work itself.

Key Takeaways

• Separate capability from specification. If an AI produces inconsistent results, ask first whether the task distinguishes the desired solution from plausible shortcuts.
• Treat the repository as a distributed specification. Read tests, interfaces, call sites, operational code, and history as connected evidence, not as unrelated files.
• Ask for a constraint map before implementation. Identify ownership, invariants, analogous patterns, compatibility requirements, and unresolved uncertainty.
• Turn conventions into selection pressure. Reusable examples, meaningful tests, architectural records, and automated checks help an assistant generalize toward the repository’s actual design.
• Reward honest uncertainty. A system that asks for clarification when the codebase does not determine a unique answer is more reliable than one that always produces confident code.

The future of AI assisted software development will not be decided only by who has the largest model or the longest context window. It will be decided by who can make implicit constraints visible and actionable.

A powerful model without a strong specification is like a brilliant engineer dropped into a city with no map, no local contacts, and no knowledge of its laws. It may build something impressive. It may even build something that works for a while. But it has no reason to know whether the result belongs there.

The real promise of a living expert on a codebase is not that it can remember more files. It is that it can help recover the system’s intent from the evidence scattered across those files.

That reframes the central question. We should stop asking only, “Can the AI write this code?” The more important question is: what would allow the AI to recognize the code that this system, with its history and constraints, is actually asking for?