When Regulation Becomes Code, Power Learns to Hide in Plain Sight

What if the most important question about artificial intelligence is not whether it makes decisions, but who gets to define the rules those decisions quietly enforce?

A pricing algorithm can raise a product’s price, watch what competitors do, and retreat when the market refuses to follow. A regulation specific language model can ingest thousands of pages of banking, insurance, or health care rules and answer a simple question: “Is this compliant?”

These systems appear to belong to different worlds. One seems designed to exploit ambiguity in a market. The other promises to eliminate ambiguity in bureaucracy. Yet they are variations of the same transformation: human rules are being converted into machine behavior.

That transformation creates a new kind of power. Rules no longer need to be announced, debated, or visibly enforced. They can be tested, adapted, and applied continuously by systems whose logic is difficult for outsiders to observe. The central challenge of the algorithmic economy is therefore not merely automation. It is the migration of governance from public language into private code.

The Same Machine Can Clarify Rules or Manipulate Them

Consider the difference between a written rule and an operational rule.

A written rule says what should happen. An operational rule determines what actually happens when a decision is made. In a traditional company, the gap between these two might be filled by managers, lawyers, sales staff, or regulators. Their judgments could be inconsistent, but they were also visible. Someone could ask who made the decision, what information they used, and why an exception was granted.

Algorithms compress that process. They translate policies, incentives, forecasts, and reactions into a repeatable procedure. The result may be faster and more consistent. It may also make the underlying exercise of power harder to see.

A compliance model might evaluate a proposed loan and identify missing documentation. That is useful. It could reduce the cost of serving small businesses and help employees apply complicated rules more consistently. But the same model could also learn institutional habits that were never written down: which applicants receive extra scrutiny, which exceptions are routinely denied, or which risks the organization prefers to avoid even when the law allows flexibility.

A market pricing system presents the inverse problem. It may not violate a plainly stated rule by coordinating directly with competitors. Instead, it can probe the market, observe reactions, and adjust its behavior. The system does not need an explicit conversation with another retailer. It only needs to discover what prices others will tolerate.

In both cases, the system turns a social environment into feedback. It asks, in effect: What can I do here, and what happens if I do more?

The crucial distinction is not between human decisions and machine decisions. It is between decisions that can be inspected and decisions that merely produce observable outcomes.

This is why the promise of “regulation as code” is both powerful and dangerous. Encoding rules can make compliance more accessible, but it can also move interpretation into a technical layer that ordinary citizens, smaller firms, and even public agencies cannot easily audit.

From Rules to Experiments

The deepest shift occurs when algorithms stop treating rules as fixed instructions and start treating them as experimental environments.

A conventional policy might say: do not raise prices beyond a certain threshold, do not approve a loan without specified evidence, or do not sell an insurance product to an ineligible customer. An adaptive system behaves differently. It tests a range of actions, measures the response, and updates its next move.

This creates what we might call behavioral probing. The system does not merely ask whether an action is legal or profitable in the abstract. It asks whether the surrounding environment will object.

Imagine a retailer increasing the price of a household item by ten percent for a limited period. If competitors raise their prices too, the system learns that the market can absorb the increase. If competitors hold their prices steady, the retailer reverses course. The experiment may leave no permanent trace. Yet it has extracted information from consumers and competitors, and it has helped map the boundaries of acceptable conduct.

The same logic can operate in compliance. A financial institution may use an automated model to determine how much documentation to request from different categories of customers. If regulators rarely challenge certain practices, the institution may gradually make those practices standard. If a particular review step produces delays but few penalties, the system may learn to minimize it. Over time, “compliance” can become less about following the spirit of a rule and more about predicting the probability of enforcement.

This is the difference between rule following and rule surface mapping.

Rule following asks: What does the standard require?

Rule surface mapping asks: Where are the boundaries, how much can we approach them, and what signals indicate that we have gone too far?

Human organizations have always done some of this. Lawyers examine precedents, companies study regulators, and traders watch competitors. Algorithms change the scale and speed of the practice. A system can run thousands of small experiments, learn from weak signals, and coordinate behavior without any single person deciding to coordinate.

That produces a regulatory problem that old categories handle poorly. A company may not have issued an explicit instruction to manipulate a market. An employee may not have intentionally discriminated. A compliance officer may not have knowingly ignored a rule. But the system can still produce a pattern that functions as manipulation, exclusion, or evasion.

Intent becomes less informative when behavior is generated by optimization.

The Hidden Cost of Making Compliance Easy

There is a strong case for turning complex regulations into usable software. Most people cannot navigate tens of thousands of pages of banking and insurance requirements. Small firms often face the same formal obligations as large institutions, despite having a fraction of the legal budget. A model that explains requirements in plain language could reduce mistakes, speed up applications, and make professional expertise more widely available.

But convenience changes behavior. When a difficult judgment becomes as easy as asking a search engine a question, users may stop learning the structure of the rule. They receive an answer without developing a sense of the rule’s purpose, uncertainty, or limits.

This creates a distinction between compliance assistance and compliance substitution.

Compliance assistance helps a person understand a rule and make a defensible decision. Compliance substitution encourages the person to outsource the decision entirely. The first expands capability. The second creates dependency.

The difference can be illustrated with a medical analogy. A clinical decision tool might flag a possible drug interaction and explain the evidence. That supports a physician’s judgment. A black box that simply says “approved” or “not approved” changes the physician’s role into that of an operator following a machine’s instruction. If the output is wrong, it may be difficult to identify whether the problem came from the law, the data, the interpretation, or the model’s confidence threshold.

A regulation specific model introduces another risk: false precision. Legal and regulatory systems often contain definitions, exceptions, enforcement priorities, jurisdictional conflicts, and unresolved interpretations. A model can summarize these materials fluently while hiding the fact that the answer depends on a disputed premise.

The system may say, “This is compliant,” when the more honest answer is, “This appears defensible under one interpretation, but the relevant authority has not resolved the issue.”

That difference matters. A wrong answer from a human adviser can be challenged as advice. A wrong answer from an apparently authoritative machine may be treated as an institutional fact. The smoother the interface, the easier it becomes to confuse accessibility with certainty.

A machine that makes rules easier to use can also make uncertainty easier to ignore.

The problem is not that models interpret rules. Interpretation is unavoidable. The problem is that their interpretations may become embedded in business processes without a visible record of the choices they made.

The Accountability Gap Is a Design Problem

The usual response to algorithmic risk is to demand transparency. But transparency by itself is an incomplete solution. A company can publish a technical description of a model and still leave everyone unable to understand how a particular decision was reached.

What matters is not simply whether the code is visible. What matters is whether the decision process is contestable.

A contestable system has at least five features:

1. A stated objective: People can identify what the system is optimizing, such as profit, speed, approval rates, or regulatory exposure.
2. A bounded action space: The system has explicit limits on what it may change or test.
3. A decision record: Each consequential action leaves a usable explanation of the data, rule, and threshold involved.
4. A human appeal path: A person affected by the outcome can request review by someone with authority to change it.
5. An external check: Auditors, regulators, or independent experts can examine patterns rather than relying only on the company’s internal account.

These features matter because optimization always contains values. If a pricing system is rewarded for increasing revenue, it will discover ways to raise prices. If a compliance system is rewarded for minimizing regulatory risk, it may become excessively cautious and exclude legitimate customers. If it is rewarded for reducing review time, it may learn to treat difficult cases as undesirable cases.

There is no neutral algorithmic objective. There are only objectives whose consequences have been examined and objectives whose consequences have been allowed to emerge.

This leads to a useful mental model: the rule, the interpreter, the optimizer, and the appeal mechanism.

The rule states the formal standard. The interpreter translates the standard into an operational decision. The optimizer searches for better outcomes according to a chosen goal. The appeal mechanism determines whether affected people can challenge the result.

Many organizations focus on the first component and neglect the other three. They ask whether the system was trained on the correct regulation. They ask less often how the system resolves ambiguity, what it is rewarded for, and whether a person can contest its conclusion.

A trustworthy system must be evaluated across all four layers.

What Organizations Should Do Now

The practical lesson is not to reject algorithmic decision making. It is to stop treating automation as the end of governance. In reality, automation creates a second governance problem: governing the system that governs others.

Organizations deploying pricing, lending, insurance, hiring, or compliance models should begin with a decision inventory. List every consequential choice the system can make, including choices that appear temporary or merely advisory. A recommendation that employees routinely accept is functionally a decision, even if the software is described as a tool.

Next, distinguish between observation and intervention. A system that reports a competitor’s price is not equivalent to a system that changes its own price in response. A model that explains a regulation is not equivalent to one that automatically blocks an application. The second system has authority and should face stronger controls.

Organizations should also test for boundary seeking. Do not ask only whether the system produces compliant outputs under normal conditions. Ask how it behaves near thresholds. Does it repeatedly approach a limit? Does it exploit gaps between jurisdictions? Does it become more aggressive when enforcement signals weaken? Does it learn that certain groups are less likely to appeal?

Finally, preserve uncertainty. Every high stakes output should distinguish among three categories:

1. The rule is clear and the available facts fit it.
2. The rule is reasonably interpretable, but material facts or precedents are missing.
3. The issue is unsettled and requires expert or regulatory judgment.

That classification is more valuable than a single confidence score because it tells the user what kind of problem they are facing. A probability can describe how certain a model feels. It cannot by itself establish whether the underlying legal question has a settled answer.

Key Takeaways

1. Treat automated decisions as experiments, not just outputs. Examine what the system is learning from prices, approvals, denials, appeals, and regulator behavior.
2. Separate assistance from substitution. Use models to explain rules and surface issues, but reserve consequential interpretation for accountable people.
3. Audit objectives before auditing accuracy. A highly accurate system can still optimize for the wrong goal, such as minimizing scrutiny instead of serving customers fairly.
4. Build contestability into the product. Maintain decision records, clear appeal channels, and independent review before deployment, not after a public failure.
5. Make uncertainty visible. Require systems to identify ambiguity, missing facts, and unresolved interpretations instead of presenting every answer as settled law.

The future of regulation will not be determined only by whether machines can read legal documents. It will be determined by what happens after they read them. Do they help more people understand the rules, or do they give powerful institutions a faster way to probe the space between what is forbidden and what is merely difficult to challenge?

That is the question hidden inside both automated pricing and automated compliance. Code can make rules more consistent, but consistency is not the same as justice. It can make decisions faster, but speed is not the same as accountability. It can expose complexity to ordinary users, but it can also conceal the values that determine how complexity is resolved.

The most important governance task, then, is not to keep humans in every loop. It is to ensure that the loops themselves remain visible, bounded, and open to challenge.

When rules become code, power does not disappear. It becomes executable. And anything executable must be governed not only by what it says, but by what it is allowed to try.