The Agent-Ready Web: What Builders Should Do

Three Eras in Three Years

For most of the web's life, crawling was governed by an honor system. robots.txt is a request, not a fence. It worked because the deal underneath it was fair: search engines copied your page and sent you readers in exchange.

Generative AI broke the exchange rate, and the response arrived in three distinct waves.

Era one, roughly 2022 to mid-2024: the free-for-all. Training corpora got assembled from whatever was reachable. Most site owners had no idea which bots were visiting, let alone why. Tooling to find out barely existed outside server logs.

Era two, mid-2024 through 2025: blocking became one click. In September 2024, Cloudflare shipped a single toggle to block AI crawlers, free plans included. By July 2025 the company said more than a million customers had turned it on. Then on July 1, 2025, it went further: new domains were asked during onboarding whether to allow AI crawlers, with block as the starting position. The announcement carried 48 endorsers, an unusually wide coalition that included the Associated Press, The Atlantic, Condé Nast, Gannett, TIME, Reddit, Pinterest, Quora, and Stack Overflow.

That same day, an experiment called pay per crawl appeared. The mechanism was elegant: a site sets a price, unpaid crawlers receive HTTP 402 Payment Required, and paying ones retry with credentials attached.

In September 2025 came the Content Signals Policy, an extension to robots.txt that splits permission into three separate questions instead of one. Should this page be indexed for search? Used as input to a generated answer? Used for training? It was released under CC0 and applied automatically to about 3.8 million domains on managed robots.txt files.

Era three, 2026: identity and money. Blocking works, but blocking alone is blunt. If you want to allow some agents and not others, you first need to know which agent is actually knocking. Two things followed. Web Bot Auth applies HTTP Message Signatures (RFC 9421) with Ed25519 keys so a bot can cryptographically prove who it is, replacing IP allowlists and easily-spoofed user-agent strings. And x402, an open payment scheme built on that long-dormant 402 status code, moved under a Linux Foundation body in 2026 with members including AWS, Cloudflare, Anthropic, and Circle.

Then, on July 1, 2026, the controls were rebuilt around the three-way split from Content Signals. Search, Agent, and Training became independent switches available on every plan tier, free included. And a date was set: from September 15, 2026, domains newly onboarding get Training and Agent blocked by default on pages that display ads, while Search stays allowed. Existing customers keep whatever they've configured and have to change it themselves. The stated logic is worth repeating because it's the clearest sentence anyone has written about this: an ad is a signal that the site owner meant for a person to land there.

There's a second edge to that date. Crawlers that serve more than one purpose, Googlebot and Bingbot and Applebot among them, get judged by all of their behaviors, so blocking Training blocks them entirely. That coupling isn't aimed at publishers. It's aimed at the one company whose crawler nobody can afford to refuse.

Three years. Open by default to closed by default, decided largely by one company's onboarding screen.

---

Where We Actually Are: The Crawl-to-Refer Gap

The single most useful number in this whole debate is a ratio: how many pages does an AI company fetch from you, for every visitor it sends back?

Cloudflare began publishing these in July 2025, and the spread is startling.

Read the Google row and then read the Anthropic row again. Traditional search took roughly two to five pages to send one visitor. Answer engines take thousands.

A caveat the publisher states about its own metric, and which almost every write-up drops: traffic from native mobile and desktop apps carries no referrer header, so referrals are undercounted, and the companies with the most app usage look worse than they are. The direction is not in doubt. The exact multiple is.

Two more numbers frame the picture. Across 2025, AI bots accounted for about 4.2% of HTML requests, which is roughly what Googlebot alone accounted for. And by mid-2026, more than half of all traffic on the internet was non-human.

Meanwhile the human side kept shrinking. When Google shows an AI summary, the share of searches producing a click to a website falls sharply. Cloudflare's July 2026 report put it starkly: for every hour spent online looking for information, only about fifteen minutes is spent on the open web.

If you've been wondering why your analytics look worse while your content performs fine, this is the shape of the answer. Our guide to AEO and GEO covers what to do about the measurement side.

---

The Four Gates Every Agent Request Clears

Here's the framing that makes the last two years of announcements click into place.

When a human loads a page, the infrastructure question is simple: serve it or don't. When an agent makes a request, four questions have to be answered, usually in the same round trip.

1. Compute. The agent runs somewhere. Agent sessions are unusual workloads: one user, one agent, one task, alive for a long time but idle most of it while waiting on a model. That's a very different shape from a web server handling thousands of concurrent visitors, and it has pushed infrastructure vendors toward lightweight isolates that hibernate when idle rather than containers that bill by the clock.
2. Identity. Which agent is this, and who authorized it? A user-agent string is a claim, not a credential. This is what Web Bot Auth exists to fix, and why the useful distinction is no longer bot versus human. It's which bot, sent by whom, allowed to do what.
3. Access. What may it reach? Reading a public article, calling an internal API, and acting on someone's logged-in account are three different permissions that most sites currently treat as one.
4. Payment. What, if anything, does it owe the source? This is the newest gate and by far the least settled.

Notice that these were four separate product categories two years ago: hosting, bot management, access control, and payments. The bet across the industry is that for agent traffic they collapse into one decision. That's the actual strategy behind most of what you've been reading about. It also explains why infrastructure companies suddenly care about publisher economics, a topic they avoided for a decade.

---

Who Holds the Gates

Now the uncomfortable part, and it deserves saying plainly even if you like the direction this is going.

A single company sits in front of roughly a fifth of the web. That company wrote the bot identity specification, operates the key registry it depends on, decides which bots are verified, decides unilaterally when to remove one, sets the crawler defaults for customers on free plans, acquired a content licensing marketplace in January 2026, acts as merchant of record for the payments, and has filed with a competition regulator asking that its largest competitor be forced to split its crawler by purpose.

Any one of those is reasonable. Together they describe a private party with a great deal of discretion over who gets to read the public web.

The de-verification power is the sharp end. In August 2025, Perplexity was removed from the verified bot list after a report alleged it was using undeclared crawlers with rotating identities to reach pages that had blocked it. Perplexity disputed the attribution and called the report a publicity stunt. Whatever the truth, the process is worth noticing: a report, a decision, an enforcement action across a large share of the web, and no documented appeal.

It's also worth knowing that the identity layer is less settled than the coverage suggests. As of August 2026, Web Bot Auth exists as individual drafts at the IETF. None has been adopted by a working group and none is an RFC. Claims that a standards body has finalized it circulate widely and are wrong. Treat it as a vendor implementation that two large platforms have co-signed, which is a real thing but not the same as a standard.

Courts are moving faster than standards bodies, incidentally, and in a direction that rewards fences over signs. In July 2026 a US court let a claim proceed holding that circumventing anti-bot measures is independently actionable under the DMCA's anti-circumvention provision, separate from copyright infringement. Earlier that year, another court held that a user's consent doesn't authorize an agent to access a platform that has revoked its access. And the largest copyright settlement in US history, approved in July 2026, priced pirated training material at roughly $3,100 per work.

The infrastructure argument and the legal argument are converging on the same conclusion from opposite ends: declare what you allow, and make the boundary real.

---

What Shipped vs What Is Still a Press Release

This is the section to bookmark. The narrative around the agentic web is running well ahead of what anyone can actually buy, and the gap is easy to miss because both get announced the same way.

The practical rule that falls out of this: adopt at general availability, prototype on beta only with an exit path, and never put a revenue line in a plan that depends on something in the bottom half of that table.

---

The Adoption Gap Is the Opportunity

In April 2026, Cloudflare scored the top 200,000 domains on how ready they are for agent traffic and published the results. They are remarkable.

Fewer than 15. Out of 200,000. Their own summary of how agent-ready the web is: "The short answer: not very."

Compare that to search engine optimization, where every competitor you have is doing the same forty things you are. Here, the basics are unclaimed. Most of them take an afternoon.

There's a reason markdown negotiation matters more than it sounds. When a crawler fetches a modern page, it pays for your navigation, your cookie banner, your analytics tags, and your framework's hydration payload before it reaches a sentence of your actual argument. Serving clean markdown to clients that ask for it cuts the tokens dramatically. Cheaper to read means more likely to be read completely, and more likely to be quoted accurately.

---

The Agent-Ready Checklist

Ordered by payoff per hour of work.

Step six deserves emphasis because it reverses the obvious instinct. In July 2026, Cloudflare moved off billing per crawl and gave the reason: more than half of crawl traffic from good bots goes to re-fetching pages that haven't changed. In their words, a single page "might be crawled once and then cited in thousands of answers, or crawled over and over and never used at all." If the company whose whole business is counting crawls says crawl counts don't measure value, believe them.

If you want the file-by-file mechanics for step one, we've written that up separately in llms.txt vs robots.txt vs ai.txt. For step four, the protocol layer is covered in the agentic web protocol wars.

---

The Trap: Readable Is Not the Same as Paid

Here's where a lot of otherwise sensible plans go wrong.

The story goes: block the crawlers, create scarcity, charge for access, get paid. The blocking part worked. The charging part, so far, has not.

Pay per crawl launched in July 2025 and remains in closed beta more than a year later. No dollar figure, transaction count, or number of paying crawlers has ever been published. The named participants are small companies. None of the largest AI labs pays.

The most instructive data point came from Stack Overflow, which turned pay per crawl on in early 2026. Here's Josh Zhang of Stack Overflow describing what happened: "when we turned on Pay Per Crawl and we started serving, I think as a 402, some of the traffic from those bots that used to just get a block of 403, they stopped sending traffic our way." They didn't pay. They left.

There's a widely quoted statistic that over a billion HTTP 402 responses go out per day. Read it carefully. A 402 is a price quote that nobody accepted. It measures refusal, not revenue.

Then in July 2026 the billing unit changed from the crawl to the use, meaning payment when your content shows up in a generated answer. That's a better idea. It's also an admission that the first idea didn't price. The company running it calls it an experiment, and its CEO has said publicly that the settlement rail needed for this at internet scale would have to handle far more than a million transactions per second, and that nobody has built it.

Meanwhile, every deal that has actually moved money in this space has been a direct bilateral license between one AI company and one large rightsholder, bypassing all of the intermediary infrastructure entirely.

So the honest summary for anyone building something: being easy for agents to read pays off today. Being paid by agents does not yet exist as a market. Do the first. Don't budget for the second.

The leverage was real, though, and that's the part worth appreciating. Making refusal one click created bargaining power that publishers previously did not have, and dozens of licensing deals followed. The infrastructure didn't become the market. It became the reason a market could be negotiated at all.

---

What This Means If You Read and Learn Online

Step back from the infrastructure for a second, because there's a consequence for anyone who uses the web to learn.

Two things are happening at once. The open web is getting harder to reach, gate by gate. And the interface to it is shifting from a list of links you evaluate yourself to a paragraph of synthesized text you mostly have to trust.

Both trends move value in the same direction: away from finding information, which is now nearly free, and toward judging it, keeping it, and being able to show where it came from. A generated answer can be produced infinitely. A record of what a specific person read, what they thought was worth keeping, and exactly which passage it came from cannot.

That's the durable layer, and it's the reason we build Glasp the way we do. When you use Glasp's web highlighter, the thing you save isn't a summary of a page. It's the sentence you chose, with its source attached and a date on it. When you run YouTube video summaries, the timestamps stay attached to the transcript so the claim stays checkable. Your Kindle highlights land in the same place. And because highlights are public by default, what one reader marks as worth keeping becomes a signal for everyone else, which is exactly the kind of judgment no model produces on its own.

There's a practical version of this too. If your notes are provenance-first, they survive the shift. You can hand them to an AI assistant as context without handing over your judgment, which is the difference between using a model and outsourcing your thinking to one. We've written more about that tension in the AI thinking trap and about why curation is becoming the scarce skill in the human curator in the age of AI.

The web is being rebuilt for machines. The part worth protecting is the part that records what humans decided mattered.

---

Frequently Asked Questions

What actually changes on September 15, 2026?

The old single "block AI bots" toggle gives way to three independent controls: search, agent, and training. For domains newly onboarding to Cloudflare, training and agent access is blocked by default on pages that display ads, while search stays allowed. Existing configurations aren't rewritten for you, which sounds reassuring but means the opposite of what people assume: if you never set the new controls, you've made a choice by not making one. Note also that crawlers serving several purposes at once get judged by all of them, so blocking training also blocks Googlebot.

Should I block AI crawlers on my site?

It depends on what you get from them, and the three-way split finally lets you answer that properly. If appearing inside AI answers brings you readers or customers, allow the AI input category, because that's the one that refers people back. Training is a separate question with a different answer for most people, since it produces neither citation nor traffic. Blocking everything is a real option, but understand you're trading away discovery, and for most sites in 2026 that's a bigger cost than the training exposure.

Does blocking crawlers actually make AI companies pay?

Not so far. Payment schemes have been in limited beta since mid-2025 with no published revenue, and the largest labs don't participate. What blocking has produced is negotiating leverage, which led to a number of direct licensing deals for large publishers. If you're not a large publisher, treat blocking as control over your content rather than as a revenue plan.

Is Web Bot Auth a real standard I should implement?

It's real and it's implemented by several large platforms, but it is not yet a standard. As of August 2026 it exists as individual Internet-Drafts at the IETF, with no working group adoption and no RFC. If you operate a crawler or agent, adopting it is sensible because it's how you get treated as a known actor. If you run a website, you don't implement it, you just benefit when the bots hitting you do.

What's the single highest-return thing to do this week?

Set your three crawler controls deliberately instead of by default, then make your key pages cheap to parse: real HTML headings, content present without JavaScript, dates and authorship visible. Everything else on the checklist matters, but those two take an afternoon and put you ahead of most of the top 200,000 domains.

How do I tell whether any of this is working?

Stop counting crawler hits and start counting two things: whether your content appears in answers when you ask the major assistants about your topic, and whether anyone arrives from those assistants. Crawl volume moves for reasons that have nothing to do with your value, which is why even the vendors who sell crawl analytics have moved off it as a success metric.

---

Closing Thought

The interesting shift isn't that AI companies are crawling the web. It's that the web quietly changed its answer to a question it had answered the same way since 1994.

For thirty years the default was yes, take it, and tell us if you'd rather we didn't. As of 2026 the default is increasingly no, declare yourself first. That happened in about a year, and it was decided less by legislation or standards bodies than by the settings screen a few million site owners clicked through without reading.

Whether that's a rescue or an enclosure depends on where you're standing. Either way, the practical response is the same: decide what you allow instead of inheriting it, make yourself easy to read for the agents you want, and keep your own record of what you learned and where it came from. That last one is the piece no infrastructure company can do for you, and it's the piece that keeps its value no matter which way the gates swing.

Start with the sentence you'd want to find again. Glasp is built for exactly that, and it's free to try.