Search Indexes Are the DNS Layer for AI Agents

I tried to build a handshake protocol for AI agents. Put a token in /.well-known/agent.json, told the agent to fetch it, extract the token, and include it in the next request. Basic stuff. Four HTTP calls.

It didn't work.

Not because the protocol was wrong, but because ChatGPT straight up told me it can't do that. Here's what it said:

"I do not behave like a browser or crawler. I behave like a retrieval-mediated agent. Meaning: search index → allowed fetch → parse. NOT: navigate → execute JS → explore."

And then this:

"Search indexes are currently acting as the de facto DNS layer for agents."

I've been thinking about this for days.

What's actually happening

When you ask ChatGPT something that needs web content, you probably imagine:

agent → website

What's actually happening:

agent
  ↓
search index (Bing, usually)
  ↓
fetch layer (grabs URLs from search results)
  ↓
your content (maybe)

The agent doesn't decide to visit your site. The search index decides whether your site gets visited. The agent just sees whatever the fetch layer hands it.

This has some weird implications.

You can't handshake with something that can't navigate

My handshake protocol required the agent to:

Fetch URL A
Extract data from the response
Use that data to construct URL B
Fetch URL B

Agents can't do step 3. Each fetch is independent. There's no state between requests. The agent can't take something it learned from one HTTP response and use it to construct the next request.

This isn't a bug. It's the architecture. The fetch layer is stateless by design.

All that llms.txt stuff doesn't matter yet

I've been building llms.txt files, agent.json manifests, structured context endpoints. Everyone's doing this. It feels like the right thing to do.

But here's the problem: agents don't look for these files. They don't navigate to /.well-known/ anything. They query a search index and fetch whatever comes back.

If your llms.txt isn't in the search results for the query the agent is running, the agent will never see it.

All the work on agent-readable formats is Gate 2 work (can the agent understand you?) when we haven't solved Gate 1 (can the agent reach you?). Gate 1 is currently outsourced entirely to search indexes.

What's missing

The agent told me something that stuck:

"The real protocol layer for the agent web may end up looking far closer to BGP + TLS than to sitemap + robots.txt."

We're missing the infrastructure. Specifically:

Agent DNS: How does an agent resolve "I need the canonical endpoint for Stripe's payment API" into an actual URL? Right now: search for it. That's not a protocol, that's a hack.
Agent TLS: How does an agent know the endpoint it's talking to is actually Stripe and not someone pretending to be Stripe? Right now: it doesn't. It trusts whatever the search index returned.
Capability negotiation: How does an agent discover what an endpoint can do before calling it? Right now: hope there's an OpenAPI spec somewhere, or just try stuff.

MCP is a piece of the puzzle but it's more like HTTP than DNS. It tells you how to talk to an endpoint, not how to find and verify endpoints in the first place.

Where this probably goes

Prediction: within a few years, serious agent-facing endpoints will need to publish something like a signed manifest. Identity, capabilities, trust chain, maybe pricing. Before any agent action happens.

Like how TLS went from optional to required. First it was "nice to have HTTPS." Then browsers started warning on HTTP. Then HTTP basically stopped working for anything real.

Same thing will happen with agent endpoints. Right now, agents will call anything the search index gives them. That won't last. Too much attack surface. Too easy to impersonate.

What to do now

Short term: SEO matters more than you think for AI visibility. If you're not in the search index, agents can't find you. All the agent-specific optimization in the world doesn't help if you fail Gate 1.

Medium term: Build the agent-readable infrastructure anyway. llms.txt, structured context, MCP endpoints. When agents do get direct navigation (and they will), sites with this stuff ready will have a huge advantage.

Long term: Watch for whoever builds the resolution and trust layer. That's the Cloudflare/Let's Encrypt play for the agent web. That's where the real leverage will be.

I'm going to keep poking at this. Keep trying handshake protocols that don't work. Keep asking agents about their own limitations. There's something here.