agents can't read your website, and the fix is one Accept header

I watched an agent summarize a cookie banner this week. Confidently. The fetch returned a consent wall and four hundred divs, the actual article never rendered, and the model worked with what it had, which was nothing.

That is what most of the web looks like to an agent. No JS runtime, no clicking "accept all", no scrolling past your hero animation. If your content arrives client-side, a plain fetch gets a skeleton, and every token of that skeleton comes out of the reader's context budget before any reasoning starts.

The fix is web tech older than most frameworks: content negotiation, which has been in HTTP since the nineties and lives today in RFC 9110 section 12. When the request says it prefers markdown, send markdown. Run this and watch it work:

curl -H "Accept: text/markdown" https://artnikitin.dev/

Clean markdown, whole page, zero soup. Same URL, one header, and the response comes back as text/markdown with an x-markdown-tokens: 661 header on top so the client knows what it is about to pay for. The HTML version of that same page is 334,333 bytes. The markdown is 2,652.

who else does this, with numbers

I assumed this was a hobbyist trick. It is not. I ran the same curl against the docs I hit most, fetched each page twice (once as a browser, once as markdown) and counted bytes on the wire, uncompressed, on 2026-08-26.

• Stripe: docs.stripe.com/payments is 455,209 bytes of HTML and 5,339 bytes of markdown. Stripe documents the .md suffix (payments.md works too), and every link inside the markdown already points at the .md version of the next page, so an agent never falls back into HTML by accident.
• Cloudflare: developers.cloudflare.com/workers/ goes from 210,085 bytes to 6,915. The markdown opens with a blockquote pointing at a per-section llms.txt, and the response carries vary: accept. Cloudflare also sells this as a zone feature, in beta since 2026-02-12: flip a toggle and the edge converts your HTML for any request that asks. Their own blog post went from 16,180 tokens to 3,150 that way.
• Vercel: vercel.com/docs drops from 920,105 bytes to 5,951. Their docs started answering the header on 2026-01-13, and the follow-up engineering post measured a 500 KB page down to 3 KB. It also recommends a <link rel="alternate" type="text/markdown"> tag in the HTML head for clients that never send the header.
• Claude's own docs: platform.claude.com/docs/en/intro is 933,168 bytes of HTML. Nearly a megabyte, for a page whose markdown is 5,251 bytes. That is a 178x difference, and the second number is the one the model actually needs.
• Mintlify, which hosts a lot of the docs you read, does it for every site on the platform: .md suffix, or Accept: text/markdown, or Accept: text/plain. Their own docs page: 375,514 bytes down to 3,258.

Then the fun ones.

react.dev answers Accept: text/markdown with text/plain and hands you the raw MDX source, <Intro> tags and all. Close enough. github.com answers with a 406 Not Acceptable and an empty body, which is at least honest. MDN ignores the header and sends HTML like it is 2019. nextjs.org/docs also ignores the header, but ships a 47,748-byte llms.txt, so the index exists even though the pages do not negotiate.

what the afternoon actually looks like

Three pieces, in the order I would build them.

The route. Read the Accept header, and if text/markdown is in it, render the page's source markdown instead of the HTML template. If you write in markdown already (most docs and most blogs do), this is a lookup, not a conversion. If you don't, run the rendered HTML through a converter once at build time and cache it. Cloudflare's toggle is the zero-effort version of this for sites on their network.

The Vary header. This is the one that will ruin your week if you skip it. Send Vary: Accept on any URL that can answer two ways, or your CDN caches the markdown and serves it to the next human, who gets a wall of text instead of your site. Stripe, Vercel and Cloudflare all send it. Two of the sites I tested send no Vary at all and still returned the right body when I alternated headers, which means their cache key includes Accept somewhere I can't see. Yours might not.

The index. Add llms.txt, a proposed markdown file at /llms.txt that tells a model what your site holds and where, with llms-full.txt as the everything-in-one-file variant. Stripe's is 90,052 bytes and opens with instructions to the model ("never hardcode an old version number from training data") before the first link. Claude's docs ship a 63,970-byte one. Svelte's is 1,676 bytes. Size is not the point; the point is that an agent's first request can be the index instead of your homepage.

The smallest site on my list, that portfolio from the curl above, does all three and adds a /md/ path prefix as a second door for clients that can't set headers, plus an llms.txt with a "Machine access" section that says so. The response headers give away the hosting: CloudFront in front, Lambda behind, the same server-rendered Next.js app described in the writeup of how that portfolio runs Next.js on Lambda. The markdown route sits beside the HTML route in one app. No second service, no scraper, no pipeline.

what to do on the client side

Your fetch tool should send Accept: text/markdown, text/html;q=0.8 on every request. Sites that negotiate will hand you the good version; sites that don't will behave exactly as before, because that is what content negotiation was designed for. Then check the Content-Type of what came back, and if it is HTML, strip it before the model sees it. The model should never be the thing that finds out the page was a cookie banner.

And log the bytes. The gap between 933,168 and 5,251 is not a rounding error, it is your bill.

We spent two years making models better at reading. Serving something readable is faster.