Building Agent-Ready Surfaces for Information

Traditional SEO optimizes for a ranking algorithm that then hands a human a list of blue links. The human does the reading, the comparing, the deciding. Agent-readiness is a different problem: the thing consuming your page might have to parse it, extract a fact, and act on that fact with no human in the loop to notice if the extraction went wrong. That changes what "well-structured" means. A page can rank fine and still be nearly useless to a crawler-driven agent — client-rendered content, no schema, an ambiguous heading hierarchy, a canonical URL that changes per session. None of that hurts a human reading the rendered page. All of it hurts a fetch-and-parse pipeline.

Start with what actually gets read

The first filter is brutally simple: can the content be read without executing JavaScript? Most agent fetchers — the crawler behind a RAG ingestion job, a browsing tool backing an LLM, a scheduled indexer — do a plain HTTP GET and parse whatever comes back. If your content only exists after a client-side fetch populates a React tree, a lot of these fetchers see an empty shell and a loading spinner in the initial HTML.

This is where server-side rendering or static generation stops being a performance nicety and becomes a correctness requirement. Next.js with the App Router, static generation, or server components solves this by default if you don't fight it — but it's worth checking. View source (not the rendered DOM) on your key pages. If the meaningful text isn't in the raw response, an agent that doesn't execute JS never sees it either.

# Bad: content only exists after client-side hydration
curl https://example.com/services | grep "our services"
# → nothing, because the list renders from a useEffect fetch

# Good: content is present in the server-rendered HTML
curl https://example.com/services | grep "our services"
# → <h2>Our services</h2><ul><li>...

Structured data does the disambiguation work

Prose is ambiguous by design — that's what makes it prose. A paragraph saying "starting at $150/hr, available in the Denver metro" is easy for a human to parse and genuinely hard for a model to extract with confidence, because "starting at" and "$150" and "Denver metro" aren't tied together by anything except word order and convention.

JSON-LD closes that gap by saying the same thing in a form that isn't up for interpretation:

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Service",
  "name": "Infrastructure Consulting",
  "areaServed": "Denver Metro Area",
  "offers": {
    "@type": "Offer",
    "priceCurrency": "USD",
    "price": "150",
    "unitText": "per hour"
  }
}
</script>

The prose and the JSON-LD should say the same thing — schema isn't a place to sneak in claims you wouldn't put in visible text, since that gap is exactly what triggers spam heuristics on the crawling side. For a personal or consulting site, the types worth having are Person or Organization sitewide, Article on writing, and Service or CreativeWork on project pages. FAQPage schema is worth it specifically because it maps almost one-to-one onto how answer engines want content shaped: a direct question, a self-contained answer, no requirement to read the rest of the page for context.

Canonical URLs and one address per fact

Agents that cite sources need a stable thing to cite. If the same content is reachable at /blog/post, /blog/post/, /blog/post?utm_source=twitter, and /en-us/blog/post, and none of them declares a canonical, you've handed the extraction pipeline four different identities for one piece of content. Best case, it picks one arbitrarily. Worst case, it treats them as different sources and your citation count (or your apparent authority on the topic) gets fragmented across duplicates instead of consolidated behind one URL.

The fix is unglamorous and already standard practice: a <link rel="canonical"> on every page, no trailing-slash inconsistency, tracking parameters stripped away, and a sitemap.xml that lists the canonical form only. None of this is AI-specific — it's the same hygiene that's been correct SEO practice for a decade. What's changed is the cost of getting it wrong: a search engine mostly just deduplicates internally, but a RAG pipeline that ingests both URLs as separate documents will happily contradict itself.

Hierarchy is a parsing aid, not a style choice

Heading structure (h1 → h2 → h3, in order, without skipping levels for visual reasons) matters more to a parser than to a human, who mostly infers structure from whitespace and font size anyway. An agent extracting "the answer to X" from a page is frequently doing something close to: find the heading whose text matches the question, take the content until the next heading of the same or higher level. Skip a level, or use a div styled to look like a heading instead of an actual h3, and that extraction breaks silently — no error, just a wrong or truncated answer.

The same logic applies to lists, tables, and definition markup. A price comparison as an actual <table> with <th> headers is machine-parseable. The same comparison as a series of styled <div>s with CSS grid is visually identical to a human and structurally invisible to anything parsing markup rather than rendering it.

llms.txt: useful convention, not a guarantee

llms.txt — a plain-text file at the site root pointing to your key pages and a plain-language summary of the site — is worth having, but it's worth being honest about what it currently is: a community convention maintained through llmstxt.org, not a standard backed by W3C, IETF, or any of the major AI labs. As of 2026, adoption sits around 10% of sites that bother to check, and the crawlers that matter most — GPTBot, ClaudeBot, PerplexityBot, Google-Extended — largely don't fetch it; they crawl and parse HTML directly, the same as they always did. Google's own webmaster team has said on record that they don't support it and don't plan to.

That doesn't make it worthless. It costs almost nothing to add, it's genuinely useful for the growing set of agent tooling that does read it deliberately (coding agents pulling docs, browsing tools given an explicit "check llms.txt first" instruction), and it forces you to write a one-paragraph summary of your own site, which is a useful exercise regardless of who reads the file. Just don't treat it as a substitute for the things that actually move the needle: renderable HTML, structured data, and clean URLs. Put effort there first.

APIs and docs: the same rules, sharper

If you're exposing an API or a docs site meant for agent consumption rather than just human developers, the same principles apply with less room for ambiguity. An OpenAPI spec is the JSON-LD of APIs — it turns "read the prose and figure out the request shape" into "parse a machine-readable contract." Error responses that return structured JSON with a stable code field beat error pages that return an HTML stack trace with a 200 status, which happens more often than it should and is invisible to a human clicking around but breaks any agent checking response.ok.

For docs specifically: one fact stated in one place, versioned pages that don't silently change meaning under a stable URL, and code examples that are copy-paste correct rather than illustrative pseudocode. An agent extracting a code sample doesn't know it's pseudocode. It will run it.

The actual difference from SEO

SEO optimizes for a ranking function that a human then filters through judgment, brand trust, and a dozen blue links to compare against. Agent-readiness optimizes for a single extraction: an agent reads your page once, in isolation, and either gets the right fact or doesn't — there's no scanning ten results and triangulating. That raises the bar on precision and lowers the tolerance for ambiguity, marketing hedging, and "it's somewhere on the page if you read carefully." The overlap with good SEO is real — clean URLs, structured data, fast server-rendered pages help both — but the failure mode is different. Bad SEO costs you ranking position. Bad agent-readiness gets your content parsed wrong, or not parsed at all, with no ranking signal to tell you it happened.