Crystallize logo

llms.txt and Markdown versions of the store for language models and agents

Language models and agents read text, and a rendered storefront page is mostly markup. Give them the store in Markdown: an llms.txt that explains the store and links to its sections, an llms-full.txt with the whole catalogue, and a Markdown version of every page built from the same Discovery data as the HTML.

llms.txt

/llms.txt follows the llms.txt convention: a Markdown file at the root of the site that tells a language model what the store is and where to read more. Write it for a model that knows nothing about your business:

  • What the store sells, and to whom. One paragraph.
  • How prices work: markets and currencies, whether catalogue prices include VAT, who sees which, volume prices.
  • How the catalogue is structured: variants and what distinguishes them, compatibility rules, how specs are typed and which classification codes they carry.
  • How to read more: that every page has a Markdown version and how to get it, and where your MCP server is, if you have one (see Agentic commerce).
  • Links to the Markdown version of each category, brand or collection, with a one-line description and a product count.
# Your Store

> Cordless power tools, batteries and accessories for trade and DIY customers in Norway, Sweden and the Netherlands.

- Markets and currencies: Norway in NOK (25% VAT), Sweden in SEK (25% VAT), the Netherlands in EUR (21% VAT). The first path segment decides the market: /no, /sv, /nl.
- Prices in the catalogue exclude VAT. Consumers see prices incl. VAT; trade customers excl. VAT.
- Every page has a Markdown version: append .md to its URL, or request it with Accept: text/markdown.
- MCP server for AI assistants: https://your-store.example/api/mcp

## Catalogue

- [Drills](https://your-store.example/en/drills.md): 36 products. Drill drivers, combi drills and impact drivers.
- [Saws](https://your-store.example/en/saws.md): 41 products. Circular, mitre and jig saws.

/llms-full.txt is the whole catalogue in one file: every product with its variants, SKU, GTIN, price per market, stock, classification and every typed spec with its unit. A model or agent that can read one large file then needs no other request. Build both files from Discovery in a route, cache them with the rest of your Discovery reads, and refresh them on publish (see Caching and revalidation).

A Markdown version of every page

Serve every page as Markdown as well, at the same URL with .md appended (/en.md for the front page, /en/drills/cordless-drill-driver-18v.md for a product), and to requests that ask for Accept: text/markdown:

// proxy.ts: same URL, two representations.
export function proxy(request: NextRequest) {
  const { pathname } = request.nextUrl;
  const accept = request.headers.get('accept') ?? '';
  if (pathname.endsWith('.md') || (accept.includes('text/markdown') && !accept.includes('text/html'))) {
    const url = request.nextUrl.clone();
    url.pathname = `/md${pathname.replace(/\.md$/, '')}`;
    return NextResponse.rewrite(url);
  }
  const res = NextResponse.next();
  res.headers.set('Vary', 'Accept'); // caches must key on Accept
  return res;
}

The Markdown route answers with the page's content, and points back to the HTML page as canonical:

return new Response(page.body, {
  headers: {
    'content-type': 'text/markdown; charset=utf-8',
    'x-robots-tag': 'noindex',
    link: `<${page.canonical}>; rel="canonical"`,
    vary: 'Accept',
  },
});
  • `X-Robots-Tag: noindex` keeps the Markdown out of search results, where it would compete with the HTML page.
  • Announce it from the HTML page with <link rel="alternate" type="text/markdown" href="….md">. In Next.js that is alternates.types['text/markdown'] in the page metadata.
  • Use absolute links in the Markdown, to the .md versions of other pages, so an agent can keep reading in Markdown.

What goes in it

  • A product page: the name, a short description, a table of variants (SKU, GTIN, price excluding and including VAT, stock), the spec sheet as a table with units, what is in the box, compatibility and relations as links, and the breadcrumb.
  • A category page: the introduction and a table of products with brand, key specs, price and stock, each linking to the product's Markdown.
  • Prices: say whether a price includes VAT and in which currency, in the table header.
| Products | Brand | Specifications | EUR Net (excl. VAT) | Stock |
|---|---|---|---|---|
| [Cordless drill driver 18V](https://your-store.example/en/drills/cordless-drill-driver-18v.md) | Example Brand | 18 V · 60 Nm · 1.1 kg | €123.14 | In stock |

The same facts everywhere

Build the HTML spec sheet, the JSON-LD and the Markdown from the same typed rows, read once from Discovery, with one function that formats a value and its unit. Then the three versions can't disagree, and an answer engine that compares them finds the same voltage, weight and price wherever it looks. See Structured data.

tip

See it live

Tools Universe serves llms.txt, llms-full.txt and a Markdown version of every page, such as the front page and a category. superfast.shop/llms.txt does the same for the whole set of demo stores.

Keep exploring

Crawlers and sitemaps: robots.txt for AI crawlers, hreflang sitemaps and metadata in head

Decide which crawlers may read the store and what they may use it for, list every page in every language in the sitemap, and make sure the HTML each crawler gets carries its title, canonical and hreflang in head. Most AI crawlers don't run JavaScript, so what isn't in the served HTML doesn't exist for them.