Crawlers and sitemaps: robots.txt for AI crawlers, hreflang sitemaps and metadata in head

Name the AI crawlers explicitly, in their own group. Several of them look for a group with their own user agent, and listing them makes letting them in (or not) a visible choice that is easy to change.

// app/robots.ts
import type { MetadataRoute } from 'next';

const PRIVATE = ['/api/', '/oauth/', '/*/cart', '/*/account'];
const AI_AGENTS = [
  'GPTBot', 'OAI-SearchBot', 'ChatGPT-User',
  'ClaudeBot', 'Claude-User', 'Claude-SearchBot',
  'PerplexityBot', 'Perplexity-User',
  'Google-Extended', 'Applebot-Extended',
];

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      {
        userAgent: '*',
        allow: '/',
        disallow: PRIVATE,
        // contentsignals.org: may be searched and used as live input to AI answers; training left unstated.
        other: { 'Content-Signal': 'search=yes, ai-input=yes' },
      },
      { userAgent: AI_AGENTS, allow: '/', disallow: PRIVATE },
    ],
    sitemap: 'https://your-store.example/sitemap.xml',
  };
}

// app/robots.ts
import type { MetadataRoute } from 'next';

const PRIVATE = ['/api/', '/oauth/', '/*/cart', '/*/account'];
const AI_AGENTS = [
  'GPTBot', 'OAI-SearchBot', 'ChatGPT-User',
  'ClaudeBot', 'Claude-User', 'Claude-SearchBot',
  'PerplexityBot', 'Perplexity-User',
  'Google-Extended', 'Applebot-Extended',
];

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      {
        userAgent: '*',
        allow: '/',
        disallow: PRIVATE,
        // contentsignals.org: may be searched and used as live input to AI answers; training left unstated.
        other: { 'Content-Signal': 'search=yes, ai-input=yes' },
      },
      { userAgent: AI_AGENTS, allow: '/', disallow: PRIVATE },
    ],
    sitemap: 'https://your-store.example/sitemap.xml',
  };
}

• Disallow per-shopper and technical routes: cart, account, checkout, the API, OAuth and preview routes.
• Don't disallow search results. A crawler that may not fetch a page never sees its noindex. Let it crawl search results (their links lead to products), and mark them noindex, follow.
• No `Host:` line. It isn't part of the standard, and validators flag it.
• `Content-Signal` (contentsignals.org) states what crawlers may use the content for: search, ai-input (live answers) and ai-train. Leave out what you haven't decided.

Search engines, answer engines and link previews read <title>, the meta description, the canonical and hreflang from the served HTML, and Google ignores a canonical or hreflang outside <head>.

• Server-render the metadata into <head>. A framework that streams it into <body> for ordinary browsers needs to put it in <head> for crawlers (in Next.js, htmlLimitedBots; see Next.js App Router).
• Behind a CDN, check the cached response. If one cached shell serves many URLs, it can't hold metadata that depends on the path, and every crawler that gets the cached copy sees no title at all. Prerender a shell per URL, or render the head per request.
• Verify per user agent on production. curl a product and a category page with the real Googlebot desktop and smartphone strings and with an AI crawler's, and check that <title> comes before </head>.

UA='Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/131.0.0.0 Safari/537.36'
curl -s -A "$UA" https://your-store.example/en/drills/cordless-drill-driver-18v | grep -bo '<title>\|</head>' | head -3

UA='Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/131.0.0.0 Safari/537.36'
curl -s -A "$UA" https://your-store.example/en/drills/cordless-drill-driver-18v | grep -bo '<title>\|</head>' | head -3

• The canonical is the item's own path in the current language, built from the path Discovery returns, not from a typed URL field.
• Filtered and sorted views point at the clean category. Strip filter, sort and page parameters from the canonical.
• A shortcut path (an item placed in a second folder) points at the item's own path.

List every page in every language, each with hreflang alternates to the same item in the other languages. Paths are translated, so join them on itemId, not on the slug:

// app/sitemap.ts (one browse per language, joined on itemId)
const LANGS = ['en', 'no', 'sv'] as const;
const HREFLANG = { en: 'en', no: 'nb', sv: 'sv' } as const;

export default async function sitemap() {
  const all = await Promise.all(LANGS.map(async (l) => [l, await allItems(l)] as const)); // itemId, path, updatedAt
  const byItem = new Map<string, Partial<Record<(typeof LANGS)[number], string>>>();
  for (const [l, items] of all) for (const i of items) byItem.set(i.itemId, { ...byItem.get(i.itemId), [l]: `https://your-store.example/${l}${i.path}` });
  return all.flatMap(([l, items]) =>
    items.map((i) => {
      const urls = byItem.get(i.itemId)!;
      return {
        url: urls[l]!,
        lastModified: i.updatedAt,
        alternates: { languages: { ...Object.fromEntries(Object.entries(urls).map(([k, u]) => [HREFLANG[k as 'en'], u])), 'x-default': urls.en ?? urls[l]! } },
      };
    }),
  );
}

// app/sitemap.ts (one browse per language, joined on itemId)
const LANGS = ['en', 'no', 'sv'] as const;
const HREFLANG = { en: 'en', no: 'nb', sv: 'sv' } as const;

export default async function sitemap() {
  const all = await Promise.all(LANGS.map(async (l) => [l, await allItems(l)] as const)); // itemId, path, updatedAt
  const byItem = new Map<string, Partial<Record<(typeof LANGS)[number], string>>>();
  for (const [l, items] of all) for (const i of items) byItem.set(i.itemId, { ...byItem.get(i.itemId), [l]: `https://your-store.example/${l}${i.path}` });
  return all.flatMap(([l, items]) =>
    items.map((i) => {
      const urls = byItem.get(i.itemId)!;
      return {
        url: urls[l]!,
        lastModified: i.updatedAt,
        alternates: { languages: { ...Object.fromEntries(Object.entries(urls).map(([k, u]) => [HREFLANG[k as 'en'], u])), 'x-default': urls.en ?? urls[l]! } },
      };
    }),
  );
}

• `lastModified` from `updatedAt`, which Discovery returns on every hit.
• Leave out history paths (old paths after a rename), unpublished languages and noindexed pages.
• Above a few thousand URLs, split the sitemap into a sitemap index by type (pages, categories, products) or by language.
• Read the full list reliably. A job that lists every item pages through Discovery with retries, merges the results, and checks the total against a search count, because counts can vary during a re-index.

allItems here is one Discovery browse per shape and language, paged with endToken, selecting itemId, path and updatedAt.