Caching Discovery reads and revalidating on publish

Cache every published Discovery read on the server, keyed on the query and its variables, and tag it so you can clear it later. In Next.js with Cache Components, that is a 'use cache' function:

import { cacheLife, cacheTag } from 'next/cache';

const ENDPOINT = `https://api.crystallize.com/${process.env.CRYSTALLIZE_TENANT_IDENTIFIER}/discovery`;

export async function discovery<T>(query: string, variables: Record<string, unknown> = {}): Promise<T> {
  'use cache';
  cacheLife('hours');
  cacheTag('discovery');
  return request<T>(query, variables);
}

async function request<T>(query: string, variables: Record<string, unknown>): Promise<T> {
  for (let attempt = 0; ; attempt++) {
    const res = await fetch(ENDPOINT, {
      method: 'POST',
      headers: { 'content-type': 'application/json' },
      body: JSON.stringify({ query, variables }),
    });
    const text = await res.text();
    try {
      // Rate limits and gateway errors answer with HTML, not JSON: retry with backoff.
      if (res.status === 429 || res.status >= 500) throw new Error(`Discovery ${res.status}`);
      const json = JSON.parse(text);
      if (json.errors) throw new Error(json.errors.map((e: { message: string }) => e.message).join('\n'));
      return json.data as T;
    } catch (e) {
      if (attempt >= 4) throw e;
      await new Promise((r) => setTimeout(r, 250 * 2 ** attempt));
    }
  }
}

import { cacheLife, cacheTag } from 'next/cache';

const ENDPOINT = `https://api.crystallize.com/${process.env.CRYSTALLIZE_TENANT_IDENTIFIER}/discovery`;

export async function discovery<T>(query: string, variables: Record<string, unknown> = {}): Promise<T> {
  'use cache';
  cacheLife('hours');
  cacheTag('discovery');
  return request<T>(query, variables);
}

async function request<T>(query: string, variables: Record<string, unknown>): Promise<T> {
  for (let attempt = 0; ; attempt++) {
    const res = await fetch(ENDPOINT, {
      method: 'POST',
      headers: { 'content-type': 'application/json' },
      body: JSON.stringify({ query, variables }),
    });
    const text = await res.text();
    try {
      // Rate limits and gateway errors answer with HTML, not JSON: retry with backoff.
      if (res.status === 429 || res.status >= 500) throw new Error(`Discovery ${res.status}`);
      const json = JSON.parse(text);
      if (json.errors) throw new Error(json.errors.map((e: { message: string }) => e.message).join('\n'));
      return json.data as T;
    } catch (e) {
      if (attempt >= 4) throw e;
      await new Promise((r) => setTimeout(r, 250 * 2 ** attempt));
    }
  }
}

The same idea works in any framework: a server-side cache keyed on query and variables, a tag or key prefix you can purge, and a lifetime long enough that most requests are cache hits.

• Build the variables in a fixed order. Sort SKU lists and object keys. An argument whose order depends on an API answer misses the cache on the next render.
• Choose the lifetime for cost as well as freshness. On a hosted cache every revalidation is a write. One build revalidated every key once a minute, and cache writes became most of its hosting bill. Hourly revalidation plus a publish webhook gives fresh content at a fraction of the cost.
• Keep per-shopper data out of shared cache entries. Contract prices and taste-ranked lists are cached per organisation or per shopper, with their own tags, or computed per request.

Editors preview unpublished content through Discovery. Read it on the server with publicationState: draft, behind your preview mode, and never cache the answer:

query Preview($path: String!) {
  search(language: en, publicationState: draft, filters: { path: { equals: $path } }, pagination: { limit: 1 }) {
    hits { itemId name path publicationState }
  }
}

query Preview($path: String!) {
  search(language: en, publicationState: draft, filters: { path: { equals: $path } }, pagination: { limit: 1 }) {
    hits { itemId name path publicationState }
  }
}

• Only in preview mode. Turn it on with your own check (a signed preview cookie or a secret), render those pages dynamically, and keep them out of every cache.
• Mark preview pages `noindex`, and keep preview routes out of the sitemap.

Editors open the preview from a custom view in the Crystallize App, which frames your storefront next to the item they are editing. Preview with custom views covers the preview route, draft reads inside this cached client, and the headers and cookies the App's frame needs.

Publishing in Crystallize updates Discovery within seconds. Your cache doesn't know that until you tell it. Register a Core webhook for publish and unpublish on items, and point it at a route in your storefront:

for (const event of ['publish', 'unpublish']) {
  await core(
    `mutation($input: CreateWebhookInput!) {
       createWebhook(input: $input) {
         __typename
         ... on Webhook { id }
         ... on BasicError { errorName message }
       }
     }`,
    {
      input: {
        name: `storefront-revalidate-${event}`,
        concern: 'items',
        event,
        method: 'POST',
        url: 'https://your-store.example/api/revalidate',
        headers: [{ name: 'x-revalidate-secret', value: process.env.CRYSTALLIZE_WEBHOOK_SECRET }],
        preserveDefaultPayload: true,
        graphqlQuery:
          'query ($id: ID!, $language: String!) { item(id: $id, language: $language) { ... on Item { id tree { path } } ... on Product { variants { sku } } } }',
      },
    },
  );
}

for (const event of ['publish', 'unpublish']) {
  await core(
    `mutation($input: CreateWebhookInput!) {
       createWebhook(input: $input) {
         __typename
         ... on Webhook { id }
         ... on BasicError { errorName message }
       }
     }`,
    {
      input: {
        name: `storefront-revalidate-${event}`,
        concern: 'items',
        event,
        method: 'POST',
        url: 'https://your-store.example/api/revalidate',
        headers: [{ name: 'x-revalidate-secret', value: process.env.CRYSTALLIZE_WEBHOOK_SECRET }],
        preserveDefaultPayload: true,
        graphqlQuery:
          'query ($id: ID!, $language: String!) { item(id: $id, language: $language) { ... on Item { id tree { path } } ... on Product { variants { sku } } } }',
      },
    },
  );
}

This is a setup script, run once against the Core API. It is not storefront code.

The route checks the secret and clears the tag. Clearing the one discovery tag refreshes everything built from Discovery: pages, JSON-LD, the sitemap, Markdown versions and llms.txt.

import { revalidateTag } from 'next/cache';

export async function POST(req: Request) {
  if (req.headers.get('x-revalidate-secret') !== process.env.CRYSTALLIZE_WEBHOOK_SECRET) {
    return new Response('Forbidden', { status: 403 });
  }
  revalidateTag('discovery', { expire: 0 });
  return Response.json({ revalidated: ['discovery'] });
}

import { revalidateTag } from 'next/cache';

export async function POST(req: Request) {
  if (req.headers.get('x-revalidate-secret') !== process.env.CRYSTALLIZE_WEBHOOK_SECRET) {
    return new Response('Forbidden', { status: 403 });
  }
  revalidateTag('discovery', { expire: 0 });
  return Response.json({ revalidated: ['discovery'] });
}

On a large catalogue, read path and sku from the payload and clear narrower tags (product:<sku>, category:<path>) instead. See Webhooks for webhook concerns and payloads.

• Publishing is asynchronous. publishItem answers before the published version exists, and Discovery indexes it shortly after.
• New shapes and components need a re-index before they appear in the Discovery schema. The new schema reaches every node over minutes, and an error answered in that window can be cached for that query.
• Ship code that queries a new field only after the field answers reliably. Query it several times in a row before you deploy the page that depends on it.
• Cached reads must not throw. A cached function that throws during background revalidation keeps serving the error. Catch, log and fall back to an empty or previous result.

During a large build or a re-index, Discovery can answer HTTP 200 with errors: [{ message: "Response Error" }]. Retry on that message too, cap prerender concurrency, or render pages on demand instead of all at build time. Totals can also vary while an index is rebuilt. A job that must see every item retries, merges the results and checks the count against a search total.

Orders appear in orders(customerIdentifier) after seconds to minutes. Read by id for anything you just wrote, and never use the order list to decide whether an order already exists. Key on your own metadata instead.