Crystallize logo

Vector Search Personalization

Discovery vectors let you rank your whole catalogue for each individual shopper, from their first click, in the same Discovery query you already use for search and categories. You describe products in a vocabulary of your own, such as roast, flavour, region or margin band. Each shopper is described in that same vocabulary, based on what they search, browse and buy.

The index scores every product against the shopper using cosine similarity. You can then add your own weighted rules for relevance, margin, stock and recency with rankBy, and rankExplain shows how much each signal contributed to a product's position. Nothing is trained and there is no extra infrastructure: these are sparse, named vectors you define yourself, not machine-learning embeddings.

Use context.userTaste for personal ranking and nearestTo for "more like this". An AI agent can also send a customer's taste vector with each search, so the results it reads are already ranked for that customer.

Every visitor gets their own storefront. Discovery vectors rank your whole catalogue for the person in front of it, from the first click, in the same query you already use for search and categories. No purchase history, no offline model, no third-party personalization layer.

The idea is simple. You describe products in a vocabulary of your own — roast, flavour, region, margin band, whatever sells in your business. A shopper is described in that same vocabulary, built from what they search, browse and buy. The index scores every product against the shopper with cosine similarity and sorts. Then you put your own rules on top — relevance, margin, stock, recency, each with its own weight — and every position can come back with an explanation.

Nothing is trained. These are sparse, author-supplied, named vectors — not dense machine-learning embeddings, and not a k-nearest-neighbour model. The vocabulary is the model.

What you get

  • Hyper-personal from the first visit. A shopper who taps "chocolate" sees a chocolate-first shelf on the next request. Search, categories and recommendations all follow the same profile.
  • Your rules stay in charge. Taste is one term in the score. Margin, stock and campaign weight sit next to it, and you decide the balance per surface.
  • One query, no new infrastructure. Ranking happens inside the Discovery index, at search speed, on every page of results. Nothing to train, nothing to sync.
  • Explainable. rankExplain returns the contribution of every signal, so merchandisers tune weights against numbers and shoppers can be told why.
  • Ready for agents. An AI agent holds the customer’s taste vector and sends it with every search, so what it reads is already ranked for that customer.

Typical uses. Search results that reorder per shopper, category pages that open with the right products, "more like this" and pairing suggestions from a basket, cold-start onboarding from three picks, B2B catalogues ranked by a buyer’s assortment, and content or course catalogues ranked by a learner’s interests.

This page covers the whole path: modelling a vocabulary, attaching taste to items, indexing, ranking the catalogue by your own rules (rankBy), ranking it for one shopper (context.userTaste), and "more like this" (nearestTo). General filtering, faceting, sorting and pagination are documented on the Discovery API reference pages — this page stays focused on vectors and ranking.

How it works

A vector is a list of numbers. Discovery keeps one per item per vocabulary, materialized at index time, and you send one per shopper at query time. The index scores each item by how well its vector points in the same direction as the shopper’s (cosine similarity) and sorts by that score. Because the score is computed inside the index, it works on the first visit, on every page of results, and for every search term.

Five concepts, and the rest is plumbing:

ConceptWhat it is
VocabularyA named set of dimensions, each carrying a weight list. A tenant can have several; each is scored on its own.
DimensionOne axis inside a vocabulary — roast, flavor, region. Carries a weight list.
EntryA dimensionId:value key attached to one item, in a meaningful order.
Item vectorBuilt at index time. Every entry becomes a key with the weight its dimension and position give it.
Shopper vectorThe same shape, sent with the query as context.userTaste. The index scores every item against it.

Two properties follow from this model and are worth keeping in mind while you design:

  • Shoppers and products share the keys. A shopper who likes flavor:chocolate at 1.0 is compared to products that carry flavor:chocolate. Nothing has to be learned — the vocabulary is the model.
  • Vocabularies are summed. Send one shopper vector per vocabulary and the index adds the per-vocabulary cosines. A product matching two vocabularies outranks a product matching one.

Before you start

Two APIs are involved, and the distinction matters. Authoring — vocabularies, taste entries, publishing and indexing — is done with authenticated Core API mutations. Querying is done against the public Discovery API with no credentials.

APIEndpointAuthUsed for
Corehttps://api.crystallize.com/@<tenant>/coreAccess token pairVocabularies, taste entries, publishing, indexing
Discoveryhttps://api.crystallize.com/<tenant>/discoveryNoneQuerying

Core takes @tenant with an at-sign; Discovery takes the bare tenant. Vocabulary and index mutations live on Core, not on the PIM API.

warning

The vector surface is hidden until ranking is enabled for the tenant

Ranking is enabled per tenant. Until ranking is enabled for a tenant, the vector arguments — rankBy, context and nearestTo — are simply absent from the tenant’s Discovery schema. Referencing them fails GraphQL validation; it does not return unranked results.

If ranking is enabled but a query still passes a vector argument where it is not available, it is refused with an error stating that ranking is not available for this tenant.

The surface can also shrink again if ranking is disabled for a tenant. Treat the vector arguments as a capability to detect at runtime, not as a constant — introspect the schema (or probe once and cache) rather than assuming they exist.

1. Design the vocabulary

Decide what a person could plausibly have a preference about. That is your dimension list. Two tests:

  • Could a shopper move a slider for it? Roast level, yes. SKU prefix, no. Dimensions nobody has taste about add noise to every score.
  • Does assignment order mean anything? If the first flavour listed is the signature note, give the dimension a positional weight list. If not, one weight is enough.

Keep genuinely different concerns in separate vocabularies. What a coffee tastes like and where it comes from are questions a shopper may weigh differently, so they are two vocabularies rather than five dimensions in one. Remember the vocabularies are scored independently and summed, so splitting a concern out lets a shopper (and you, via rankBy) weight it on its own.

taste     roast    [1.0]              one roast level, order meaningless
          flavor   [1.0, 0.6, 0.3]    ordered: the signature note dominates
          body     [0.8]              matters, but less than roast

origin    region   [1.0]
          process  [1.0, 0.5]

Values become the second half of every key, so keep them slug-shaped and stable: flavor:blackcurrant, not flavor:Black Currant. Renaming a value later means rewriting every item that used it and re-indexing.

A vocabulary does not have to be about taste in the narrow sense. Margin band, stock depth, seasonality, audience or tier all work as dimensions when you want them to be part of a similarity score. Use rankBy (below) when they should stay separate and be weighted on their own.

Weights are positional

A dimension’s weight is a list. One element means every entry in that dimension weighs the same. More than one means entry i takes weight[min(i, length − 1)], clamping to the last element once you run past the end.

weight: [1.0, 0.6, 0.3]    1st entry  1.0
                           2nd entry  0.6
                           3rd entry  0.3
                           4th entry  0.3   (clamped to last)

2. Create the vocabulary

upsertVocabulary creates or replaces a vocabulary in full. It is a full replace, not a patch, so send the whole dimension list every time — any dimension you omit is gone.

# Core APIhttps://api.crystallize.com/@<tenant>/core
mutation UpsertVocabulary($input: UpsertVocabularyInput!) {
  upsertVocabulary(input: $input) {
    name
    dimensions { id weight }
    lastUpdated
  }
}
{
  "input": {
    "name": "taste",
    "dimensions": [
      { "id": "roast",  "weight": [1.0] },
      { "id": "flavor", "weight": [1.0, 0.6, 0.3] },
      { "id": "body",   "weight": [0.8] }
    ]
  }
}

Read it back with vocabulary(name: "taste") { dimensions { id weight } }.

3. Attach taste to items

setItemTaste writes the ordered entries for one item, one language, one vocabulary. Two vocabularies means two calls per item.

# Core API
mutation SetItemTaste($input: SetItemTasteInput!) {
  setItemTaste(input: $input) {
    __typename
    ... on Product  { id }
    ... on Folder   { id }
    ... on Document { id }
    ... on ItemNotFoundError { message }
    ... on UnknownError      { message }
  }
}
{
  "input": {
    "itemId": "<ITEM_ID>",
    "language": "en",
    "vocabulary": "taste",
    "entries": [
      { "key": "roast:light" },
      { "key": "flavor:berry" },
      { "key": "flavor:blackcurrant" },
      { "key": "flavor:caramel" },
      { "key": "body:light", "weight": 1.2 }
    ]
  }
}

Order is the input to positional weights

Entries are read in array order, per dimension. Above, flavor:berry takes 1.0, flavor:blackcurrant 0.6 and flavor:caramel 0.3. The optional per-entry weight overrides the vocabulary weight for that entry only — and still consumes its positional slot, so it does not shift the entries that follow.

The key format is validated

Every key is <dimensionId>:<value>. Everything before the first colon must be a declared dimension in that vocabulary. Both mistakes reject the whole call, so there are no partial writes — fix the key and resend the full set of entries.

Bad entryRejection message
{ key: "chocolate" }Malformed taste entry key "chocolate": expected "<dimensionId>:<value>"
{ key: "mood:cozy" }Unknown dimension "mood" in vocabulary "taste". Known dimensions: roast, flavor, body

If you also assign topics for the same concepts, keep one source of truth: derive the taste entries from the topic assignments (for example by storing the key in the topic’s meta) rather than maintaining two parallel lists.

4. Publish

setItemTaste writes to the draft version. Your shopper-facing Discovery queries rank against the published version, so if items were published before you attached taste, publish them again or the served vectors will be empty.

# Core API
mutation Publish($ids: [ID!]!, $language: String!) {
  publishItems(ids: $ids, language: $language) { __typename }
}
warning

Skipping the publish is a silent failure

Skipping this step produces no error. Queries still return results, and the order may even change — it just has no relation to taste, because the ranked (published) documents carry no vectors. The check in step 7 is the only thing that catches it.

5. Index

igniteDiscoApi materializes the vectors. Run it after any change to vocabularies or taste entries, not only the first time — an unindexed change has no effect.

# Core API
mutation Index {
  igniteDiscoApi {
    __typename
    ... on BulkTaskIgnition { id type status createdAt }
    ... on UnauthorizedError { message }
    ... on UnknownError { message }
  }
}

The mutation is asynchronous. Poll the returned task until it reports complete, then allow a few minutes for Discovery to propagate the new index.

query Task($id: ID!) {
  bulkTask(id: $id) {
    ... on BulkTask { id status }   # pending, started, complete, error
  }
}

Every search reports when the index last finished — confirm the timestamp moved after your run:

{ search { summary { profiling { lastIndexCompletedAt } } } }

From then on the schema includes context, rankBy and nearestTo, and every vocabulary you created is a value of the TenantVocabularyIdentifier enum. A vocabulary only becomes a valid enum value after the next index run — until then, queries referencing it fail schema validation.

6. Rank your catalogue

Before any shopper enters the picture, decide how your catalogue should be ordered when nobody has said anything yet. rankBy turns any number or date on a product into a ranking signal — one term per signal, each with its own weight. The same mechanism carries the shopper’s taste later, so the rules you set here are the baseline every personalized result is built on.

What can drive the order: margin, stock, sales velocity, newness, campaign priority, review score, return rate (with a negative weight) — any signal you can express as one of the five term types below.

# Discovery APIhttps://api.crystallize.com/<tenant>/discovery
query Catalogue($rankBy: RankByInput!) {
  search(term: "espresso", rankBy: $rankBy, pagination: { limit: 24 }) {
    hits {
      ... on product {
        name
        score
        rankScore
        rankExplain { signal index contribution }
      }
    }
  }
}
{
  "rankBy": {
    "terms": [
      { "signal": "relevance",    "weight": 1.0 },
      { "signal": "fieldBoost",   "weight": 0.6, "field": "margin",     "normalize": true },
      { "signal": "fieldBoost",   "weight": 0.5, "field": "sold_30d",   "normalize": true },
      { "signal": "inStockBoost", "weight": 0.4, "field": "stock_default" },
      { "signal": "recency",      "weight": 0.3, "field": "publishedAt", "halfLifeDays": 60 }
    ],
    "tieBreaker": "sku",
    "explain": true
  }
}

The score of a product is the weighted sum of its terms — rankScore = Σ weight × term.

rankScore = 1.0·relevance + 0.6·margin + 0.5·sold_30d + 0.4·inStock + 0.3·recency

terms must hold at least one term. tieBreaker is required — it settles ties deterministically and is a per-tenant enum (TenantRankByTieBreaker). Negative weights are legal penalties, so a high return rate can push a product down.

The five signals

SignalRequiredOptionalWhat it scores
relevanceText match against term. Contributes 0 when there is no term.
fieldBoostfieldnormalizeAny rankable number on the item, optionally normalized across the window.
inStockBoostfieldFavours items with stock in the given stock field.
recencyfield, halfLifeDaysDecays with the age of a date field.
tasteCosinevocabulary, fromThe shopper’s taste (from: userTaste) or similarity to a reference item (from: nearestTo). See steps 7–9.

Zero weights are dropped

A term with weight: 0 is dropped before scoring. If every term is 0, the query falls back to plain relevance and skips the rerank window entirely — no rerank runs, and rankScore comes back null.

Normalization defaults

normalize applies min–max normalization across the rerank window, so signals on different scales can share a weight. The default depends on the signal — set normalize explicitly to override it.

Signalnormalize defaultWhy
relevanceonUnbounded magnitude.
fieldBoostonUnbounded magnitude.
tasteCosineoffAlready roughly 0–1.
recencyoffAlready roughly 0–1.
inStockBoostoffAlready roughly 0–1.

Multi-valued fields are collapsed to one value

Anything with several values per item — fields under variants (such as price_* / stock_*), fields under shortcuts, or a repeatable component — is collapsed before scoring. The direction is fixed and cannot be overridden:

  • fieldBoost → lowest. The "from" price a shopper is shown.
  • inStockBoost → highest. Buyable in any variant counts as in stock.
  • recency → highest. The most recent date wins.

The rankable fields are per-tenant enums

The valid field and tie-breaker values are derived from your index settings, not free-form:

  • TenantRankByField (fieldBoost / inStockBoost / recency field, and the field of a fieldBoost) is built from your NUMBER and DATE filterable attributes, with facet fields excluded — it is not "any numeric field".
  • TenantRankByTieBreaker is built from your sortable fields (token, number and date).
  • TenantVocabularyIdentifier is built from your vocabularies, and only after the next index run.

rankScore and rankExplain

rankScore on each hit is the raw, un-normalized value the hits were ordered by — it is the actual sort key, distinct from score, which stays text relevance. rankScore is null when no rerank ran (no rankBy, or every term dropped).

With explain: true, each hit carries one rankExplain entry per term: { signal, index, contribution }, in rankBy.terms order. index disambiguates a repeated signal (two fieldBoost terms, say), and the contributions sum to rankScore by construction. rankExplain is populated only with explain: true, and is null on the userTaste-only and nearestTo-only paths, which have no terms.

{
  "name": "Cobán Dark",
  "rankScore": 1.71,
  "rankExplain": [
    { "signal": "relevance",    "index": 0, "contribution": 0.42 },
    { "signal": "fieldBoost",   "index": 1, "contribution": 0.51 },
    { "signal": "fieldBoost",   "index": 2, "contribution": 0.30 },
    { "signal": "inStockBoost", "index": 3, "contribution": 0.40 },
    { "signal": "recency",      "index": 4, "contribution": 0.08 }
  ]
}

7. Rank by a shopper vector

Build a vector that describes what this visitor wants and pass it as context.userTaste. The index does the cosine, the summing across vocabularies and the sort.

input UserTasteInput {
  vocabulary: TenantVocabularyIdentifier!   # enum, generated per tenant
  weights:    JSON!                         # { "dimensionId:value": number }
  magnitude:  Float!                        # L2 norm of weights
}

input ContextInput {
  userTaste: [UserTasteInput!]!             # one entry per vocabulary
}

userTaste is an array with one entry per vocabulary. It must be non-empty, and a vocabulary may appear at most once. weights is a JSON map of dimensionId:value to a number — positive means likes, negative means dislikes. Those keys contain a colon, which is not a valid GraphQL name, so pass the context as a variable rather than inline.

# Discovery API
query SearchWithTaste($context: ContextInput!) {
  search(context: $context, pagination: { limit: 24 }) {
    summary { totalHits }
    hits {
      ... on product {
        itemId
        name
        rankScore
        defaultVariant { sku defaultPrice firstImage { url } }
      }
    }
  }
}
{
  "context": {
    "userTaste": [
      {
        "vocabulary": "taste",
        "weights": { "flavor:berry": 1.0, "roast:light": 1.0 },
        "magnitude": 1.4142135623730951
      },
      {
        "vocabulary": "origin",
        "weights": { "region:kenya": 1.0 },
        "magnitude": 1.0
      }
    ]
  }
}

The two entries are scored separately and added, so Kenyan light-roast berry coffees rise above coffees that satisfy only one of them.

You supply the magnitude

magnitude is the length of the shopper vector — one number that lets the index compare direction (what they like) without being fooled by size (how much they like it). It is sqrt(Σ w²): square every weight, add them up, take the square root. For { "flavor:berry": 1.0, "roast:light": 1.0 } that is sqrt(1 + 1) = 1.414.

type SparseVector = Record<string, number>;

const magnitude = (w: SparseVector) =>
  Math.sqrt(Object.values(w).reduce((sum, x) => sum + x * x, 0));

const toUserTaste = (vocabulary: string, weights: SparseVector) => ({
  vocabulary,
  weights,
  magnitude: magnitude(weights),
});
warning

A wrong magnitude silently skews every ranking

A miscomputed magnitude raises no error. It quietly distorts every cosine, so the results look plausible and are wrong. Compute it in exactly one place, unit test it against a known case, and prune near-zero weights before sending — they cost payload and contribute nothing beyond rounding.

Where the shopper vector comes from

The vector is UI state or session state, whichever you have. Nothing about the shopper is stored in Crystallize — you send the vector with each query.

  • Sliders and chips. A slider from light to dark spread over roast:light, roast:medium, roast:dark; a chip that sets flavor:chocolate to 1.
  • Cold start from picks. Let a new visitor pick two or three products, read their taste, rebuild the vectors with the same positional rule, and sum them. No history needed.
  • Decayed session behaviour. Add a little weight for every product a shopper opens or adds, decay it over time, and send the result. The profile lives in the browser or your session store.
  • Agents. An AI agent holds a structured taste vector for a customer and sends it with every search, so the results it reads are already ranked for that customer.

Check that it works

Run the same query with and without context and compare the order. If the two match, no vectors reached the index — go back to step 4 (publish) and step 5 (index).

8. Blend taste with your rules

context alone ranks purely by taste. To put the shopper on top of the catalogue rules from step 6, add one tasteCosine term to the same rankBy. Taste terms read the shopper vector from context, so pass both.

# Discovery API
query Blended($context: ContextInput!, $rankBy: RankByInput!) {
  search(term: "espresso", context: $context, rankBy: $rankBy, pagination: { limit: 24 }) {
    hits {
      ... on product {
        name
        rankScore
        rankExplain { signal index contribution }
      }
    }
  }
}
{
  "rankBy": {
    "terms": [
      { "signal": "relevance",    "weight": 1.0 },
      { "signal": "fieldBoost",   "weight": 0.6, "field": "margin", "normalize": true },
      { "signal": "inStockBoost", "weight": 0.4, "field": "stock_default" },
      { "signal": "recency",      "weight": 0.3, "field": "publishedAt", "halfLifeDays": 60 },
      { "signal": "tasteCosine",  "weight": 1.2, "vocabulary": "taste", "from": "userTaste" }
    ],
    "tieBreaker": "sku",
    "explain": true
  }
}
rankScore = 1.0·relevance + 0.6·margin + 0.4·inStock + 0.3·recency + 1.2·cos(taste)

The split is explicit: the first four terms are your rules and apply to everyone, the last is this shopper. Raise the taste weight and the shelf becomes more personal; lower it and your rules take over. One tasteCosine term per vocabulary lets you weight vocabularies independently (taste at 1.2, origin at 0.3), which plain context cannot do. tasteCosine contributes 0 for items with no vector in that vocabulary, so unmapped products fall back to your rules rather than disappearing.

9. More like this

nearestTo ranks by similarity to a reference item instead of a hand-built vector. Use it for "similar products" on a product page, or for pairing suggestions from a basket.

# Discovery API
{
  search(
    nearestTo: { vocabulary: taste, like: { sku: "ET-006" }, k: 12 }
  ) {
    hits { ... on product { itemId name rankScore } }
  }
}

The shape is { vocabulary, like: { sku | itemId }, k }. Pass exactly one of sku or itemId. The anchor item is excluded from the results, and an anchor with no vector for that vocabulary yields an empty result.

k replaces pagination.limit when nearestTo is set, and is itself capped by the rerank window. nearestTo without rankBy scores from the anchor vector only — context.userTaste is not combined into it. To blend neighbours with stock, margin or taste, add a tasteCosine term with from: nearestTo to a rankBy.

The rerank window

options.rerankWindow is the single knob shared by all three rerank paths (rankBy, context and nearestTo). The top-N candidates are scored and re-sorted; results beyond N keep plain relevance order. The default is 500 and the hard server cap is 2000. Rerank cost is per candidate, so a larger window costs more.

search(
  term: "espresso",
  rankBy: $rankBy,
  options: { rerankWindow: 1000 }
) { hits { ... on product { name rankScore } } }
warning

Paging happens inside the window

Under ranking, skip offsets into the reranked window — a skip past the window returns nothing. Size the window for the depth you intend to page, not just the first page.

How ranking fits the rest of the query

Ranking only orders the filtered set. It composes with the rest of the query rather than replacing it:

  • filters, facets and term apply as usual. Ranking decides the order inside the filtered set; it does not change which items match.
  • An explicit sorting selects which candidates enter the window. The rerank then imposes the final order on top. When context or rankBy is present, the reranked order is what you get back.
  • Items without a vector are still returned. They score 0 on taste and sort after the ones that match, rather than disappearing.
  • Discovery is public. Ranking needs no credentials, so a browser can call it directly with the shopper vector built client-side.

The vector arguments are accepted on search, autocomplete, browse, and the children queries on topics and folders. Deep coverage of filtering, faceting, sorting and pagination lives on the Discovery API reference pages.

Troubleshooting

SymptomCauseFix
Order unrelated to taste, no errorsTaste is on the draft onlyPublish the items, index again, retest
Vocabulary "does not exist in enum"No index run since it was createdRun igniteDiscoApi, wait for complete
context or rankBy unknown in schemaRanking not enabled for the tenant / not indexedConfirm ranking is enabled for the tenant, then index and wait for complete plus a few minutes of propagation; detect the capability rather than assuming it
Malformed taste entry keyKey has no colonUse dimensionId:value
Unknown dimensionPrefix is not a declared dimensionAdd it to the vocabulary, or fix the key
Rankings subtly wrong, no errorsmagnitude miscomputed client-sideAssert sqrt(Σ w²) against a known case
tieBreaker validation errorRequired field missing on rankByAdd a tieBreaker field

The whole sequence

Core        upsertVocabulary            once per vocabulary (full replace)
Core        setItemTaste                once per item, per vocabulary (writes draft)
Core        publishItems                the step that is easy to miss
Core        igniteDiscoApi              poll bulkTask until "complete", then let it propagate
Discovery   search(rankBy:)             ranked by your rules: margin, stock, velocity
Discovery   search(context:)            ranked for this shopper
Discovery   search(rankBy: + context:)  your rules, plus this shopper
Discovery   search(nearestTo:)          ranked by a reference item

Re-run the index whenever vocabularies or taste entries change. Everything before Discovery is authenticated and server-side; the queries themselves need no credentials.