Monday AI Insight. Most RAG projects pick an embedding model in about five minutes: open a leaderboard, take something near the top, embed the corpus, move on. Then six months later a better or cheaper model ships, and the team finds out that switching means re-embedding every chunk they have ever stored.

Here's the lesson I wish I'd learned earlier: your embedding model is a schema decision, not a config value. Every vector in your index is tied to the exact model (and settings) that produced it. Treat the choice like a database migration, because that is what changing it becomes.

1. Vectors from different models don't mix

An embedding is only meaningful relative to other vectors from the same model. A query embedded with model B, searched against documents embedded with model A, gives you similarity scores that look valid but compare points from two unrelated spaces. Nothing errors. Results just quietly get worse.

OpenAI's embeddings guide shows this implicitly in its code-search example: functions are indexed with text-embedding-3-small, and the natural-language query is embedded "using the same model" before computing cosine similarity.

The same applies to settings, not just model names. With OpenAI's third-generation models you can pass a dimensions parameter (the defaults are 1536 for text-embedding-3-small and 3072 for text-embedding-3-large). A 256-dimension query vector can't be compared with a 1536-dimension document vector at all.

Takeaway: define an "embedding version" = model + dimensions + preprocessing (prefixes, chunking, truncation). Every vector you store belongs to exactly one version, and queries must use the same one.

2. Pick with your own eval, not just the leaderboard

The MTEB paper (Muennighoff et al., 2022), which introduced the benchmark behind the popular leaderboard, covered 8 task types, 58 datasets and 112 languages, and its headline finding was that no single text embedding method dominated across all tasks. Models tuned for semantic similarity (STS) performed badly on retrieval, and retrieval models performed badly on STS.

So a high average score tells you a model is generally good. It doesn't tell you it's good at your task: retrieving your documents for your users' questions.

A small, cheap eval is enough to decide:

# eval_embeddings.py: compare candidate models on your own queries # golden: list of (query, set_of_relevant_chunk_ids), 50-100 real user queries def recall_at_k(search_fn, golden, k=10): hits = 0 for query, relevant_ids in golden: top_ids = {r.chunk_id for r in search_fn(query, k=k)} if top_ids & relevant_ids: hits += 1 return hits / len(golden) for version in ["oai-3-small-1536", "oai-3-large-1024", "e5-base-v2"]: print(version, recall_at_k(make_search(version), golden))

Build a separate index per candidate on a sample of your corpus, run the same golden queries, and compare recall@10 alongside cost and latency. The winner is often not the model at the top of the leaderboard.

3. Read the model card: prefixes, input types and max length

Some embedding models expect you to tell them whether a text is a query or a document. Get this wrong and quality drops without any error message.

  • E5 models (for example intfloat/e5-base-v2): the model card says each input should start with "query: " or "passage: ". Its FAQ is blunt: this is how the model was trained, "otherwise you will see a performance degradation."
  • Cohere Embed (v3 and higher): input_type is required. Use search_document for what you store and search_query for what users type.
  • Max input length: OpenAI's third-generation embedding models accept up to 8,192 tokens. The E5-base-v2 model card says long texts are truncated to at most 512 tokens. If your chunks are longer than that, the end of every chunk is silently ignored.

Takeaway: prefix rules, input types and token limits are part of your embedding version. Write them into the same config as the model name so the indexer and the query path can't drift apart.

4. Smaller vectors can be fine (and much cheaper to store)

OpenAI's text-embedding-3 models were trained with Matryoshka Representation Learning (Kusupati et al., 2022), which packs the most useful information into the first dimensions. That's why the dimensions parameter works. OpenAI reports that on MTEB, text-embedding-3-large shortened to 256 dimensions still outperforms the full 1536-dimension text-embedding-ada-002.

From OpenAI's launch post, average MTEB scores were 61.0% (ada-002), 62.3% (3-small) and 64.6% (3-large). On multilingual MIRACL they were 31.4%, 44.0% and 54.9%.

Two practical notes from the docs:

  1. Prefer the dimensions API parameter. If you shorten vectors yourself, you must re-normalize them (L2) afterwards.
  2. OpenAI embeddings come back normalized to length 1, so cosine similarity is just a dot product, and cosine and Euclidean distance produce identical rankings.

Storage scales linearly with dimensions. My own arithmetic: as float32, a 1536-dimension vector is 6,144 bytes and a 3072-dimension vector is 12,288 bytes, before any index overhead. Across 10 million chunks that's roughly 61 GB vs 123 GB of raw vectors. Test a shorter size in your eval before you pay for the full one.

5. Do the re-embedding math before you commit

Here is a worked example using OpenAI's published launch prices ($0.02 per 1M tokens for 3-small, $0.13 per 1M for 3-large; check the current pricing page before you plan):

  • Corpus: 10 million chunks × ~500 tokens = 5 billion tokens
  • text-embedding-3-small: 5,000 × $0.02 = ~$100
  • text-embedding-3-large: 5,000 × $0.13 = ~$650

OpenAI's Batch API supports /v1/embeddings at a 50% discount, with results within 24 hours and a separate rate-limit pool. A single embeddings batch can hold up to 50,000 embedding inputs, so a large corpus means many batches.

The token bill is usually not the hard part, though. The hard part is the operational work: re-chunking, rebuilding the index, and switching traffic without serving a half-migrated index.

6. Make migration boring: version everything

Store the version next to every vector, and migrate by building a new index alongside the old one:

CREATE TABLE chunk_embeddings ( chunk_id bigint REFERENCES chunks(id), emb_version text NOT NULL, -- e.g. 'oai-3-small-1536-v2' content_hash text NOT NULL, -- hash of the exact text embedded embedding vector(1536) NOT NULL, PRIMARY KEY (chunk_id, emb_version) );

The migration playbook:

  1. Pick the new version with the eval from section 2.
  2. Backfill the new emb_version in the background (Batch API if you can wait). Skip chunks whose content_hash hasn't changed since the last run with that version.
  3. Dual-write: new and updated chunks get embedded with both versions until cutover.
  4. Shadow-test: run real queries against both indexes and compare recall on your golden set.
  5. Cut over by changing one config value that sets both the query model and the index version together.
  6. Keep the old version for a rollback window, then delete it.

The rule that prevents most incidents: the query path reads its embedding version from the same place the index does. If they can ever disagree, one day they will.

The checklist

  • Define an embedding version: model + dimensions + prefixes/input type + chunking
  • Choose with recall@k on 50-100 real queries, not the leaderboard average
  • Read the model card for prefixes, input types and max tokens
  • Try shorter dimensions (re-normalize if you truncate manually)
  • Estimate re-embed cost: chunks × tokens × price, and use batch pricing
  • Store emb_version + content_hash with every vector; migrate with a parallel index

Your embedding model will change. Make sure that when it does, it's a planned migration and not a silent quality drop.

What embedding model are you running in production today, and have you ever had to re-embed?

References