Your RAG is only as good as its retrieval. Here's how to benchmark it.

A retrieval miss doesn't look like an error. When the right passage isn't retrieved, the LLM still answers from the wrong data. Nothing in your logs points at retrieval.

In this article I will show you how to use Truvec to debug and optimize your retrieval.

The 5 knobs

Diagram of a basic RAG pipeline: documents are split into chunks, embedded and stored in a vector index; a user question is embedded, the top-k chunks are retrieved and sent to the LLM with the question.

Everything before the LLM changes which chunks come back:

  1. Chunking
  2. Embedding model
  3. Dense or hybrid search (BM25 + vectors)
  4. Reranker
  5. Top k

Your contracts, support articles and code have their own vocabulary. You can not rely on other people setup to build your own retrieval system.

1. Get a golden dataset

Questions about your docs, each tied to the chunk that answers it.

UI: Data Upload → Golden Set → Create Dataset → Generate Dataset. Review the questions, fix the bad ones, then Commit Dataset. Committed means locked: every run uses the same questions.

Truvec Golden Set page: a committed dataset of 10 questions, each linked to its expected chunk and marked Valid.

API:

API=https://app.truvec.dev AUTH="Authorization: Bearer $TOKEN" # from POST /api/auth/login curl -s -X POST "$API/api/v1/documents/upload?project_id=$PROJECT" -H "$AUTH" \ -F "file=@help-center.pdf" -F "chunking_strategy=recursive" \ -F "chunk_size=1000" -F "chunk_overlap=200" DATASET=$(curl -s -X POST $API/api/v1/golden-datasets -H "$AUTH" \ -H "Content-Type: application/json" \ -d "{\"project_id\": $PROJECT, \"name\": \"v1\", \"number_of_questions\": 20}" | jq .id) curl -s -X POST $API/api/v1/golden-datasets/$DATASET/commit -H "$AUTH"

2. Benchmark the models

UI: Comparison → Compare models → pick the dataset, tick the models, Run Comparison.

Truvec comparison of two embedding models: OpenAI text-embedding-3-small wins with MRR 0.900 against 0.850 for Cohere Embed English v3, both at 90% hit rate, next to cost and latency tables.

Look at that table. Same hit rate: 90% for both. MRR splits them: 0.900 vs 0.850. And the winner is the one priced at $0.020 per 1M tokens, not $0.100.

Hit rate says the right chunk is in the top k. MRR says how high. If you pass 3 chunks to the LLM, that's the one to watch.

API:

curl -s -X POST $API/api/v1/benchmarks -H "$AUTH" \ -H "Content-Type: application/json" \ -d "{\"project_id\": $PROJECT, \"golden_dataset_id\": $DATASET, \"name\": \"small vs large\", \"model_ids\": [1, 2], \"top_k\": 5}"

3. Optimize: one change at a time

Keep one baseline. Add hybrid search in one configuration, a reranker in another. Same questions.

UI: Comparison → Compare strategies. The Hybrid search and Reranker buttons copy the baseline and change one thing.

Truvec Compare strategies form: a baseline with OpenAI text-embedding-3-small, a Dense or Hybrid (BM25 + dense) switch and a reranker selector.

API: the first variant is the baseline.

curl -s -X POST $API/api/v1/benchmarks -H "$AUTH" \ -H "Content-Type: application/json" -d "{ \"project_id\": $PROJECT, \"golden_dataset_id\": $DATASET, \"name\": \"Does a reranker help?\", \"top_k\": 5, \"variants\": [ {\"model_id\": 1}, {\"model_id\": 1, \"pipeline\": {\"retrieval\": \"hybrid\"}}, {\"model_id\": 1, \"pipeline\": {\"reranker\": {\"provider\": \"cohere\", \"model\": \"rerank-v3.5\"}}} ] }"

Then read each one against the baseline:

curl -s $API/api/v1/benchmarks/$BENCHMARK -H "$AUTH" \ | jq '.results.comparison[] | {config: .label, mrr, mrr_delta: .vs_baseline.mrr_delta, fixed: .vs_baseline.fixed, broken: .vs_baseline.broken}'

fixed: questions the baseline missed and this setup found.
broken: the opposite.

Fixing 4 and breaking 3 is not the same as fixing 1 and breaking none, even if the averages match.

What to expect

  • Hybrid shines on product names, error codes and SKUs. On plain prose it can add nothing, or hurt.
  • A reranker can make a cheap model beat an expensive one. It can also cost more per query than the embedding.
  • Read the questions a change broke before you ship it.

Put it in CI

Diagram of a production RAG pipeline with indexing, query and evaluation lanes. A golden dataset feeds retrieval evaluation (recall@k, MRR, nDCG) first, then generation evaluation.

It's all API calls, so run it when your documents change or before you switch models. Retrieval first, generation after.

Free plan, no card, no provider keys to set up. About 15 minutes to a first leaderboard on your own docs.

👉 https://truvec.dev
API docs: https://docs.truvec.dev