How to Evaluate RAG Retrieval in 2026: Question Sets, Metrics That Mislead, and a Practical Recipe

Many RAG failures get blamed on "the LLM" when the generator never had a chance: the passage it needed was not in the context, or it was buried under near-duplicates. Retrieval deserves its own evaluation. It is also easy to get wrong, because the score depends on what you call "relevant", how you built the questions, which k you chose and how many questions you asked.

This article covers the whole chain as it stands in late 2026: labelled question sets (including synthetic ones and their traps), the metrics and where each misleads, statistical noise, lexical versus paraphrased questions, evaluating chunking, hybrid search and rerankers, LLM-as-judge and its documented biases, end-to-end metrics, current tools and benchmarks, and a recipe you can apply this week.

Numbers of my own come from small, reproducible experiments in my public lab repo, applied-llm-lab. They show how metrics behave; they are not a benchmark.

Why evaluate retrieval on its own

Retrieval quality changes answer quality in ways the generator cannot fix. Google researchers defined sufficient context as context that "contains all the necessary information to provide a definitive answer" and found that insufficient context can make models less likely to abstain: for one model the incorrect-answer rate rose from 10.2% with no context to 66.1% with insufficient context [2]. A weak retriever can actively produce confident wrong answers.

Serious RAG work measures the retriever directly. Anthropic's Contextual Retrieval write-up used "1 minus recall@20" and reported failure rates falling from 5.7% to 3.7% with contextual embeddings, to 2.9% with contextual BM25 added, and to 1.9% with reranking [1], all measured before any answer was generated.

Retrieval-only metrics are also cheaper, deterministic once labels are fixed, and they localise problems: if recall@20 is fine but answers are wrong, look at the prompt or generator; if recall@20 is poor, no prompt will save you.

Step zero: decide what "relevant" means

Every retrieval metric compares a ranked list against a set of relevant items. Before choosing a metric, decide what the unit of relevance is:

Chroma's chunking study notes that document-level benchmarks like MTEB and BEIR "cannot take chunking into account" and measures token-wise precision, recall and intersection over union instead [18]. RAGChecker works at claim level: claim recall is the share of ground-truth claims entailed by the retrieved chunks [19].

In my lab a chunk counts as the answer chunk only if it contains the whole answer fact. With fixed-size chunks some facts are cut in half, so for those questions no chunk can be relevant and the miss is guaranteed. Relevance definition and chunker interact, and the metric reflects both.

Building the labelled question set

Questions come from three sources, in decreasing order of realism: real user queries from logs or tickets, subject-matter experts, and LLM-generated (synthetic) questions.

A useful record holds the question, the relevant span(s) or claims, optional graded relevance, a short reference answer, and tags such as lexical / paraphrased, single-hop / multi-hop, has-identifier and unanswerable. Tags let you see that a change helped one slice and hurt another. Label spans or claims rather than chunk IDs if you will ever compare chunkers: chunk-ID labels silently become wrong when you re-chunk.

Synthetic question sets: useful, but biased in known ways

Generation is now a standard feature. Ragas builds a knowledge graph from your documents, then synthesizers produce single-hop or multi-hop, specific or abstract queries with configurable length and style [9]. Langfuse documents routes via Ragas, DeepEval's synthesizer, or a custom loop writing one question per chunk [17]. Chroma's chunking study shows careful filtering: queries whose evidence excerpts were not exact matches in the corpus were rejected, and near-duplicate or weakly related queries were removed with embedding thresholds [18].

The research on trusting synthetic sets is nuanced:

• Synthetic queries do not look like real ones. A September 2026 study found authentic queries averaged 6.8 words versus 15.7 for synthetic ones, real users concentrated on a few topics while synthetic questions spread evenly, and optimising on synthetic queries picked a hybrid retriever costing up to 8 times the latency of the fastest configuration, whose sparse half did not help short real queries [29].
• Synthetic collections inflate absolute scores and favour their own model family. Rahmani et al. found GPT-4 more lenient than human assessors on borderline passages, scores generally higher on synthetic data, and systems from the generator's model family favoured; the bias matters most for absolute scores and less for relative comparisons [27]. Their earlier SIGIR 2024 paper showed synthetic test collections can be used reliably for retrieval evaluation [26].
• Synthetic sets rank retrievers better than generators. Van Elburg et al. report that synthetic benchmarks "reliably rank the RAGs varying in terms of retriever configuration" but "do not consistently produce reliable RAG rankings when comparing generator architectures" [28].

Two more traps follow from how per-chunk generation works. A question written while looking at one chunk tends to reuse its vocabulary, which flatters lexical retrievers (measure this on your own set; see below). And it is answerable from exactly one chunk by construction, so multi-chunk and unanswerable questions are under-represented.

Practical stance: use synthetic questions for coverage and for comparing retriever configurations, keep a smaller human-written or log-derived set as the arbiter, and report them separately. If they disagree about the winner, believe the real one.

The metrics, precisely

Here are the definitions most tools use. Let rank be the 1-based position of a relevant item, k the cutoff, and R the set of relevant items for one question. Every metric is averaged over questions.

Notes on origins and variants:

• MRR comes from the TREC-8 QA track, where a question scored "the reciprocal of the rank at which the first correct response was returned," or 0 if none of the top five was correct [3].
• nDCG was introduced by Järvelin and Kekäläinen for graded relevance with a rank discount [4]. With one binary relevant item it reduces to 1 / log2(rank + 1): 1.00 at rank 1, 0.63 at rank 2, 0.50 at rank 3. Haystack's nDCG evaluator uses ground-truth scores as graded relevance when present and binary relevance otherwise [15].
• "Recall" is overloaded. Haystack's DocumentRecallEvaluator has single_hit (score 1 if any ground-truth document is retrieved, i.e. hit rate) and multi_hit (full score only if all are) [14]. Say which one you mean.
• Context precision is not one formula. In Ragas, the LLM-based version averages precision@k over the positions of chunks judged relevant, so ranking matters; the ID-based version is plain precision without rank weighting [5]. DeepEval's contextual precision also weights by position, judges against expected_output, and warns that scores from different evaluation modes are not comparable [10].
• Context recall (Ragas, DeepEval) splits the reference answer into claims and checks which can be attributed to the retrieved context [6][11]; Ragas also has non-LLM and ID-based variants. DeepEval's contextual relevancy is reference-free: the share of statements in the retrieved context relevant to the input [12].

When each metric misleads

Hit rate@k with a large k hides ranking. Hit@20 can stay flat while the relevant chunk slides from rank 1 to rank 15. That matters if the generator sees only the top 5, or handles long contexts unevenly: "Lost in the Middle" found performance "is often highest when relevant information occurs at the beginning or end of the input context" [38]. Chroma, by contrast, argues LLMs are relatively insensitive to position and down-weights nDCG-style metrics for RAG [18]. Both can hold for different models and context lengths, so report a rank-sensitive metric next to hit rate instead of picking one.

MRR ignores everything after the first relevant item. For a question needing three facts from three chunks, MRR rewards one at rank 1 and is blind to the other two. Use recall@k or claim-level context recall for multi-fact questions.

Precision@k is capped by the number of relevant items. With one relevant chunk, the best precision@5 is 0.20. Low precision matters when the extra chunks are distractors that pull the generator off course [2].

Incomplete labels punish novel retrievers. Unlabelled items count as irrelevant. BEIR found many datasets "subject to a lexical bias" from pooling with lexical systems; after missing TREC-COVID judgments were annotated, a dense retriever's nDCG@10 rose from 0.654 to 0.735, moving above BM25 [20]. Labels made by looking at BM25 results make new embedding models look worse.

Chunk-level metrics are not comparable across chunkers. Larger chunks contain more facts, so they win hit rate almost by definition while wasting context. Chroma's token-level IoU exposes this: a recursive splitter at 800 tokens with 400 overlap (the chunking settings OpenAI Assistants used by default at the time of the study) scored an IoU of 1.5, against 6.9 for the same splitter at 200 tokens with no overlap, and reducing overlap improved IoU because overlap adds redundant tokens [18].

The same metric name gives different numbers in different tools. Rank-weighted versus unweighted context precision, single_hit versus multi_hit recall, LLM-judged versus ID-based: pin the tool, version and variant in your eval report.

How the metrics behave on a toy set: my lab results

The experiments in applied-llm-lab build a tiny pipeline from scratch:

• Day 01, chunking: fixed-size chunks cut sentences in half; sentence packing kept every sentence whole at a similar chunk size.
• Day 02, BM25 from scratch: on 5 questions, sentence-packed chunks put the whole fact in BM25's top result 5/5 times, versus 2/5 for fixed-size chunks (4/5 with overlap).
• Day 03, TF-IDF cosine with NumPy: the same hit counts as BM25 (5/5 with sentence chunks). On 3 paraphrased questions both scored 0/3, so paraphrases need learned embeddings.

Day 04 added metrics: a 6-paragraph corpus about RAG pipelines and 12 questions, each asked in the document's own words ("original") and mostly in other words ("paraphrased"). The answer chunk is the one containing the whole fact. The harness core, in the same style:

def first_relevant_rank(ranked_chunks, fact, k=5): """1-based rank of the first top-k chunk that holds the whole fact, or None.""" for rank, chunk in enumerate(ranked_chunks[:k], start=1): if fact in " ".join(chunk.split()): # normalise whitespace before matching  return rank return None def summarize(ranks): """Average hit@1, hit@3 and reciprocal rank (MRR@5) over all questions.""" n = len(ranks) return { "hit@1": sum(r == 1 for r in ranks) / n, "hit@3": sum(r is not None and r <= 3 for r in ranks) / n, "mrr@5": sum(1 / r for r in ranks if r) / n, } assert summarize([1, 2, None]) == {"hit@1": 1 / 3, "hit@3": 2 / 3, "mrr@5": 0.5} 

BM25 and TF-IDF cosine gave identical numbers in every row, so the table shows one value per row:

What the toy set shows about the metrics:

1. Hit@3 alone would call overlap a clear win on paraphrases (0.58 to 0.67). But hit@1 dropped from 0.50 to 0.42 and MRR barely moved (0.53 to 0.54). In each of the three rank-2 cases a neighbouring overlap chunk ranked first, and in two of them it held only part of the fact. Overlap creates near-duplicates that compete with the real answer chunk: "hit rate hides ranking" in miniature.
2. Paraphrasing costs every configuration. The best chunker still lost 2 of 12. Per-question ranks from rerunning the script show both misses hinge on an abbreviation the text spells differently: "retrieval-augmented generation" versus "RAG", and "RRF" versus "reciprocal rank fusion". No shared terms, no lexical match.
3. Two lexical scorers with different maths ranked identically. On a tiny corpus with short chunks, the differences between BM25 and TF-IDF cosine do not show up.

Then the most important caveat.

Statistical noise: one question in 12 is 0.08

With 12 questions, each question moves a score by 1/12 = 0.083. The gap between 0.42 and 0.50 hit@1 above is one question.

Confidence intervals make this concrete. Using the Wilson score interval for a proportion:

The intervals for 8/12 and 12/12 overlap. Even the most dramatic result in my table, sentence packing beating fixed-size chunks 12/12 versus 8/12 on original questions, is not statistically convincing on its own: the four questions fixed-size missed were all hit by sentence packing, and none went the other way, which gives an exact two-sided McNemar (sign) test p-value of 2 × 0.5⁴ = 0.125.

Three habits fix most of this:

• Use paired tests on the same questions. Smucker, Allan and Carterette found "little practical difference between the randomization, bootstrap, and t tests" for IR, preferred the randomization test, and argued Wilcoxon and sign tests should be discontinued for differences between means [39]. Miller's paper on error bars for LLM evals recommends inference on question-level paired differences, clustered standard errors when questions come in related groups (say, several generated from one document), and power analysis [40].
• Size the set for the difference you care about. The 95% half-width for a proportion near 0.8 is roughly 1.96 × √(0.16 / n): about ±0.08 at n = 100 and ±0.04 at n = 400. A few dozen questions cannot detect a 3-point improvement.
• Report counts, not only ratios. "10/12" tells the reader what "0.83" hides.

A paired randomization test needs only the standard library:

import random def paired_randomization_test(a, b, trials=10_000, seed=0): """Two-sided p-value for mean(a) - mean(b), flipping signs of per-question differences.""" diffs = [x - y for x, y in zip(a, b)] observed = abs(sum(diffs)) rng = random.Random(seed) extreme = sum( abs(sum(d if rng.random() < 0.5 else -d for d in diffs)) >= observed for _ in range(trials) ) return (extreme + 1) / (trials + 1) 

Run on the per-question hit@1 scores of the sentence-packing and fixed-size rows above, it returns about 0.12, matching the exact test. If you prefer a library, ranx provides paired Student's t-test, Fisher's randomization test and Tukey's HSD alongside its metrics [42].

Lexical versus paraphrased questions

Paraphrases test the gap between how documents are written and how users ask. Lexical retrievers win on exact identifiers; BEIR found BM25 "a robust baseline" that dense models often failed to beat zero-shot [20]. Dense retrievers handle different wording but have their own limits: Weller et al. prove the number of top-k subsets a single-vector embedding can return is bounded by its dimension, and built LIMIT, on which state-of-the-art embedders fail despite very simple queries [25]. Reasoning-heavy queries break both: on BRIGHT's 1,384 real queries, a model scoring 59.0 nDCG@10 on BEIR scored 18.3, and explicit reasoning about the query added up to 12.2 points [23].

For your own set:

• Stratify into lexical (shares key terms with the answer span), paraphrased, identifier lookups and multi-hop or reasoning questions, and report every metric per stratum.
• Measure lexical overlap automatically (share of question tokens found in the answer span) and compare the synthetic and real splits. A high-overlap set makes BM25 look better than it will be in production.
• Write paraphrases deliberately, including abbreviations, synonyms and user jargon. My two universal misses were both abbreviation mismatches.

Evaluating chunking, hybrid search and rerankers

Chunking. Use span or claim labels so one question set fits every chunker. Report hit@k with a rank-sensitive metric and a cost measure (tokens sent to the generator, or token-level IoU [18]); otherwise bigger chunks always win. If you can, fix the context budget in tokens, not in number of chunks.

Hybrid search. Reciprocal rank fusion scores each document as Σ 1 / (k + rank) over the input rankings, with k = 60 fixed in a pilot study where "the choice was not critical" [41]. Fusion helps only if the inputs make different mistakes; my BM25 and TF-IDF ranked identically, so fusing them would add nothing. On FreshStack, a 2025 benchmark built from Stack Overflow questions and technical documentation, fusing four retrievers beat each one on most metrics and topics [24]. Report latency next to quality: the sparse half of a hybrid can help long synthetic questions but not short real ones, at several times the latency [29].

Rerankers. Evaluate the stages separately: the first stage needs high recall at the candidate depth you rerank (Anthropic reranked 150 candidates down to 20 [1]), the reranker needs good nDCG@k or MRR@k at the depth the generator sees. Rerankers do not always help: on FreshStack one improved some topics and lowered scores for some dense and fusion runs on others [24].

LLM-as-judge for retrieval

Labelling relevance by hand does not scale, so LLM judges are now everywhere: they power LLM-based context precision and recall, and they label pools for benchmarks.

The case for them is real. Thomas et al. found LLMs as accurate as human labellers in their experiments, and LLM labelling in production at Bing proved "not just more efficient but more accurate than the status quo" [30]. UMBRELA reproduced this with GPT-4o on TREC Deep Learning 2019 to 2023 and supported assessments in the TREC 2024 RAG track [31]. The biases are real too:

Alaofi et al. also found that agreement scores such as Cohen's kappa did not reliably predict how gullible a judge was [32], so "kappa looks fine" is not enough. Clarke and Dietz argue LLM-based assessment still cannot replace human assessment, especially for benchmarks meant to guide future research [33].

A practical middle ground is what ARES formalised: an automatic judge at scale, corrected by a small human-labelled set through prediction-powered inference, using "only a few hundred human annotations" [36].

Retrieval-only versus end-to-end metrics

• Faithfulness (Ragas): share of answer claims inferable from the retrieved context [7]. No reference needed, but it measures grounding, not correctness: a faithful answer from the wrong passage is still wrong.
• Answer correctness (Ragas): an F1-style claim overlap with the reference answer, combined with semantic similarity [8].
• Nugget evaluation (TREC 2024 RAG track): answers scored by the atomic facts they cover. Automated nugget evaluation agrees strongly with human-based variants at run level, but per-topic agreement, which you need to diagnose individual failures, is not yet established [37].

Read both kinds together:

The "bad retrieval, good answer" cell is why end-to-end metrics alone are not enough: on public-knowledge questions the generator can mask a broken retriever, which stops working the moment questions concern your private documents.

Tools and benchmarks in late 2026

Public benchmarks are for shortlisting embedding models and rerankers. RTEB exists precisely because public scores can overstate real-world performance through train-test overlap [22]. The decision should come from your own labelled set.

A practical evaluation recipe

1. Define relevance as spans or claims, not chunk IDs, and write it down.
2. Start with real or expert-written questions, labelled with spans, a reference answer and stratum tags, including some unanswerable ones. Size it with the interval arithmetic above: around 100 detects only large differences.
3. Add synthetic questions for coverage with filtering (exact-match evidence, deduplication, relevance threshold) in a separate split, and compare its length and lexical overlap with the real split.
4. Freeze the setup: corpus snapshot, question-set version, tool versions, judge model and prompt.
5. Report a small panel per stratum: hit@k at the generator's k, MRR@k or nDCG@k, recall@N at the reranker's candidate depth, plus tokens in context and latency.
6. Report counts and 95% intervals; compare configurations with a paired randomization or bootstrap test.
7. Change one thing at a time: chunker, retriever, fusion, reranker. Evaluate first stage and reranker separately.
8. Validate any LLM judge against human labels and adversarial passages, using a different model family from your generator.
9. Run end-to-end metrics too and read them with the 2×2 table above.
10. Read individual failures. On the toy set, the abbreviation pattern is visible only in per-question ranks, not in the averages.
11. Feed production queries back into the real split so the set follows your users, not your generator.

Checklist

• [ ] Relevance unit (span, claim, chunk, document) documented
• [ ] Real or expert questions are the decision set; synthetic questions a separate split
• [ ] Questions tagged: lexical / paraphrased / identifier / multi-hop / unanswerable
• [ ] Synthetic split checked for length, lexical overlap and topic coverage
• [ ] One presence metric (hit@k or recall@k) plus one rank metric (MRR or nDCG)
• [ ] k matches what the generator or reranker receives
• [ ] Metric variant and tool version pinned
• [ ] Per-stratum counts with 95% intervals; paired tests for comparisons
• [ ] Context-cost metric for chunking comparisons
• [ ] LLM judge validated on human labels and adversarial passages
• [ ] Individual failures reviewed, not just averages

What I'm testing next

Next in the lab: hybrid search with reciprocal rank fusion, measured with the same Day 04 harness, and a learned-embedding retriever on the paraphrased set. Because BM25 and TF-IDF cosine ranked identically on this corpus, fusing those two alone is unlikely to change anything; the paraphrased questions, including the two abbreviation misses, are where a learned-embedding retriever should make the difference. With only 12 questions, I will read any change there as a direction, not a result.

If you evaluate retrieval in production, I would like to know which metric panel you settled on and how many questions it took before the numbers stopped moving.

Sources

1. Anthropic, "Introducing Contextual Retrieval" (2024): https://www.anthropic.com/engineering/contextual-retrieval
2. Google Research, "Deeper insights into retrieval augmented generation: The role of sufficient context" (blog on Joren et al., ICLR 2025): https://research.google/blog/deeper-insights-into-retrieval-augmented-generation-the-role-of-sufficient-context/
3. Voorhees and Tice, "The TREC-8 Question Answering Track", LREC 2000: http://www.lrec-conf.org/proceedings/lrec2000/pdf/26.pdf
4. Järvelin and Kekäläinen, "Cumulated gain-based evaluation of IR techniques", ACM TOIS, 2002 (DOI 10.1145/582415.582418): https://trepo.tuni.fi/handle/10024/65718
5. Ragas docs, Context Precision: https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_precision/
6. Ragas docs, Context Recall: https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_recall/
7. Ragas docs, Faithfulness: https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/faithfulness/
8. Ragas docs, Answer Correctness: https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/answer_correctness/
9. Ragas docs, Testset Generation for RAG: https://docs.ragas.io/en/stable/concepts/test_data_generation/rag/
10. DeepEval docs, Contextual Precision: https://deepeval.com/docs/metrics-contextual-precision
11. DeepEval docs, Contextual Recall: https://www.deepeval.com/docs/metrics-contextual-recall
12. DeepEval docs, Contextual Relevancy: https://www.deepeval.com/docs/metrics-contextual-relevancy
13. Haystack docs, Evaluators: https://docs.haystack.deepset.ai/docs/evaluators
14. Haystack docs, DocumentRecallEvaluator: https://docs.haystack.deepset.ai/docs/documentrecallevaluator
15. Haystack docs, DocumentNDCGEvaluator: https://docs.haystack.deepset.ai/docs/documentndcgevaluator
16. Langfuse docs, Evaluation core concepts: https://langfuse.com/docs/evaluation/core-concepts
17. Langfuse docs, Synthetic datasets: https://langfuse.com/docs/datasets/example-synthetic-datasets
18. Smith and Troynikov, "Evaluating Chunking Strategies for Retrieval", Chroma Technical Report (2024): https://www.trychroma.com/research/evaluating-chunking
19. Ru et al., "RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation", NeurIPS 2024: https://arxiv.org/abs/2408.08067
20. Thakur et al., "BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models": https://arxiv.org/abs/2104.08663
21. Enevoldsen et al., "MMTEB: Massive Multilingual Text Embedding Benchmark", ICLR 2025: https://arxiv.org/abs/2502.13595
22. Hugging Face, "Introducing RTEB: A New Standard for Retrieval Evaluation" (Oct 2025): https://huggingface.co/blog/rteb
23. Su et al., "BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval", ICLR 2025: https://arxiv.org/abs/2407.12883
24. Thakur et al., "FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documents" (2025): https://arxiv.org/abs/2504.13128
25. Weller, Boratko, Naim and Lee, "On the Theoretical Limitations of Embedding-Based Retrieval" (ICLR 2026): https://arxiv.org/abs/2508.21038
26. Rahmani et al., "Synthetic Test Collections for Retrieval Evaluation", SIGIR 2024: https://arxiv.org/abs/2405.07767
27. Rahmani, Ramineni, Craswell, Mitra and Yilmaz, "Towards Understanding Bias in Synthetic Data for Evaluation" (2025): https://arxiv.org/abs/2506.10301
28. van Elburg, van der Putten and Marx, "Can we Evaluate RAGs with Synthetic Data?" (2025): https://arxiv.org/abs/2508.11758
29. Kucia and Gawlik, "Beyond Benchmark Scores: How Synthetic and Authentic Query Distributions Diverge in RAG Evaluation" (Sep 2026): https://arxiv.org/abs/2609.14579
30. Thomas, Spielman, Craswell and Mitra, "Large language models can accurately predict searcher preferences": https://arxiv.org/abs/2309.10621
31. Upadhyay et al., "UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor" (2024): https://arxiv.org/abs/2406.06519
32. Alaofi, Thomas, Scholer and Sanderson, "LLMs can be Fooled into Labelling a Document as Relevant", SIGIR-AP 2024: https://arxiv.org/abs/2501.17969
33. Clarke and Dietz, "LLM-based relevance assessment still can't replace human relevance assessment": https://arxiv.org/abs/2412.17156
34. Yu, Li, Zuccon, Mackenzie and Leelanupab, "When LLM Judges Inflate Scores: Exploring Overrating in Relevance Assessment" (2026): https://arxiv.org/abs/2602.17170
35. Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena", NeurIPS 2023: https://arxiv.org/abs/2306.05685
36. Saad-Falcon et al., "ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems": https://arxiv.org/abs/2311.09476
37. Pradeep et al., "The Great Nugget Recall: Automating Fact Extraction and RAG Evaluation with Large Language Models" (2025): https://arxiv.org/abs/2504.15068
38. Liu et al., "Lost in the Middle: How Language Models Use Long Contexts", TACL 2024: https://aclanthology.org/2024.tacl-1.9/
39. Smucker, Allan and Carterette, "A Comparison of Statistical Significance Tests for Information Retrieval Evaluation", CIKM 2007: https://ciir-publications.cs.umass.edu/getpdf.php?id=744
40. Miller, "Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations" (2024): https://arxiv.org/abs/2411.00640
41. Cormack, Clarke and Buettcher, "Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods", SIGIR 2009: https://cormack.uwaterloo.ca/cormacksigir09-rrf.pdf
42. ranx on PyPI: https://pypi.org/project/ranx/
43. Muhammad Hussnain Jamil, applied-llm-lab (Days 01 to 04): https://github.com/Hussnainoffical/applied-llm-lab