Does semantic code search actually beat grep on a real repo?

The pitch for embedding a codebase is always the same demo: you ask "where do we lock users out?", grep finds nothing because the code calls it a cooldown, and vector search lands on the right function. It is a good demo. The question is how often it holds up on code nobody picked for the demo.

So here is a measurement on a repository most Python developers have read at some point, psf/requests at tag v2.32.3, with ten questions where grep for the obvious phrase returns zero lines. The pipeline copies the one the osFoundry desktop app uses for its local folder index (chunking rules from folder_index.rs, hybrid ranking from folder_search.rs), rebuilt in under 200 lines of Python and run on an 8 GB M1 Mac, CPU only. The short answer: embeddings alone found the right chunk in the top 5 for 4 of 10 questions. A plain keyword ranking of the same question found 6. The two fused together found 7 in the top 10, and 9 once tests and docs stopped competing with the source.

Chunking with line numbers

Embeddings work on pieces, not files. The app cuts each file into chunks of about 700 tokens with an 80-token overlap, approximating a token as 4 characters, and keeps the first and last line of every chunk so a hit can be reported as path:start-end.

TARGET_TOKENS, OVERLAP_TOKENS = 700, 80 def chunk_with_lines(text): lines = text.splitlines() target, overlap = TARGET_TOKENS * 4, OVERLAP_TOKENS * 4 # ~4 chars per token  out, start = [], 0 while start < len(lines): chars, end = 0, start while end < len(lines) and chars < target: chars += len(lines[end]) + 1 end += 1 body = "\n".join(lines[start:end]) if body.strip(): out.append((body, start + 1, end)) # 1-based, inclusive  if end >= len(lines): break back, idx = 0, end while idx > start and back < overlap: # step back ~80 tokens  idx -= 1 back += len(lines[idx]) + 1 start = idx if idx > start else end # always move forward  return out 

Chunks are cut on line boundaries, never mid-line, and the last guard matters: a minified file with one 200 KB line would otherwise step back to the same start forever.

The scan before it skips node_modules, dist, build, .git, .venv and similar folders, lockfiles by full name (package-lock.json and pnpm-lock.yaml don't end in .lock, which is how an extension-only filter misses them), files over 1 MB, and anything with a null byte in its first 512 bytes. On requests that left 117 of 129 files (10 binary, 1 over 1 MB, 1 excluded SVG), 625,261 bytes, and 318 chunks.

One thing the 4-characters rule hides: code tokenizes worse than prose. Measured with bge-m3's own tokenizer, the median chunk was 771 tokens, not 700, and the longest was 2,050. That matters for the next step.

Embedding locally

from sentence_transformers import SentenceTransformer model = SentenceTransformer("BAAI/bge-m3", device="cpu") model.max_seq_length = 2048 texts = [f"{c['path']}\n{c['text']}" for c in chunks] # the path carries meaning too vecs = model.encode(texts, batch_size=2, normalize_embeddings=True) 

The scripts run with uv run, which installs sentence-transformers and torch from the inline metadata at the top of each file. For comparison, the same chunks also went through bge-small-en-v1.5, a 33M-parameter English model about 17 times smaller (its queries get the model's recommended "Represent this sentence for searching relevant passages: " prefix).

Ten minutes for a 625 KB repository is slow, about 2 seconds per chunk on this machine, and the cost scales with the code, not the number of files. It is a one-time cost: the app keys chunks by a hash of their text, so an unchanged chunk keeps its vector, and editing one function re-embeds a handful of chunks. The small model's speed comes with a catch the table shows plainly: at 512 tokens it read only the first part of 235 of the 318 chunks.

There is one rule the app's source flags with a red marker, and it is easy to break in your own code. The query has to be embedded by the same model that built the index. A bge-m3 chunk vector and a query vector from another 1,024-dimensional model have the same shape, so nothing errors, and the results are simply wrong.

Keywords, vectors, and fusing them

grep was given the phrase you would type, case-insensitive, over the same 117 files. As a fairer keyword baseline, the whole question also went through BM25 over the chunks, with identifiers split on _ and camelCase so that should_strip_auth matches "strip". (The app's keyword side is tantivy over whole files, mapped onto each file's first chunks; ranking per chunk is a simplification.)

The two rankings are merged with Reciprocal Rank Fusion at k = 60, the same constant the app uses, taking the top 30 from each list:

def rrf(lists, k=60, n=30): s = Counter() for lst in lists: for r, i in enumerate(lst[:n]): s[i] += 1 / (k + r + 1) return [i for i, _ in s.most_common()] 

RRF ignores the scores and only uses positions, which is the point. A BM25 score and a cosine similarity are not on the same scale, and normalising one against the other invents a relationship that isn't there. A chunk ranked 3rd by one list and 5th by the other beats a chunk ranked 1st by one and absent from the other.

The answer for each question is the chunk containing the target function's def line, chosen by reading the code before running anything. The number reported is that chunk's rank among all 318.

Results

Ranks are out of 318 chunks; grep counts are matching lines. "Hybrid" is BM25 fused with bge-m3.

Three things stand out.

A phrase grep returning zero does not mean keywords are useless. Ranked over the question's words, BM25 put the answer in the top 5 for 6 of the 10. "Redirect", "proxy", "401" and "UTF-16" all appear in the code; only the phrasing around them doesn't. The single-word greps show the opposite problem: 625 lines for GET, 98 for bytes. They find it somewhere in a pile you then read by hand.

The embedding wins are real but specific. bge-m3 found should_strip_auth first for a question that shares no wording with it, and moved rebuild_method from 65th under BM25 to 10th. The small model put atomic_open first for "never left half-written if the process dies", where BM25 had it 58th; that function's docstring says "atomic fashion", and the model knows what atomic means for a file. And vectors fail in ways keywords don't. On the bare identifier raise_for_status, bge-m3 ranked the defining chunk 70th, and bge-small 244th.

Fusion is the safer default, not the best on every row. Apart from the one question nothing found, hybrid was never the worst column and usually close to the best one, which is what you want from a tool you call without knowing in advance which kind of question you have. "How big is the body" failed everywhere: super_len never says what it measures in words a question would use.

Tests and docs are the biggest competitor

Most of the misses were not random. For "why does a POST turn into a GET", bge-m3's first result was tests/test_requests.py, around test_http_303_changes_post_to_get. For "read saved logins from a file in the home folder" it was the changelog entry about .netrc in HISTORY.md, and for both "move the upload back to the start" and "how big is the body" a section of docs/user/advanced.rst. Tests and docs describe behaviour in plain words, which is exactly what a question is, so they are often closer to it than the implementation.

That is not a wrong answer, since the test names the function. But the ranking changes a lot once they are filtered out. Searching only the 84 chunks under src/ with the same vectors, hybrid search found 9 of 10 in the top 10. The cheapest improvement on this repo was not a bigger model but a path filter, or at least labelling test and doc hits so they can be skipped.

The smaller model kept up despite reading only the first 512 tokens of most chunks, probably because a chunk's opening lines, with the def and its docstring, carry most of what it is about. Ten questions on one repository is not enough to call either model better. It is enough to see that the choice of model mattered less than fusing with keywords and keeping tests out of the way.

Running it

git clone --depth 1 --branch v2.32.3 https://github.com/psf/requests.git python3 chunk.py requests chunks.json uv run embed.py chunks.json BAAI/bge-m3 vec-m3.npy uv run search.py chunks.json vec-m3.npy BAAI/bge-m3 queries.json SCOPE=src/ uv run search.py chunks.json vec-m3.npy BAAI/bge-m3 queries.json 

The first embed.py run downloads bge-m3 from Hugging Face, about 2.3 GB. Write your queries before looking at the results, and pick the answers by reading the code first. It is very easy to tune the wording of a question until the vector search wins.

The osFoundry desktop app does the same pipeline from its Indexes screen, with bge-m3 on the device and BM25 and vectors fused in its folder_search agent tool. The chunking entry in its glossary covers the trade-offs of chunk size and overlap, and the app is free from the osFoundry download page.