How does a search box find one word in a million pages in the blink of an eye? It certainly isn't reading them all while you wait. I've been coding for 30+ years and I'd never actually built one, so I did.

PySearch is a small full-text search engine in Python, with every idea explained in plain English. It has no dependencies and it's nowhere near Elasticsearch, but you can read the whole thing.

pip install fromscratch-pysearch

The librarian who read every book

The docs open with a picture I love. Imagine a librarian who has read every book in the library. Ask her "which books mention dinosaurs?" and she doesn't open a single one, because she kept notes while she was reading.

A search engine works the same way: it does the hard work before you ask. Here's a small library:

from pysearch.engine import SearchEngine engine = SearchEngine() engine.add("trex", "The T. rex was a huge dinosaur with tiny arms and huge teeth") engine.add("volcano", "A volcano erupts hot lava and ash") engine.add("fossils", "Fossils show us how a dinosaur lived. Dinosaur bones turn to stone") engine.add("birds", "Birds are living dinosaurs that fly and sing") engine.add("lava", "Lava cools and hardens into new rock") for query in ["dinosaur", "lava", "dinosaur AND bones", "volcano OR fossils"]: print(query) for doc_id, score in engine.search(query): print(f" {doc_id:8} {score:.3f}") 
dinosaur fossils 1.022 birds 0.511 trex 0.511 lava lava 0.916 volcano 0.916 dinosaur AND bones fossils 2.631 volcano OR fossils fossils 1.609 volcano 1.609

Behind that, the text was chopped into words, the words were filed in a notebook, and the matches were ranked. Each step is its own short Python file.

Cutting sentences into word cards

First, the tokenizer. Picture the sentence written on a long strip of paper. You cut it into word cards, lowercase each one and bin the punctuation. Then you bin the boring cards too: "the", "a", "is", "and" and friends are stop words, so common they tell you nothing.

>>> from pysearch.tokenizer import tokenize >>> tokenize("The cat is on the mat") ['cat', 'mat']

Finally, a little stemmer trims each card to its root, so "dinosaurs" and "dinosaur" count as the same word. That's why the birds page turned up above even though it only says "dinosaurs".

The stemmer is deliberately simple and just snips common endings. Here's the fossils page as word cards:

>>> tokenize("Fossils show us how a dinosaur lived. Dinosaur bones turn to stone") ['fossil', 'show', 'us', 'how', 'dinosaur', 'liv', 'dinosaur', 'bon', 'turn', 'stone']

"liv" and "bon" look odd, but only the computer reads these cards. As long as "bones" in a page and "bones" in a question get trimmed the same way, they still match.

The librarian's notebook

Now the clever bit. A normal index goes from a book to its words, like a table of contents. An inverted index flips it round and goes from a word to the books that contain it:

"dinosaur" -> {fossils: 2, birds: 1, trex: 1} "lava" -> {volcano: 1, lava: 1}

In PySearch that's literally a dictionary of dictionaries: word, then document, then how many times the word appears. Finding a word is one dictionary lookup instead of a trip through every page.

>>> from pysearch.index import InvertedIndex >>> index = InvertedIndex() >>> index.add_document("trex", "The T. rex was a huge dinosaur with tiny arms and huge teeth") >>> index.add_document("fossils", "Fossils show us how a dinosaur lived. Dinosaur bones turn to stone") >>> index.add_document("birds", "Birds are living dinosaurs that fly and sing") >>> sorted(index.search("dinosaurs")) ['birds', 'fossils', 'trex'] >>> index.get_term_frequency("dinosaur", "fossils"), index.get_document_frequency("dinosaur") (2, 3)

How much does the notebook actually help? I made 100,000 fake pages of 50 random words each, hid the word "stegosaurus" on one of them, and timed it on a modest two-core machine running Python 3.14 (your numbers will differ):

indexed 100,000 pages in 7.9 s index lookup: [('page42', 11.512925464970229)] in 0.110 ms reading every page: ['page42'] in 223.7 ms

Filing everything took a few seconds, once. After that the lookup came back in about a tenth of a millisecond, against roughly two tenths of a second to read every page, and the read-every-page loop was even cheating by skipping the stemming.

Which book comes first? A report card

Finding matches is half the job. If ten books mention dinosaurs, which should you read first? PySearch uses TF-IDF, which multiplies two questions together.

TF (term frequency) asks how often the word appears in this book. The fossils page says "dinosaur" twice, so it beats the pages that say it once. IDF (inverse document frequency) asks how rare the word is across all the books. It's log(N / df): five books, three mention dinosaurs, so log(5 / 3) = 0.511. "Lava" is in only two of the five, so it's worth more: log(5 / 2) = 0.916.

When you search for several words, the docs describe it like a school report card: each word is a subject, each gets a grade, and the grades are added up. That's why "dinosaur AND bones" gave the fossils page 2.631: 1.022 for dinosaur, plus 1.609 for bones, a word that appears in only one book.

The docs are upfront about the sharp edge here. A word that's in every book scores an IDF of exactly zero, which zeroes the whole score. A library with one book shows it:

>>> solo = SearchEngine() >>> solo.add("only", "Dinosaurs roar") >>> solo.search("dinosaur") [('only', 0.0)]

It still finds the book, it just can't tell you it's relevant. Smoothing the formula to log(N / df) + 1 is left as an exercise.

Understanding your question

The query parser turns what you typed into a little tree. Words side by side mean "all of them", and longer chains read left to right:

>>> from pysearch.query import parse_query >>> parse_query("cat AND dog OR fish") OrQuery(left=AndQuery(left=TermQuery(term='cat'), right=TermQuery(term='dog')), right=TermQuery(term='fish'))

Query words get exactly the same clean-up as the books did, so fox, searches for fox, and state-of-the-art becomes state AND art. If nothing searchable is left, or an operator has nothing on one side, you get a QueryParseError instead of a guess.

There's also NOT, which works like sorting a pile of books: grab everything that matches the left side, then put back any book that mentions the word on the right. Using the same little library:

>>> for doc_id, score in engine.search("dinosaur NOT bones"): ... print(f" {doc_id:8} {score:.3f}") birds 0.511 trex 0.511

The fossils page, which topped the plain "dinosaur" search, is gone because it mentions bones. The scores didn't change, because the word after NOT only filters and never adds to a score. (A book can't be a better match for a word it doesn't contain.)

Closing the library for the night

Everything so far lives in memory, which vanishes when the program stops. So the librarian writes every book into a notebook, a JSON file, and files them all again in the morning:

from pysearch.engine import SearchEngine from pysearch.persistence import load_engine, save_engine engine = SearchEngine() engine.add("cats", "Cats purr and nap in the sun") save_engine(engine, "library.json") tomorrow = load_engine("library.json") print(tomorrow.search("purr")) # [('cats', 0.0)] -- one book, so that IDF of zero again

It saves the original text rather than the index, plus a format number so a future version refuses an old file instead of misreading it. Saving is done like a careful student copying out homework: write on a fresh sheet, wait for the ink to dry (fsync), then swap it in for the old one in a single step. A crash halfway through leaves yesterday's notebook untouched.

What real search engines do that this doesn't

PySearch keeps the simple version of nearly everything, and you notice some of it quickly when you play with it.

Real engines use a proper stemmer (Porter, Snowball) or a dictionary. PySearch just snips endings, so "running" becomes runn and "runs" becomes run, and they don't match each other. "ran" doesn't match either. The notebook also stores counts rather than positions, so there's no "tiny arms" phrase search and no bonus for words that sit close together.

Ranking is simpler too. Real engines use BM25, which stops a word mentioned 50 times from beating one mentioned 5 times by tenfold, and allows for document length. Google adds links, freshness and much more. And at real scale the posting lists get compressed and sharded across many machines instead of living in one Python dictionary, with a crawler feeding it pages. PySearch starts from text you hand it.

Any of those would make a good weekend exercise. I was surprised how little the core needs: word cards, a notebook from words to books, and a report card for ranking.

Try it

PySearch is part of From scratch in Python, a series of 18 projects that rebuild real systems: a database, an operating system, git, a tiny GPT and more. Get the whole series with pip install fromscratch-pystack, or start at https://robcox2910.github.io/

How this was made: I designed PySearch and directed every part of it, but most of the code was written with Claude Code, under strict test-driven development, and this article was drafted with Claude's help too. Every output shown above comes from actually running it.