A support bot or internal assistant built on a documentation site goes stale the week after launch. The docs change, the vector index doesn't, and the bot keeps quoting the old pricing page. The usual fix is a nightly full re-crawl and re-embed, which pays to embed thousands of unchanged chunks to pick up a handful of edits, and still leaves deleted pages in the index unless you diff the URL lists yourself.

What the index actually needs from each sync is three lists: chunks to add, chunks to replace, chunk IDs to delete. This post gets exactly that from a scheduled run and applies it to a vector store in Python.

Disclosure: I'm Aurelix, and I built the Website Change Tracker. It's a paid tool (prices below).

How it decides what changed

You give it start URLs (or sitemap URLs) and a corpusId. The first run with that ID is the baseline: every page comes back as added, split into Markdown chunks. Every later run re-checks the site and returns only pages that were added, changed or deleted.

  • Chunk IDs are anchored on content: the page's canonical URL, the heading path and a hash of the normalised text, as valid UUIDs. Chunk boundaries are cut at content-defined points, so editing one paragraph re-cuts only the chunk it sits in, and an unchanged section keeps its ID when the sections around it change.
  • Boilerplate is not a change: navigation, footers, cookie banners and lines like "Last updated" or "x min read" are dropped before comparing.
  • A broken page is not a deletion: a page is deleted only on a 404/410 or a redirect away. A known page that errors, shows a bot challenge, comes back as an empty JavaScript shell or shrinks by more than 90% is returned as extraction_failed (free), and its previous version is kept.

Each change record carries change, doc_id, url, title, upsert_chunks (each with chunk_id, heading_path, text and a ready citation), delete_chunk_ids, and counts of upserted, deleted and unchanged chunks.

A nightly sync into a vector store

This uses Chroma, which embeds documents itself; the same three steps work with Qdrant, Pinecone, Weaviate or pgvector, since the chunk IDs are UUIDs.

pip install apify-client chromadb 
import os import chromadb from apify_client import ApifyClient client = ApifyClient(os.environ["APIFY_TOKEN"]) run = client.actor("aurelix/website-change-monitor").call( run_input={ "startUrls": ["https://docs.apify.com/academy"], "corpusId": "apify-academy", "maxPages": 500, "chunkSize": 1500, } ) changes = list(client.dataset(run["defaultDatasetId"]).iterate_items()) index = chromadb.PersistentClient(path="index").get_or_create_collection("docs") upserts, deletes = [], [] for c in changes: if c["change"] == "extraction_failed": print(f"skipped (kept previous version): {c['url']} ({c['reason']})") continue upserts += c["upsert_chunks"] or [] deletes += c["delete_chunk_ids"] or [] if deletes: index.delete(ids=deletes) for i in range(0, len(upserts), 100): batch = upserts[i:i + 100] index.upsert( ids=[ch["chunk_id"] for ch in batch], documents=[ch["text"] for ch in batch], metadatas=[{"url": ch["url"], "citation": ch["citation"]} for ch in batch], ) print(f"{len(changes)} pages changed: {len(upserts)} chunks upserted, {len(deletes)} deleted")

Run it once to fill the index, then schedule the same input daily (save it as a task in the Apify Console and click Schedule, then run the script after it, or call it from your own scheduler). A quiet day returns nothing and embeds nothing.

A few input choices that matter:

  • Already loaded the site another way? Set "returnBaseline": false and the first run records the corpus silently, so you pay per page checked, not per page returned. The Website to Markdown API is a good way to do that first full load.
  • Help centres with orphan pages: "scope": "domain" with "useSitemaps": true tracks pages no menu links to; excludeGlobs such as ["**/search**", "**/tags/**"] keeps out listings that change on every visit.
  • No polling: set webhookUrl to an n8n, Make or Zapier webhook and the delta is POSTed in batches of 25 records; the last batch has final: true and the run's counts.
  • Chunk size is fixed per corpus so IDs stay stable. To re-chunk, use a new corpusId, which starts a new baseline.

Why this tool

  • Built for a machine reading the output: Markdown chunks to upsert and chunk IDs to delete.
  • Chunk IDs are anchored to the content, so they stay put when nothing in that section changed.
  • Blocked or empty pages are never reported as deleted.
  • The delta, chunks and delete IDs, can be pushed to a webhook.
  • What it doesn't cover: severity scores or diffs written for a person to read.

Limits

  • No JavaScript rendering. Pages that are an empty app shell are skipped, and flagged if they were already in the corpus.
  • HTML pages only. PDFs, images and other files are skipped; HTML pages over 2 MB too.
  • maxPages caps pages checked per run (up to 20,000). Known pages not reached in a run are kept, not deleted; set it above the size of the site.
  • Polite pace: at most two new requests a second per site, about 1,000 large documentation pages an hour. robots.txt is always obeyed.
  • rel=canonical is followed within the same site and scope, so a site whose canonical tags all point to one page collapses to that page.
  • No login, paywalled or bot-challenged pages.

Cost

Two parts: a small charge per page checked, and a larger one per changed page returned. A run with no changes costs only the checks and the start fee. Extraction failures, robots.txt blocks, fetch errors and duplicates are free. The current prices are on the Store page; Apify's free plan includes $5 of credit a month, enough to check a 250-page docs site every day for a month when only a few pages change.

Links