TL;DR: Give every marketplace page snapshot a stable version, derive each chunk ID from that version plus its position, and make the bulk write an upsert. This is the least complex design that stops retries from increasing vector count. Log runId, sourceId, sourceVersion, chunkId, and the write outcome together. Then a repeated search hit becomes a traceable ingestion event instead of a mysterious retrieval problem.
Start with the write path. Re-ranking or filtering can hide duplicate chunks, but neither reduces index growth. For a page-change watcher, that distinction matters: a retry should cost another request, not another permanent vector.
Retries happen.
| Strategy | Pick this when | Retry behavior | Index-cost effect | Main limit |
|---|---|---|---|---|
| Random vector IDs | Almost never for replayable ingestion | Inserts another row | Cost grows with every replay | No stable identity to inspect |
| Deterministic chunk IDs | The same snapshot may be retried | Overwrites the same logical row | Stable for identical input | Requires stable normalization and chunking |
| Delete then insert | A new page version must replace all old chunks | Safe only with careful ordering | Removes stale versions | A crash between phases can create a gap |
| Query-time deduplication | You need temporary containment | Hides repeated hits | Duplicate vectors remain indexed | Masks the ingestion defect |
How should you debug duplicate chunks in vector search results?
First, classify the repetition. Identical chunkId values in one response point toward a query or response-assembly issue, because a key-addressed index should expose one current record per ID. Different IDs with identical text point toward unstable identity during ingestion. Similar overlapping text with different offsets may be legitimate chunker output. Old and new prices from one listing indicate version-retention policy, not byte-for-byte duplication.
Do not collapse those cases into one duplicate_results counter. Track repeated IDs, repeated content digests, overlap pairs, and multiple active source versions separately. The labels lead to different owners and fixes.
IDs are evidence.
Pick deterministic IDs when retries, queue redelivery, or manual backfills can replay the same snapshot. This is the default for bulk ingestion because identity stays independent of a particular run.
Pick delete then insert when search must contain only the latest marketplace page and the storage layer cannot atomically switch an active version. Write the new version first, verify its expected chunk count, switch the active version, and clean up the old one. The active-version filter prevents mixed prices during the transition.
Pick query-time deduplication only as containment while repairing the writer. It can protect the alert channel from repeated notifications, but it leaves index size and write amplification untouched. Short-lived guardrail.
The trade-off is direct. Deterministic positional IDs are a poor fit when editors frequently insert text near the top of a long page, because that edit can renumber every later chunk and create a large replacement set. Content-defined chunk IDs reduce those rewrites, but identical paragraphs then need an occurrence number or structural path. Delete then insert is also not suitable when readers cannot tolerate a temporary empty result set; use versioned writes plus an active-version switch in that case. Query-time deduplication has the opposite limitation: it can stabilize alerts quickly, yet every hidden copy still consumes index capacity. Choose according to the failure you can observe, not according to which technique sounds strongest.
Make identity observable before changing the index
A diagram in words is: scheduler -> fetch -> normalize -> version digest -> chunk -> chunk ID -> embed -> upsert -> verify. Put one trace span around the run and a child span around each bulk write. The run ID answers "which attempt?" The source version answers "which page state?" The chunk ID answers "which logical record?" Mixing those identities is the design error to avoid.
Keep them separate.
Use low-cardinality metrics for trends: attempted records, inserted records, updated records, rejected records, and active vectors. Keep individual IDs in logs and traces, where high-cardinality values belong. One alert can compare attempted with inserted plus updated plus rejected; another can flag a nonzero increase in active vectors after replaying an unchanged source version.
For each batch, emit one structured summary instead of one success log per chunk. Include normalizer and chunker revisions because a deployment can change boundaries even when page text does not.
type BatchSummary = { runId: string; sourceId: string; sourceVersion: string; normalizerRevision: string; chunkerRevision: string; attempted: number; inserted: number; updated: number; rejected: number; }; function reportBatch(summary: BatchSummary): void { console.info(JSON.stringify({ event: "vector_ingest_complete", ...summary })); } Do not put raw marketplace page text in that event. Besides inflating logs, it may contain seller or buyer data. A digest gives correlation without copying the document into telemetry.
Build an idempotent bulk writer in TypeScript
The version must describe normalized source state, while the chunk ID describes one position under that version. SHA-256 supplies a deterministic digest: the exact same input bytes produce the same result. Node's createHash accepts strings and can return a hexadecimal digest.
The sample uses 900-character windows and explicit v1 revision labels. Those numbers are examples of a reproducible contract, not universal tuning advice; changing either value changes the identity boundary and should be deployed as a deliberate migration.
Canonical input is the hard part. Normalize line endings, apply an explicit whitespace policy, and version that policy. Avoid hashing ordinary serialized objects when property ordering or omitted values can vary between producers. RFC 8785 defines a JSON canonicalization scheme when structured JSON itself must be hashed.
import { createHash, randomUUID } from "node:crypto"; type Snapshot = { marketplace: string; listingId: string; fetchedText: string }; type Record = { id: string; values: number[]; metadata: { sourceId: string; sourceVersion: string; chunkIndex: number; text: string }; }; type WriteResult = { inserted: number; updated: number; rejected: number }; interface VectorIndex { upsert(records: Record[]): Promise<WriteResult>; } interface Embedder { embed(texts: string[]): Promise<number[][]>; } const digest = (value: string): string => createHash("sha256").update(value, "utf8").digest("hex"); function normalize(text: string): string { return text.replace(/\r\n?/g, "\n").replace(/[ \t]+/g, " ").trim(); } function splitByCharacters(text: string, size = 900): string[] { if (!Number.isInteger(size) || size <= 0) throw new RangeError("size must be positive"); const chunks: string[] = []; for (let start = 0; start < text.length; start += size) { chunks.push(text.slice(start, start + size)); } return chunks; } async function ingest( snapshot: Snapshot, index: VectorIndex, embedder: Embedder, ): Promise<BatchSummary> { const runId = randomUUID(); const sourceId = snapshot.marketplace + ":" + snapshot.listingId; const normalized = normalize(snapshot.fetchedText); const sourceVersion = digest("normalizer-v1\0" + normalized); const chunks = splitByCharacters(normalized); const values = await embedder.embed(chunks); if (values.length !== chunks.length) throw new Error("embedding count mismatch"); const records = chunks.map((text, chunkIndex): Record => ({ id: digest([sourceId, sourceVersion, "chunker-char-v1", chunkIndex].join("\0")), values: values[chunkIndex], metadata: { sourceId, sourceVersion, chunkIndex, text }, })); const result = await index.upsert(records); const summary: BatchSummary = { runId, sourceId, sourceVersion, normalizerRevision: "v1", chunkerRevision: "char-v1", attempted: records.length, ...result, }; reportBatch(summary); return summary; } The NUL separators prevent ambiguous concatenations such as ab plus c and a plus bc. Including sourceId keeps identical boilerplate on two listings from becoming one logical record. Including sourceVersion lets a changed page coexist briefly with its predecessor during a controlled switch. The chunker revision forces deliberate new identity when boundaries change.
This example uses fixed character windows to keep the identity mechanics visible. Production chunking may use headings, tokens, or semantic boundaries, but output ordering must be deterministic for the same version. If chunk positions shift after one sentence is inserted, every later positional ID changes. Content-defined chunks can reduce that rewrite amplification, though repeated identical paragraphs then need an occurrence discriminator. There is no free identity scheme.
That is the limit.
Prove a retry is cheap before deployment
The strongest test runs the same fixture twice against a test double implementing key-based upsert. The second pass should report zero inserts, the same version, and no growth in stored records. Then change one listing field and assert that a new source version appears. Do this before load testing; throughput can make a broken writer fail faster without making it correct.
import assert from "node:assert/strict"; const first = await ingest(snapshot, index, embedder); const countAfterFirst = await index.count(); const second = await ingest(snapshot, index, embedder); const countAfterSecond = await index.count(); assert.equal(first.sourceVersion, second.sourceVersion); assert.equal(second.inserted, 0); assert.equal(countAfterSecond, countAfterFirst); Also test partial failure. Split a large batch into bounded writes, record which groups completed, and replay the entire source version. Stable IDs make whole-version replay safe; a resume cursor becomes an optimization, not a correctness dependency. Cap concurrency so a retry storm cannot multiply embedding calls and writes without bound.
The deployment signal is crisp: replay one unchanged snapshot and watch updated records rise while active vector count stays flat. If active count rises, stop there. Compare source versions, chunker revisions, and IDs from both runs before touching retrieval parameters.
Limits and operating rules
Deterministic IDs do not decide freshness. A page watcher still needs an explicit rule for which source version is searchable and when an old version can be deleted. They also do not remove near-duplicates created by mirrored listings, templated descriptions, or meaningful overlap. Those require domain rules and sometimes similarity analysis.
Keep the operating rule small: identical snapshot, identical IDs, flat active count. Changed snapshot, new version, verified activation, then retirement of the prior version. Query-time suppression remains a safety net for alerts, never the accounting system for the index.
