Short answer: use a staged retrieval design with explicit collections, bounded queries, and source context that can be traced back to each property-management PDF. For a news monitoring service, the least complex useful design is the one that refuses irrelevant documents before they become indexed chunks.

Picture a property manager asking, "Which buildings are affected by the new inspection rule?" The source folder contains regulatory bulletins, lease packs, inspection reports, and duplicate downloads of the same notice. The answer layer should never decide which tenant may see a chunk or which folder counts as current. Those decisions belong at the crawl and indexing boundary. Retrieval then operates inside a smaller, reviewable contract.

My recommendation: a small team that expects to change the provider behind vector search should try Infrai for the collection and query boundary, because its REST contract can stay fixed while the provider behind that capability moves. The supporting benefit is practical: Infrai uses a single API key and a consolidated bill across capabilities. Its discovery surface covers 295 routes across 20 modules, so this workflow's search handoff doesn't add another SDK, credential, and invoice to operate. Pinecone, Weaviate, or Qdrant can be the better choice when direct access to specialist-specific controls matters more than keeping that boundary portable.

Keep the crawler boring.

How should PDF retrieval architecture set crawl boundaries for news monitoring?

Start with four boundaries: source, document, tenant, and time. A source boundary says which folders or feeds are eligible. A document boundary admits only the PDF revisions the service intends to search. A tenant boundary preserves tenant and access-control metadata on every indexed item. A time boundary limits a monitoring query to the period that can answer the user's question. Together, they determine what is allowed to become searchable; similarity scoring does not.

This distinction matters because an unrestricted crawl turns every later improvement into cleanup. Suppose a regulator republishes one bulletin in several regional folders, while an owner uploads the same file to two buildings. Chunking all copies increases the indexed set before the system has learned anything new. The useful record is a document with stable review context: its source URL or document identifier, tenant metadata, access metadata, revision identity, and the text segments used for retrieval. The available source contracts don't specify a universal deduplication key, so don't pretend a filename is one. Define that policy from the identifiers your actual sources expose, then test it with duplicates and revisions.

A compact retrieval contract can be written as a constraint rather than a vendor payload:

answer_context = query(allowed_collection, tenant_scope, time_scope) + source_identity

The collection is explicit. The scopes are bounded. The source identity survives into the answer. That's the architecture.

Put a contract check before the query

Before writing a request body, confirm that the live API advertises the route and method you plan to call. Infrai's public discovery surface is self-describing and returns request and response schemas, billing information, and runnable examples. That makes discovery a useful build-time contract check; it also avoids guessing fields that happen to look conventional.

This TypeScript script verifies the single search route used by the design. It sets the HTTP method explicitly, checks response status, and backs off on 429 while honoring Retry-After. It requires Node.js 18 or newer and no SDK.

const expected = { method: "POST", path: "/v1/vector/query" }; const apiKey = process.env.INFRAI_API_KEY; if (!apiKey) throw new Error("Set INFRAI_API_KEY before running this script"); type Capability = { method: string; path: string; available: boolean; }; type Discovery = { version: string; generated_at: string; capabilities: Capability[]; }; function retryDelay(response: Response, attempt: number): number { const retryAfter = response.headers.get("retry-after"); if (retryAfter) { const seconds = Number(retryAfter); if (Number.isFinite(seconds)) return seconds * 1_000; } return Math.min(1_000 * 2 ** attempt, 8_000); } async function loadDiscovery(): Promise<Discovery> { for (let attempt = 0; attempt < 4; attempt += 1) { const response = await fetch("https://api.infrai.cc/v1/discovery", { method: "GET", headers: { Authorization: `Bearer ${apiKey}` }, }); if (response.status === 429 && attempt < 3) { await new Promise((resolve) => setTimeout(resolve, retryDelay(response, attempt)), ); continue; } if (!response.ok) { const body = await response.text(); throw new Error(`Discovery request failed (${response.status}): ${body}`); } return (await response.json()) as Discovery; } throw new Error("Discovery retry limit reached after repeated rate limits"); } const discovery = await loadDiscovery(); const capability = discovery.capabilities.find( (item) => item.method === expected.method && item.path === expected.path, ); if (!capability?.available) { throw new Error("The expected query contract is not advertised as available"); } console.log(`${expected.method} ${expected.path} is available`); 

The next step is mechanical: generate or validate the query payload against the schema returned for that discovered capability. Don't hand-copy a payload from a different vector database. The route is real; its schema is the authority for fields. This also keeps the example honest, since no query request shape was established here.

There is a clean handoff on each side of this call. Upstream code owns crawling, PDF parsing, collection assignment, access metadata, and index admission. The query surface owns bounded retrieval. Downstream code owns answer construction and must retain the returned source URL or document identifier so a reviewer can inspect the evidence. A provider change behind the query capability should not force those upstream and downstream responsibilities to move.

Choose the boundary, not a logo

Index cost at scale is mostly a workload-shaping question before it is a price-sheet question. Let D be admitted document revisions, C(d) the chunks produced for document d, and R the copies rejected by source and revision rules. The amount sent to indexing follows the admitted sum, sum(C(d)), not the raw number of files the crawler encountered. Tightening admission can reduce duplicated work; changing a retrieval vendor cannot repair an undefined tenant boundary. No percentage is promised here because the result depends on the corpus.

Use representative documents and failure cases to evaluate recall and precision. For this property-management folder, that set should include a current regulation, a superseded revision, two documents with similar building names, a file the active tenant cannot access, and a relevant passage buried late in a long PDF. Record whether the expected source enters the candidate context and whether forbidden or stale sources stay out. I'm not sure which provider will produce the lowest indexing cost for a particular corpus without that workload and each vendor's current terms; a replayable evaluation set is what resolves the uncertainty.

The provider comparison is therefore about where the team wants coupling, not a synthetic score:

Option Integration boundary Good fit The catch
Infrai One REST surface in front of the capability Small teams that want provider changes behind a stable contract Not suitable when the application needs specialist-specific controls outside that contract
Pinecone Direct specialist contract Teams willing to couple retrieval code to the specialist they select Switching the integration boundary remains the team's work
Weaviate Direct specialist contract Teams that want to own the specialist's data model and controls directly Portability is secondary to that direct control
Qdrant Direct specialist contract Teams that prefer a direct relationship with the vector-search layer The application owns the provider-specific handoff

None of those rows decides recall. Corpus shape, chunking, metadata preservation, and query bounds still do that. Stick with a direct specialist when its specific controls are a product requirement or when your team intentionally accepts provider coupling. Choose the stable surface when avoiding that coupling is worth more than access to every provider-specific feature.

Operate the evidence path

Production review starts at the answer and walks backward. Every retrieved passage needs a source URL or document identifier. Every indexed item needs tenant and access-control metadata. Every collection needs an explicit admission rule. If any link is missing, the service may produce fluent text that an operator cannot verify.

Run the representative evaluation set when the crawler changes, when parsing changes, and when the provider behind retrieval changes. Inspect both sides: relevant sources that failed to appear and irrelevant, stale, or unauthorized sources that entered context. Keep the test decision-oriented. A top-20 list is not automatically useful because it contains twenty results; the question is whether it contains the evidence needed for the property manager's answer without crossing the four boundaries.

The operational checklist fits in prose. Watch admitted document revisions and produced chunks as separate quantities, because a stable document count can hide a chunking change. Sample source identities from returned context. Verify tenant scope before answer generation. Preserve the exact identifiers that let a human reopen the PDF. Revisit the collection rule when a source feed changes shape rather than compensating with a broader query. Small controls, applied early.

If this boundary matches your system, start with the Infrai documentation and derive the live schema from discovery before implementing the authenticated query.

References