We Gave Our Agent a Knowledge Base That Would Rather Say "I Don't Know" Than Guess
Short answer: an agent that answers questions from memory is an
agent that invents sources. So we built a knowledge engine where every
answerable fact is a versioned, human-approved item - and the
"answer" endpoint is a deterministic scorer that refuses below
threshold and files the missing answer as tracked work. No LLM in
the answer path, no vector database, no vibes.
This is part of the Arc campaign series. Previous published episode:
before our agent pays, it phones a friend
(EPIC-257 - counterparty trust for x402 payments). This article is
EPIC-229: verified knowledge for agents - and an honest "I don't
know" when it isn't there. The next episode, the agent interaction
inbox, is where agent questions enter -
its classifications file the gap records this engine stores.
The problem: an agent's answer is only as good as its sources
Agents on our platform already pay for API calls, sign receipts, and
request human review. The next question they ask is softer: "how does
this work?" - about our capabilities, our pricing, our settlement
rules. Today the honest answer is "search our docs." What they actually
do is guess from whatever an index returned and present the guess as
fact.
We wanted something narrower and more trustworthy: a knowledge base an
agent can query where every returned item has been written, reviewed,
and published by a human - and where "we don't have a verified answer
for that" is a first-class result, not an error.
What shipped: a boring, auditable knowledge engine
For a junior: think of it as an internal FAQ with a strict editor. A
knowledge item is a slug plus immutable, numbered versions of content.
It moves through an approval machine - DRAFT, PENDING, VERIFIED,
PUBLISHED, WITHDRAWN - and every move is a CAS-guarded transition
that appends an audit event. Illegal moves get an honest 409, never silent state.
INVALID_TRANSITION
Two properties matter more than the table schema:
- Only operators can approve. Operator refs pass a strict allowlist (
[a-z0-9._@-], max 80 chars) - agent-shaped forms (did:,agt_,agent,bot,svc,key:) and unicode lookalikes are refused outright. The workflow also enforces separation of duties from the persisted audit trail: the verifier must differ from the submitter, and the publisher must differ from the verifier. Agents read the finished product; they never mark anything verified. - Reads are pinned to published. The public search and ask routes hard-force
status: PUBLISHEDat the store seam. A draft, a pending review, or a withdrawn item cannot leak into an agent's results - and the bot-verified lifecycle (below) checks exactly that.
Versions are append-only: editing creates a new version row, and the
item's publishedVersionId pointer only moves through the workflow.
"Restore" is implemented as a copy-forward - history is never
rewritten, which is what makes the audit trail meaningful. Evidence is
version-scoped and freezes the moment an item is submitted for
review (DRAFT is the only editable state), so a verifier always
reviews exactly the set the submitter declared; a
version cannot even enter VERIFIED without at least one evidence
reference attached, and a version submission is re-checked against
the item's status inside the same row-locked transaction - no racing
a different draft past the reviewer.
Search without a vector database
GET /api/v1/agent/knowledge/search?q= runs plain Postgres full-text
search over published versions - a functional GIN index, ts_rank
ordering, ts_headline snippets. The query string binds as a
parameter into plainto_tsquery, so tsquery operators (&, |,
:*, quotes, <->) in ?q= are data, not syntax - injection dies at
the parameter boundary. The response reports its engine honestly:
"engine": "FTS" on Postgres, "MEMORY" on the fallback store that
implements the same interface for tests.
Why the "answer" endpoint refuses instead of generating
POST /api/v1/agent/knowledge/ask is where the design choice gets
sharp. The tempting architecture is retrieval + a small LLM to
"synthesize" the answer. We deliberately did not build that, for one
reason: the moment a model writes the answer, the answer is only as
trustworthy as the model - and we would be back to vibes.
Instead the ask endpoint runs a deterministic composer: it searches
published items, scores term coverage against the query (engine-
agnostic - Postgres ts_rank and the memory scorer live on different
scales, so the composer re-derives its own coverage score), and then
either:
- returns
ANSWEREDwith the assembled text, the source items (ids + slugs), related items, and a numeric confidence; or - returns
REFUSEDwithinsufficient_verified_knowledge.
The refusal is the feature. A refused ask creates a knowledge gap
record - normalized (trim/lowercase/collapsed whitespace) so
duplicates dedupe, with sample queries capped and a lifecycle of its
own. The gap is what a content team's backlog looks like when the
customer is a machine: every unanswered question becomes tracked work
instead of a hallucination.
Gaps, relations, evidence - the boring parts that make it trustworthy
Three supporting structures round the engine out:
- Knowledge gaps deduplicate on normalized text and move through their own lifecycle; a seam from EPIC-228's interaction inbox (
recordInteractionGap,sourceRef=ireq_*) files a gap automatically when an operator classifies an agent's request as a knowledge gap - the missing answer gets recorded where it was noticed, without touching the 228 domain. - Typed relations link items (
SUPERSEDES, related docs) with dedup, no self-loops, and a cycle check on supersession - so "newer version" edges can never form a loop. - Version-scoped evidence attaches typed source URIs to a specific version (capped at 50), so "why should I believe this" has a pointer, not a promise.
There is also a capability registry - 256 capabilities seeded onto
production from the code itself (155 checks, 78 MCP tools, 14 manifest
entries, 8 x402 packs) so items can declare what they document, with a
drift-check endpoint comparing registry to code.
How we verified it: four audit rounds, forty-one real findings, zero guesswork
The engine went through the same adversarial loop we now apply to
every agent-facing surface - plus a newer deterministic layer:
- Local audit, round 1 - a verify bot ran a structured packet against the dev server and returned PARTIAL with two real findings: mutations accepted a missing
x-operator-idheader (silently attributing writes to "admin"), and NUL bytes in title/body fields crashed with a 500 instead of a typed 4xx. Both fixed with a shared operator-ref guard and a control-character predicate on the schema edge. - Local audit, round 2 - GREEN, 50/50 checks.
- Production audit - the same packet against
agentbadge.xyzcaught one more bug the local runs missed: NUL bytes on read paths (?q=, item ids, evidence URIs) produced unauthenticated 500s. The write-path fix had covered bodies only; path and query parameters never got the guard. One sharedrejectCtrllater, all five reproductions return typed 400s on production. - Deterministic regression - our probe bot then re-verified the fix on prod for free: 142 knowledge-surface cases, 0 failures; a
knowledge-lifecycleflow scenario now walks an item through create → replay → draft-invisible → published-visible → withdraw → invisible → terminal 409 in one deterministic probe, anddiffcompares local and prod responses case-by-case. - External code audit (Arc Studio, round 1) - a second, independent adversarial review of the source (not just the wire) found nineteen more findings - none critical, five high: operator-identity bypasses via zero-width and fullwidth lookalikes the prefix check missed, idempotency keys that ignored the request body (same key + different body silently replayed a stored 200 - a real "your withdrawal never happened" class), a TOCTOU window where a new version could be submitted while the reviewer was reading a different one, mutable evidence on published items, and unbounded gap writes on anonymous refusals. All nineteen were remediated, folded into the regression suite as fourteen dedicated tests, and re-verified locally with the probe battery.
- External code audit (Arc Studio, round 2) - a confirmation audit found zero critical but three highs and sixteen mediums - including one real regression in our own fix: the body-hash dedupe we added for double-submits wrapped every result as a success, so failed writes were cached for 24 hours. That mechanism is gone - every form now renders a single-use token. The other highs: approval gates ran before the row lock (concurrent evidence removal could slip through), and our separation-of-duties check is a two-name check under one shared admin key - real two-person control needs per-operator credentials, documented as a known limit. Mediums: a 39-hop-plus-decoy SUPERSEDES cycle bypass (now a locked recursive check in the store), lost-update edits (
expectedSeqprecondition), natural-language questions starved by strict AND search (an OR-token lane), and gap-cap semantics that could freeze dedupe bumps. All nineteen remediated and covered by a dedicated R2 regression file.
The pattern worth stealing: the expensive auditors found bug
classes; the cheap deterministic bot now owns those classes forever.
Each finding becomes a scenario or a regression test, and it runs on
every future deploy for $0.
The honest limits
- No synthesis. The composer assembles from verified items; it does not paraphrase, combine, or infer. If coverage is thin, it refuses. That is strictly less impressive than an LLM answer - and strictly more trustworthy.
- Small corpus by design. The registry seeds 256 capabilities but the knowledge base starts empty; it fills through the same approval workflow agents consume. Cold-start "thin answer" is a gap record, not a bug.
- No automatic projection yet. Published items are not yet injected into
llms.txtor.well-knownmanifests - that projection (O-229-5) is a documented open question, deferred until a design decision scopes it. - Search is lexical, not semantic. Postgres FTS finds words, not meanings. An agent asking in different vocabulary than the item's gets a refusal - which then becomes a gap. We'd rather have that signal than fuzzy matches presented as verified answers.
- Operator identity is one shared admin key. The separation-of-duties gate distinguishes operator names, not people: anyone holding the console key can post as any operator ref. Real two-person control needs per-operator credentials - we document this as a limit, not a feature.
- Idempotency is best-effort, in-process. The dedupe store lives in server memory; a restart forgets stored responses and a replayed key may run a second time (failing honestly on a slug conflict). Acceptable for an admin console - for money it would not be.
What is verified today
About 200 knowledge tests run in the server suite - workflow CAS, contract
schemas, search determinism, composer refusals, gap dedup and bounds,
relation cycles, evidence caps and immutability, SSR smoke, the
NUL/control regression cases, and a dedicated arc-r1 file covering
every remediated external finding (operator-identity bypasses,
idempotency body-binding, evidence gates, capability drift, gap
bounds). On production: migration applied 27 operations, the
search route reports engine:"FTS" on the live GIN index, 256
capabilities are seeded and verified, and the probe-bot pass shows 142
green cases with cache-control: no-store on every knowledge surface -
including the 401s.
For AI agents: the knowledge surface is public and read-only:
GET https://agentbadge.xyz/api/v1/agent/knowledge/search?q=<query>
returns published items with snippets; POST .../ask returns a
deterministic answer with sources or a typed refusal. Rate-limited at
10 req/min anonymous. A REFUSED response is not an error - it means
no verified item covers your query, and the response tells you
honestly whether the gap was recorded or dropped.
Read the arc: agent request inbox ·
x402 payment gateway ·
agent trust layers ·
A2A settlement ·
counterparty trust ·
agent interaction inbox



