I Wrote Up 938 Incidents. Then I Made Them Searchable — and Monitored the Search.

Every bug in my Kubernetes platform gets a write-up. Symptom, root cause, fix, the test that proves
the fix, and the lesson. No GitHub Issues: plain markdown in the repo, next to the code it describes.

That rule has been running for months. The repo now holds:

  • 560 bug docs
  • 378 issue and post-incident docs
  • 267 plans and 81 retros

About 1,750 documents. And at some point I noticed the obvious problem: a write-up you can't find
is a write-up you didn't write. I was re-diagnosing failures I'd already solved, because grep only
finds the words you already guessed.

Making the history searchable

The platform is a multi-cluster setup I run myself — a k3d hub, a k3s app cluster on a VPS, and
ephemeral AWS clusters — with ArgoCD, Vault, External Secrets, Prometheus and Keycloak. So the search
index is built the same way as everything else on it:

  • pgvector in Postgres on the hub, deployed by ArgoCD like any other workload.
  • An indexer job that embeds new and changed docs, prunes deleted ones, and runs every few minutes.
  • The database credentials come from Vault through External Secrets. Nothing is in git.
  • Two consumers:
    • make find-similar-docs Q="..." — run before filing a new bug, so a recurrence gets appended to the existing doc instead of becoming a duplicate.
    • /ask-docs in Slack — ask a question, get an answer summarised from the top matches, with the source paths.

The filing step matters most. This repo once refiled the same External Secrets defect under a name
nobody's glob could match. Similarity search catches that; exact filenames don't.

Then I monitored it like a production service

A search index that silently stops updating is worse than no index, because you trust its answers.
So it got a Grafana dashboard with the same questions I'd ask of any service:

  • Is it current? Rows in the index vs documents in the corpus, and index age in seconds.
  • Is it drifting? Docs whose content changed since they were embedded, and the backlog waiting.
  • Did the last run work? Last result, time since the last successful index, run duration.
  • Is it blocked? Whether the embedding API's daily quota has paused ingestion.
  • Can it be reached? Database reachability, and whether its ExternalSecret is synced.

Grafana vector-index dashboard, 2026-10-01: index rows tracking corpus docs from 1,742 to 1,754 over six hours, drift and backlog blipping to 1 and back, index age a sawtooth near zero, store reachable and ExternalSecret synced, last run success 6 minutes ago, not paused on quota, runs under 45 seconds

On a normal day the corpus line climbs a step at a time — about a dozen new write-ups in six hours on the day I took that screenshot — and the index line follows within minutes. Drift and backlog spike to 1 and drop back to 0. Index age is a sawtooth that never gets far from zero.

That shape is the whole point. It tells me the pipeline is boring, so I can trust it.

The bug on day one

I asked /ask-docs in Slack: "what's the recent issue regarding Slack and webhook issues?"

It answered confidently, citing five docs. All five were from June to August. That same day, three
Slack and webhook incidents had been filed.

The index wasn't stale — the dashboard said 1,751 of 1,753 docs, refreshed minutes earlier. The
problem was the retrieval itself:

  • Ranking was pure similarity. "Recent" in the question nudged the embedding slightly; nothing ranked or filtered by date.
  • The top 30 matches all scored between 0.69 and 0.75. In a band that tight, "closest text" barely separates anything.
  • The doc filed that day ranked 6th, one below the cutoff. The outage doc didn't make the top 30 — its text was about GitHub secrets, so it read as off-topic.
  • The model writing the summary saw only each doc's title and first 600 characters, so it couldn't tell new from old either.

The fix was small: detect recency words, pull a wider pool of 50, keep the score floor, sort newest
first, and put each doc's date in front of the model and in the source list. Questions without those
words rank exactly as before. It shipped with tests that fail if the sort, the floor, or the
word-boundary match is removed.

The lesson is the one my own write-ups keep repeating: the dashboard was green, and the answer was
wrong. Green means the checks you wrote are passing. It says nothing about the questions you
didn't think to check.

What 938 incidents actually look like

Once the history was searchable, I could ask a different question: how much of this is just my
setup, and how much would any Kubernetes team hit?

Sorting the bug and issue titles, roughly 60% are specific to how I work — throwaway four-hour
cloud sandboxes, AI coding agents, a Slack bot. The other 40% is the stuff every self-managed
cluster runs into:

  • ArgoCD apps that flip OutOfSync forever because server-side apply isn't server-side diff.
  • An ExternalSecret error that reads the same for "path missing" and "permission denied."
  • Alertmanager throttled 83–87% of the time at 21 millicores of average CPU.
  • A Grafana that had never been scraped, because a Helm subchart doesn't inherit the release label.
  • CoreDNS restarting 155 times on a one-second health check.

That 40% is the part I'd want a new teammate to read first. Now they can ask for it.

If you want to try this

You don't need my stack. You need three habits:

  1. Write the incident down where the code lives, with the root cause and the test, every time.
  2. Search before you file. Make duplicate detection a step, not a hope.
  3. Monitor the index. If you can't see that it's current, you can't trust what it tells you.

How do you find out whether your team has already solved the problem in front of you?

LinkedIn Version

I've written up 938 incidents on my Kubernetes platform. Every bug gets a markdown doc: symptom,
root cause, fix, the test that proves it.

The problem: a write-up you can't find is a write-up you didn't write. I kept re-diagnosing things
I'd already solved.

So I made the history searchable — pgvector on the cluster, an indexer that runs every few minutes,
/ask-docs in Slack, and a similarity check before anyone files a new bug.

Then I monitored the index like a production service: index age, rows vs docs, drift, backlog, last
run, quota pauses, secret sync.

On day one I asked it for "recent Slack issues." It cited five docs from the summer. Three had been
filed that same day.

The dashboard was green. The index was current. Retrieval simply had no idea what "recent" meant —
pure similarity, every score within 0.69–0.75. Today's doc ranked 6th of a top 5.

Fixed with a recency mode and dates in the answer. But the lesson is the one 938 write-ups keep
teaching me:

Green means the checks you wrote are passing. Not that the answer is right.

How does your team find out it already solved the problem in front of it?