"A practical incident-triage design that remembers failed fixes, not just successful ones, so an on-call agent learns from real operator feedback."

Hindsight Stopped My On-Call Agent From Repeating a Failed Fix

At 03:47 UTC, billing-api threw HikariPool-1 - Connection is not available, request timed out after 30000ms. Every LLM I know gives the same answer: the pool is too small, so increase it. In our team, that exact fix had already been tried, pushed Postgres past max_connections, and made the outage worse. A stateless model cannot know that. So I built an on-call agent that does.

This is how I wired Hindsight, an open-source agent memory system, into an incident triage loop, and what I learned about making an agent remember corrections instead of just documents.

The problem: the answer is in someone's head

When production breaks, the first question is: "Have we seen this before, and what did we do?" The answer usually lives in a Slack thread, a post-mortem no one can find, or the head of whoever was on call six months ago.

The root cause is also usually specific to your architecture. In my test data, the pool-exhaustion alert is not a sizing problem at all. A nightly invoice-reconciliation job (recon-batch) runs without a statement timeout and holds connections open past 02:00 UTC. The fix that worked was terminating those sessions with pg_terminate_backend and setting statement_timeout=60s on the role. The fix that failed was doubling the pool from 50 to 100.

No amount of prompt engineering gets that context into a model that has never seen your system. It has to come from memory.

What the system does

An engineer pastes an alert. The agent:

1. Recalls similar past incidents and corrections from Hindsight.
2. Asks the LLM (Groq, gpt-oss-120b with a qwen3-32b fallback) for a structured JSON verdict grounded in those memories.
3. Retains the alert and the still-unconfirmed verdict.
4. Waits for the engineer to mark the verdict right or wrong, with the actual cause, the actual fix, and any fix that failed.
5. Retains that feedback so the next similar alert benefits from it.

It is a FastAPI backend (app/main.py), a single-page UI, and one file, app/memory.py, that holds every Hindsight call. Each team gets its own memory bank via bank_id, so one team's incidents never bleed into another's.

The through-line: store the failures, not just the fixes

Most "agent memory" demos store what worked. The design decision I care about most is storing what did not work in a form the model cannot ignore.

When an engineer submits feedback, record_feedback in app/triage.py builds a plain-text record with explicit markers:

parts = [ f"Engineer feedback on incident {req.incident_id} (service {ctx['service'] or 'unknown'}).", f"The alert was: {ctx['alert'][:500]}", f"The agent's suggested root cause was {'CORRECT' if req.correct else 'WRONG'}.", ] if actual_cause: parts.append(f"CONFIRMED ROOT CAUSE: {actual_cause}") if actual_fix: parts.append(f"FIX THAT WORKED: {actual_fix}") for f in failed: parts.append(f"FIX THAT DID NOT WORK (do not suggest again for this kind of alert): {f}") 

That text is retained into Hindsight with a per-incident document_id, so a later correction to the same incident replaces the record instead of duplicating it:

client.retain( bank_id=bank_id, content=content, context="incident outcome: engineer correction", document_id=f"{incident_id}-feedback", tags=["incident", "correction", f"service:{service}"], retain_async=False, ) 

I use retain_async=False on purpose. If an engineer corrects the agent and a similar alert fires a minute later, the correction has to be recallable right away.

On the way back out, recall runs before every verdict:

resp = self._client.recall(bank_id=bank_id, query=query, max_tokens=2048, budget="mid") texts = [getattr(r, "text", str(r)) for r in (resp.results or [])][:limit] 

The markers then do their job in the prompt. Memories containing CONFIRMED ROOT CAUSE: are sorted first and labelled VERIFIED PAST INCIDENT, and the system prompt says that if a memory records a failed fix, the model must not suggest it again and must say it is avoiding it because of past incident memory. The response also returns cited_memories, and the UI shows a "Memories used" panel, so an engineer can see exactly why the agent said what it said.

Recall quality was the real work

Retaining is the easy half. Getting the right memory back for an alert that does not look like the original was where I spent most of my time.

Two alerts about the same failure rarely share words. One is a HikariPool timeout on billing-api; the next is a pgbouncer: no more connections allowed on invoice-service. Same cause, different vocabulary, different service.

So build_recall_query enriches the raw alert with domain concepts before it goes to Hindsight:

if any(k in alert_lower for k in ("hikaripool", "pgbouncer", "max_client_conn", "connection is not available")): q_parts.append("database connection pool postgres recon-batch statement_timeout") if any(k in alert_lower for k in ("getaddrinfo", "eai_again", "read udp", ":53", "lookup", "dns", "coredns")): q_parts.append("dns name resolution coredns udp lookup timeout") 

It is a hand-written list for eight failure patterns, and I am under no illusion that it generalizes. It is a pragmatic bridge between how alerts are written and how post-mortems are written.

The repo supports two Hindsight deployments behind one Memory protocol: the hindsight-client SDK against Hindsight Cloud or a self-hosted Docker container, and the Hindsight Inference API, where candidate memories are scored with hs-decide-1-flash through a Cohere-compatible rerank call. Because both sit behind the same retain/recall interface, triage.py does not know or care which one is running.

What happened: 0 of 24 without memory

To measure this honestly, I built a replay that runs the same incidents with memory off and on. When memory is off, triage.py skips both recall and retain, and feedback retains nothing, so the baseline cannot learn by accident. The dataset is 40 synthetic incidents across 8 recurring patterns. Each wave runs one incident from each pattern, with engineer feedback in between.

Wave 1 is a tie, and it should be: nothing has been retained yet, so the agent gives a generic answer, the engineer corrects it, and Hindsight stores the correction. By wave 3, every pattern was resolved correctly. The stateless baseline stayed at zero across all 24.

In wave 2, the DNS-resolution and Kafka-disk-full incidents still missed even though memory was on. Those two failed for a simple reason: the alert names were different enough that the retrieval layer did not surface the earlier correction at the right moment. The problem was not storage; it was semantic mismatch between the alert's wording and the original incident description. Once the query captured the relevant concepts, the correction became usable again.

Before: a HikariPool timeout alert, memory off. The agent suggested raising the pool size without context, which the matcher scored as a miss.

Agent verdict: "This is a connection pool sizing issue. Increase the maximum pool size and watch the Postgres max_connections ceiling."

After: the same pattern, memory on, with recon-batch cited from a prior correction.

Agent verdict: "This is likely a leaked transaction or a long-running reconciliation job holding database connections. The earlier incident memory indicates recon-batch was running without a statement timeout and the successful remediation was to terminate backend sessions with pg_terminate_backend and set statement_timeout=60s on the role. I am avoiding the generic "increase pool size" fix because it failed in a similar incident."

Lessons learned

1. Store failures explicitly. A line that says "FIX THAT DID NOT WORK" is more valuable than a line that says what worked, because it is the one thing a stateless model will keep repeating.
2. Retain the unconfirmed guess too, but label it. The agent's own verdict is stored as "NOT yet confirmed by an engineer," so it cannot be mistaken for ground truth later.
3. Memory has to fail soft. Hindsight timeouts and auth errors log a warning and triage continues with empty memory. An incident tool that crashes when its memory is down is worse than no tool. There are tests for exactly this.
4. Recall is a retrieval problem, not a storage problem. My accuracy gap was never about whether Hindsight stored the correction. It was about whether the query looked enough like it.
5. Keep the baseline honest. Memory-off mode bypasses retain entirely. Otherwise your "before" number quietly improves and your comparison means nothing.

Limitations

The incidents are synthetic, generated from eight patterns. The matcher that scores verdicts (app/judge.py) is a heuristic, not a human reviewer, and eight patterns repeated three times says little about a messy real incident history. The domain-term list in the recall query is hand-tuned. A real deployment would need real post-mortems, more patterns, and a human-reviewed evaluation.

Try it

The code is at github.com/FirdaushAnwer/ON-CALL-AGENT-WITH-MEMORY. To learn how it works under the hood, start with the Hindsight documentation, and if you want the broader idea, Vectorize's explainer on what agent memory is is a good primer. Hindsight is open source, so you can run it yourself with a single Docker command and point HINDSIGHT_BASE_URL at it.