When a Sev-1 alert rips you out of sleep at 3:14 a.m., your instinct is rarely subtle. You check the dashboard, see p99 latency spiking to six seconds on checkout, and reach for the bluntest tool available: restart the pods.
The problem isn't that on-call engineers lack competence; it's that intuitive emergency fixes in distributed systems are often landmines. If a service degrades due to database connection pool exhaustion, bouncing pods purges the shared cache, floods cold disks with read queries, and turns a localized latency spike into a cascading brownout.
The most infuriating part? We had made that exact mistake six weeks earlier. It was documented in a postmortem nobody had open.
To stop repeating past mistakes, we built RecallOps: an incident triage system backed by institutional memory. The backend leverages Hindsight to retain and recall historical incident resolutions and operational anti-patterns. But as the frontend engineer on the project, I quickly realized that sophisticated retrieval is useless if the UI doesn't make that memory instantly actionable to a sleep-deprived human.

Here is how we brought Vectorize agent memory into our Next.js triage console—and how UI primitives like per-action outcome badges, explicit "Avoid" warnings, and side-by-side memory toggles changed how our team handles production outages.

What the System Does and How It Hangs Together
RecallOps sits directly between our alerting pipeline and our on-call team. When an alert fires—such as checkout-api crossing its latency threshold—the system triggers an automated triage workflow:
Context Ingestion: The FastAPI backend pulls active alerts, recent Git deployments, container logs, and telemetry from PostgreSQL.
Cognitive Memory Recall: The backend queries Hindsight across memory banks: our shared organizational bank (kartly-sre) for similar resolved incidents and explicit negative memories (actions that previously failed).
Agentic Reasoning Loop: An LLM agent synthesizes telemetry against recalled precedents, executes diagnostic tool calls, and derives the root cause and safe remediation steps.
Server-Sent Events (SSE): The backend streams thoughts, tool invocations, recalled memories, and suggestions over an event stream.
The React Console: Our Next.js application folds the stream into a reactive UI, rendering a live timeline, an interactive memory panel, actionable suggestion cards, and a side-by-side comparison view.

Incoming Alert ──> FastAPI & Agent Loop ──> SSE Stream ──> Next.js Console │ ▲ ├─ Alert Strip ▼ │ ├─ Live Timeline Postgres Telemetry │ ├─ Memory Panel (Badges) │ │ ├─ Suggestion & Avoid ▼ │ └─ Side-by-Side Compare Hindsight Cloud ───┘

The Core Technical Story: Making Agent Memory Visible
When we first connected an LLM to our operational runbooks, the output looked like every other AI demo: a massive wall of markdown text.
During an active Sev-1 outage, nobody reads a five-paragraph essay. Furthermore, early iterations had a dangerous blind spot: naive retrieval treated incident postmortems as monolithic chunks. An incident writeup typically records several failed attempts before finding the fix that worked. When a generic agent embeds that postmortem, semantic similarity often retrieves the failed actions as viable suggestions because they share keywords with the symptom.
To solve this, we established three design principles across our frontend architecture:

  1. Granular Actions, Not Monolithic Incidents We rejected labeling an entire incident as simply "resolved." An incident is a timeline of human decisions. In our TypeScript definitions, every memory hit contains individual actions, each with its own verified outcome: 
export type Outcome = "worked" | "failed" | "partial"; export interface MemoryAction { text: string; outcome: Outcome; } export interface MemoryHit { id: string; // e.g. "INC-1987" occurred_at: string; // ISO timestamp summary?: string; actions: MemoryAction[]; // one outcome per action, never per incident }

As captured in lib/types.ts: one outcome per action, never per incident. This invariant shaped our entire presentation layer.

  1. Highlighting Anti-Patterns as Explicit "Avoid" Warnings When Hindsight recalls an action tagged with outcome: "failed", the triage engine treats it as high-priority negative memory and emits an avoid directive. In the UI, this translates into high-contrast danger callouts right inside the suggestion card.
  2. Transparent Verification via Side-by-Side Comparison To build trust with skeptical engineers, we created a comparison view (/incidents/[id]/compare) running two triage streams concurrently: one with memory enabled and one stateless. Seeing the baseline agent recommend generic pod restarts while the memory-backed agent pinpoints a specific rollback based on past precedent makes the value of memory instantly clear. --- Code-Backed Explanations Here is how this flows from Hindsight's Python client into our Next.js interface.
  3. Ingesting Negative Memory in Hindsight Service In backend/backend/app/services/hindsight_service.py, we store postmortem resolutions and isolated failed fix outcomes into Hindsight Cloud: 
async def remember_fix_outcome( self, incident_id: str, service_name: str, signature: str, attempted_action: str, outcome: str = "failed", note: Optional[str] = None, warning_directive: Optional[str] = None, ) -> bool: target_bank = self.ORGANIZATIONAL_BANK is_failed = (outcome.lower() == "failed") memory_type = "failed_fix" if is_failed else "successful_fix" if is_failed and not warning_directive: warning_directive = ( f"Do NOT attempt '{attempted_action}' on {service_name} " f"under {signature}: {note or 'caused cascading degradation'}" ) content = ( f"[{memory_type.upper()}] Incident {incident_id} on {service_name}: " f"Action '{attempted_action}'. Outcome: {outcome}. " f"{f'Warning: {warning_directive}' if is_failed else ''}" ) client = self._get_client() if client: client.retain( bank_id=target_bank, content=content, context=memory_type, metadata={"memory_type": memory_type, "outcome": str(outcome)}, tags=[memory_type, service_name], ) return True return False

Per the Hindsight documentation, tagging memory types isolates safety directives from semantic search. retrieve_unified_memory() queries both resolutions and failed fixes, enforcing a strict guardrail: negative memory warnings cannot be recommended.

  1. Streaming and Folding Events in React In web/recallops-web/lib/useTriage.ts, a custom hook consumes the SSE stream through a pure reducer: 
function reduce(s: TriageState, ev: TriageEvent): TriageState { switch (ev.type) { case "thought": return { ...s, steps: [...s.steps, { kind: "thought", id: ev.data.id, text: ev.data.text }] }; case "tool_call": return { ...s, steps: [...s.steps, { kind: "tool", id: ev.data.id, tool: ev.data.tool, output: ev.data.output }] }; case "memory_hit": { if (s.memories.some((m) => m.id === ev.data.id)) return s; return { ...s, memories: [...s.memories, ev.data], steps: [...s.steps, { kind: "memory", id: `mem-${ev.data.id}`, incident_id: ev.data.incident_id, occurred_at: ev.data.occurred_at }], }; } case "suggestion": return { ...s, suggestion: ev.data }; case "error": if (ev.data.stage === "memory") return { ...s, memoryError: ev.data.message }; return { ...s, status: "failed", fatalError: ev.data.message }; default: return s; } }

Two frontend design details made a significant operational difference:
Deduplication: Checking s.memories.some(...) prevents duplicate cards if event streams reconnect on unstable connections.
Graceful Degradation: Notice stage === "memory". If Hindsight encounters an unreachable network path, we set memoryError on the panel, but the triage stream keeps running. The LLM falls back to standard telemetry analysis, and the on-call engineer sees: "Couldn’t reach incident memory. Triage continues without past incidents."

  1. Rendering Per-Action Badges and Avoid Warnings In web/recallops-web/components/triage.tsx, MemoryPanel and SuggestionBlock render these parsed structures into glanceable components: 
function MemoryCard({ m }: { m: MemoryHit }) { return ( <article className="step-in rounded-lg border border-line bg-panel p-4"> <header className="mb-2 flex items-center justify-between gap-3"> <span className="font-mono text-code font-medium">{m.incident_id}</span> <span className="text-xs text-muted">{ago(m.occurred_at)}</span> </header> {m.summary && <p className="mb-2 text-muted">{m.summary}</p>} <ul className="flex flex-col gap-2"> {m.actions.map((a, i) => ( <li key={i} className="flex items-start justify-between gap-3"> <span>{a.text}</span> <OutcomeBadge o={a.outcome} /> </li> ))} </ul> </article> ); } export function SuggestionBlock({ s, onVerdict }: { s: Suggestion; /* ... */ }) { return ( <div className="step-in rounded-lg border border-l-2 border-line border-l-accent bg-panel p-5"> <p className="font-semibold">{s.steps ? s.root_cause : `Likely root cause: ${lower(s.root_cause)}`}</p> {s.fix && <p className="mt-2">Recommended fix: {lower(s.fix)}</p>} {s.avoid?.map((a) => ( <p key={a.text} className="mt-2 text-danger">Avoid {lower(a.text)}</p> ))} <div className="mt-5 flex flex-wrap gap-3"> <Button variant="primary" onClick={() => send("accept")}>Accept</Button> <Button onClick={() => send("reject")}>Reject</Button> <Button variant="outline" onClick={() => setMode("correcting")}>Correct</Button> </div> </div> ); }

The OutcomeBadge uses semantic styling: green for Worked, red for Failed, and amber for Partly worked. Seeing a red Failed badge next to "Restarted pods twice. Latency got worse." creates an instant cognitive brake before an engineer touches the terminal.


Behavior in Practice: A Concrete Incident Walkthrough
To see why this matters, consider a production incident on checkout-api: INC-2041.
The Alert
At 03:14 IST, our monitoring fired a Sev-1 alert:
Service: checkout-api
Metric: p99_latency
Observed Value: 6.2s (Threshold: 2.0s)
Run 1: Memory OFF (The Naive SRE Experience)
When an engineer toggles memory off—simulating standard stateless runbook assistants—the triage engine inspects recent deploys and sees commit #4412 (config: db.pool.max 40 -> 10). Without past outcome memory, it falls back to standard heuristics:
Diagnosis: "Possible causes: high CPU, traffic spike, database slowness."
Suggested Steps:
Step 1: Check CPU usage.
Step 2: Restart checkout-api pods.
Step 3: Scale up replicas.
Because the stateless model has no historical grounding, its default response is to bounce the service. Had the on-call engineer followed that suggestion, cold-cache thrashing would have cascaded across dependent services.
Run 2: Memory ON (With Hindsight Memory Badges)
When the engineer runs triage with memory active, Hindsight correlates the error signature with past postmortems within seconds:
Timeline Streaming: The agent executes get_recent_deploys and immediately emits a memory hit:

Recalled INC-1987 from 43 days ago.
Memory Panel Populates: An incident card for INC-1987 appears in the right rail with clear badges:
Restarted pods twice. Latency got worse. Failed
Rolled back pool config from deploy #3980. Recovered in 6 min. Worked
A second card for INC-1902 confirms: Rolled back pool config. Recovered in 8 min. [Worked]
Suggestion Block Surfaces Root Cause and Avoid Directive:
Root Cause: Connection pool limit reduced in deploy #4412.
Recommended Fix: Roll back the pool config from #4412.
Avoid Warning: Avoid restarting checkout-api pods. This failed in INC-1987 and made latency worse.
Owner: Payments platform team (Rahul Menon).
The engineer didn't need to dig through archived tickets. The red badge and the explicit avoid directive stopped the bad fix immediately. The configuration change was rolled back, and latency recovered within minutes.

Once resolved, the engineer submitted the resolution form, persisting the MTTR and notes back into Hindsight to protect the team during future rotations.


Lessons Learned
Building an incident console around agent memory provided several insights for designing engineering tools:

  1. Negative Memory Is More Critical Than Positive Memory In production systems, knowing what not to do is often more urgent than knowing the ideal fix. A suboptimal fix delays recovery; a known anti-pattern can take down your entire cluster. Storing failed interventions as first-class negative memories with explicit warning directives is the highest-leverage feature Hindsight provides.
  2. Never Aggregate Incident Outcomes to a Single Status If your storage system records an entire incident as simply "resolved," your retrieval pipeline will eventually recommend the failed experiments that preceded the fix. Splitting postmortems into discrete actions—each with its own worked or failed badge—is essential for safe automation.
  3. High-Stress UIs Must Minimize Reading Time Engineers responding to night alerts experience tunnel vision and cognitive fatigue. Do not present them with chat windows or verbose essays. Use semantic colors, badges, and stark avoid callouts. The UI should make the right path effortless and the dangerous path visually unmistakable.
  4. Provide a "Memory Off" Comparison to Earn Trust Experienced engineers are skeptical of AI in incident management. Giving them a side-by-side comparison toggle—where they can evaluate the memory-augmented recommendation directly against the generic stateless guess—proved to be our best adoption mechanism. Transparency builds credibility faster than claims of intelligence.
  5. Always Design for Graceful Degradation Cognitive backends can face temporary timeouts or network partitions. Your frontend triage loop must treat memory as a progressive enhancement: if memory retrieval fails, display the warning in the side panel, keep the telemetry timeline flowing, and let the engineer proceed with live diagnostics without interruption. --- Institutional knowledge shouldn't evaporate when an incident Slack channel is archived. By combining Hindsight's episodic memory with explicit UI affordances, we turned painful operational mistakes into active guardrails that protect production every day.

Hindsight GitHub: https://github.com/vectorize-io/hindsight
Hindsight docs: https://hindsight.vectorize.io/
Vectorize agent memory: https://vectorize.io/what-is-agent-memory

Built with the Code.in community.