Here's an illustrative one. Your support tool drafts replies with a model, and an agent reviews each draft before it goes out. A customer forwards a reply with a wrong refund date and asks who wrote it. Your logs say the message was "approved by agent-12". That's all.
Was the wrong date in the model's draft, or did the agent type it? Did they read the draft at all, or click Approve two seconds after it appeared? "Approved" covers all of those, and they lead to very different fixes.
A human approval step is a standard answer to risky AI actions. OWASP's LLM06:2025 Excessive Agency lists human-in-the-loop approval for high-impact actions among its mitigations, next to logging and monitoring what extensions and downstream systems do. But an approval step only helps later if you recorded what the reviewer actually did.
What "approved" hides
A single approved_by column usually flattens at least four outcomes:
- approved the draft exactly as generated;
- edited it, a little or a lot, and then sent it;
- rejected it and wrote something else;
- never decided, so a timeout or default sent it.
You want to tell these apart without guessing, and you want the model's original draft to survive the edit.
Two events, not one row
The simplest durable shape is two events. The first records the suggestion when it's generated, before anyone sees it. The second records the review and points back at the first. Nothing gets overwritten.
model -> draft --(1) ai_suggestion: id, sha256, model ref, time | v reviewer UI --(2) human_review: suggestion id, reviewer, | action, final sha256, similarity, v reason, shown_at, decided_at sent message If you update the suggestion row in place when the agent edits it, the original draft is gone. That's exactly the thing someone will ask about.
The code
import difflib import hashlib from datetime import datetime def sha(text: str) -> str: return hashlib.sha256(text.encode("utf-8")).hexdigest() def suggestion_event(suggestion_id, text, model_ref, created_at): # Written once, before anyone sees the draft. Never updated. return { "type": "ai_suggestion", "suggestion_id": suggestion_id, "sha256": sha(text), "model_ref": model_ref, # e.g. pinned model + prompt version "created_at": created_at, } def review_event(suggestion, shown_text, final_text, reviewer_id, shown_at, decided_at, reason=None): # A separate event that points at the suggestion; it doesn't edit it. if sha(shown_text) != suggestion["sha256"]: raise ValueError("shown text does not match the suggestion") if final_text is None: if not reason: raise ValueError("a rejection needs a reason code") action, similarity = "rejected", None elif final_text == shown_text: action, similarity = "approved_unchanged", 1.0 else: action = "edited" similarity = round( difflib.SequenceMatcher(None, shown_text, final_text).ratio(), 3) seconds = (datetime.fromisoformat(decided_at) - datetime.fromisoformat(shown_at)).total_seconds() return { "type": "human_review", "suggestion_id": suggestion["suggestion_id"], "reviewer_id": reviewer_id, "action": action, "final_sha256": None if final_text is None else sha(final_text), "similarity": similarity, "reason": reason, "shown_at": shown_at, "decided_at": decided_at, "seconds_to_decide": seconds, } A few choices worth explaining.
The review checks what was shown. If the UI rendered something other than the recorded suggestion, say a stale draft after a retry, the review would be attached to text nobody approved. Hashing the shown text and comparing it with the suggestion catches that at write time instead of during an investigation.
"Edited" is its own action. An edited reply is partly the reviewer's work. Recording similarity from Python's difflib.SequenceMatcher.ratio(), a float between 0 and 1, gives a rough sense of how much changed. A typo fix lands near 1. A rewrite lands much lower.
Rejections need a reason code. A short fixed list ("wrong facts", "tone", "policy", "other") is enough. Free text is fine as an extra, but codes are what you can count.
Time is recorded, not judged. seconds_to_decide is a signal. A run of unchanged approvals in under three seconds is worth a look, but the record doesn't decide that someone rubber-stamped.
The tests pin the behavior that matters:
import pytest from review import suggestion_event, review_event DRAFT = "Sorry for the delay. Your refund was issued today." T0, T1 = "2026-11-20T14:00:00+00:00", "2026-11-20T14:00:41+00:00" SUG = suggestion_event("s-81", DRAFT, "model-x@2026-10/prompt-v4", T0) def test_edit_is_not_an_approval(): final = "Sorry for the delay. Your refund was issued today, ref R-1." ev = review_event(SUG, DRAFT, final, "agent-12", T0, T1) assert ev["action"] == "edited" assert 0 < ev["similarity"] < 1 assert ev["seconds_to_decide"] == 41 def test_rejection_needs_a_reason(): with pytest.raises(ValueError): review_event(SUG, DRAFT, None, "agent-12", T0, T1) def test_shown_text_must_match_the_suggestion(): with pytest.raises(ValueError): review_event(SUG, DRAFT + " Thanks!", DRAFT, "agent-12", T0, T1) Where the text lives
These events store hashes, not text. The draft and final message stay in the systems that already hold customer messages, with their access controls. The hashes let you check later that the text you're looking at is what was suggested and what was sent. If you store text in the events too, treat them with the same sensitivity as the messages.
Whatever you choose, keep both the original draft and the final text for as long as you keep the review events. A review that points at a draft you deleted can tell you an edit happened, but not what it was.
Limits
- The record shows what the reviewer did, not whether they understood it. Time on screen isn't attention.
-
ratio()measures character similarity. A one-word change can flip the meaning and still score near 1, so read the diff when it matters. - It only covers the path through your review UI. If agents can also send messages another way, those need the same events, or your approval rate looks better than it is.
- Timeouts and auto-sends need an explicit event too. Otherwise they show up as missing reviews rather than as decisions nobody made.
Where this fits
We work on Obligra Verify, which is designed to retain the evidence your application selects. Our page on working with decision records covers human-in-the-loop approvals, where keeping the recommendation, the change and the final outcome together is the point.
This article was generated by an AI agent (Claude) for the Obligra Verify team and checked against the linked sources.