48-Hour Field Notes: Every Unparseable Alert Became 'Low' and Nothing Paged Me for 48 Hours

A log triage worker that fails open is not a triage worker; it is a very polite way of losing alerts. Over 48 hours last week I ran a small pipeline that read error lines, asked a model for a severity label, and routed each label into one of four channels. Nothing crashed, nothing restarted, and the digest channel filled up right on schedule. So why did a database timeout that should have woken me at 2 a.m. sit in a daily digest until the next morning?

I rebuilt the whole thing afterwards as a minimal repo so the failure is reproducible instead of anecdotal. Everything below runs locally, and the schema plus the contract test are the artifacts I actually kept. The lesson is not "models are unreliable"; it is that my validation layer had two doors open and I had only ever tested the hallway.

The setup: a loop cheap enough to leave running overnight

The job was deliberately boring, because boring jobs are the ones you actually leave running. Each cycle did four things.

1. Tail an error log stream and drop lines that already carried a label.
2. Batch the survivors into groups of twenty lines.
3. Ask the model for one JSON object per line: a severity and a category.
4. Map severity to a Slack channel and post the batch.

I used MonkeyCode's free model access for step 3 and its free server option to host the worker, because I wanted the loop to keep running while I slept without putting a card behind it. Disclosure: This article was prepared as part of MonkeyCode's product outreach. What I was actually evaluating was whether the loop survived an unattended night, not whether the labels were brilliant.

Hours 0-8: twenty green fixtures and a false sense of coverage

My first test suite had twenty fixtures and every single one of them passed. The fixtures were written from the examples I had already put in the prompt, so they arrived in exactly the shape the prompt requested. That is not a test suite; that is a mirror with assertions bolted onto it. Have you ever treated "no alerts" as evidence rather than as a hypothesis?

Deployment was uneventful, which I promptly mistook for correctness. The digest channel got a steady trickle of low-severity entries, the ops channel stayed quiet, and the page channel stayed quieter still. I told myself it had been a calm night for the service. It had not been a calm night; it had been an unobserved one.

The bug had two doors and I had left both of them open

Here is the version that ran for two days. It is short enough that I read it four times without seeing anything wrong with it.

# triage_v1.py: the version that ran for 48 hours from pydantic import BaseModel, Field class Label(BaseModel): severity: int = Field(ge=1, le=3) category: str ROUTE = {1: "#digest", 2: "#ops-chat", 3: "#page-oncall"} def label(text: str) -> Label: raw = ask_model(text) # dict, decoded from the response  try: return Label(**raw) except Exception: # broad on purpose, or so I claimed  return Label(severity=1, category="unknown") 

Door one: the fallback that downgraded the unknown

Any parse failure became severity=1, which routes straight to the digest channel. I wrote that line believing I was protecting the pager from noisy model output, and in doing so I made "I could not understand this" indistinguishable from "this is not important". Those two states need opposite responses, and I had collapsed both of them into one branch.

Door two: the coercion that never raised at all

Pydantic's default lax mode is happy to convert some genuinely surprising inputs. A JSON true validates as the integer 1, and a quoted "3" becomes 3 without any complaint at all. So when the model answered with a boolean or a quoted number, no exception fired and the except branch never even ran. The quoted-number case was accidentally correct, which is exactly why I trusted the pipeline for a day and a half.

The finishing touch was my logging. I logged the parsed object rather than the raw response, so the one piece of evidence that would have ended the investigation in ten minutes was never written anywhere. My logs sat downstream of the bug, which is a very comfortable place to be wrong.

Fix one: a schema that refuses to guess

I replaced the integer with a closed set of strings and turned strict mode on. If the model answers in the wrong shape, I want an exception, not a quiet repair.

# schema.py: the version that refuses to guess from typing import Literal from pydantic import BaseModel, ConfigDict, Field Severity = Literal["low", "medium", "high"] class Label(BaseModel): model_config = ConfigDict(strict=True, extra="forbid") severity: Severity category: str = Field(min_length=1, max_length=64) 

Three properties matter here. Literal turns an unknown severity into a hard error instead of a number I have to interpret, strict=True stops true and "3" from sneaking into a string field, and extra="forbid" catches the model inventing a third key that the worker would silently ignore.

Fix two: make the fallback loud and log both sides

The routing function now returns a channel name instead of a model instance, which makes the fallback impossible to bury. Unparsed output goes to its own channel, #triage-unparsed, and never to the digest.

# triage_v2.py: nothing unparsed is allowed to look low from pydantic import ValidationError from schema import Label ROUTE = {"low": "#digest", "medium": "#ops-chat", "high": "#page-oncall"} UNPARSED = "#triage-unparsed" def label(text: str) -> str: raw = ask_model(text) try: parsed = Label.model_validate(raw) except ValidationError as exc: log.warning("unparsed_label", raw=raw, errors=exc.errors()) return UNPARSED log.info("labeled", raw=raw, parsed=parsed.model_dump()) return ROUTE[parsed.severity] 

Notice that raw appears in both branches. That single change is worth more than every prompt tweak I made that week, because it means the next investigation starts from evidence instead of from a reconstruction of what I think the model said.

The contract test I should have written in hour zero

This test asserts the absence of silent success, which is the opposite of what my twenty happy-path fixtures did. Every case below must raise, and if one of them ever stops raising, the schema has drifted underneath me.

# tests/test_schema_contract.py import pytest from pydantic import ValidationError from schema import Label ADVERSARIAL = [ {"severity": 3, "category": "db"}, # int instead of the enum  {"severity": True, "category": "db"}, # bool coerced to int in lax mode  {"severity": "HIGH", "category": "db"}, # wrong case  {"severity": "critical", "category": "db"}, # not in the closed set  {"severity": None, "category": "db"}, # missing value  {"severity": "low", "category": ""}, # empty category  {"severity": "low"}, # missing key  {"severity": "low", "category": "db", "why": "x"}, # invented key ] @pytest.mark.parametrize("payload", ADVERSARIAL) def test_coercion_is_never_silent(payload): with pytest.raises(ValidationError): Label.model_validate(payload) 

Run it with pytest -q tests/test_schema_contract.py, and wire that command into the same entrypoint that starts the worker so a schema edit cannot ship unexamined. I added one routing test as well, because the schema and the router can drift apart independently of each other.

def test_unparsed_never_reaches_the_digest(monkeypatch): monkeypatch.setattr("triage_v2.ask_model", lambda _: {"severity": 3, "category": "db"}) assert label("connection timeout") == "#triage-unparsed" 

When a model labeler is the wrong tool

This pattern is fine for sorting noise and wrong for anything with a deadline attached to it. The table below is the one I now use before adding a model call to an existing pipeline.

What I would repeat and what I would not

Repeating: running the worker locally against the free model for a full day before deploying it, logging raw and parsed payloads side by side, and writing the adversarial fixture list before the second feature. The free server option was genuinely useful for the unattended part of the experiment, because the interesting failures only appeared after a few hours of real, boring traffic.

Not repeating: the broad except, the quiet default, and treating a passing happy-path suite as evidence about production. I also would not put a model label in the page path again without a rule-based check standing next to it and failing loudly.

Who should skip this entirely

If your alerts wake people up, this is a triage assistant and not an incident commander, so keep the deterministic path and let the model merely suggest categories. If you cannot commit to reviewing an unparsed queue once a day, do not create one, because an unread channel is just the quiet default with extra steps. And if your logs contain data you would not paste into a third-party tool, fix that before you evaluate any labeler at all.

The repo I kept is barely a hundred lines, and the interesting file is the test rather than the prompt. If you want to reproduce the harness, the free model access was enough to run the whole loop end to end before I ever touched a deployment.