Sharing this because I wish someone had warned me before it happened. Had an agent running in production for about six weeks.
Working fine, real users depending on it daily. Made what felt like a small prompt tweak, tested it quickly, pushed it.
But three hours later it was producing wrong outputs for almost half the queries.
Something in the updated prompt was interacting badly with one of its tools in a way I hadn't caught in testing.
The frustrating part wasn't really fixing it. It was figuring out what had actually changed.
I had overwritten the old prompt.
No proper version history for it, no easy way to see the previous state, nothing I could just roll back to. Ended up digging through old Slack messages and trying to piece together what the prompt looked like before.
Once I figured out the change, the actual fix took maybe twenty minutes. What stuck with me was that I would never normally work like this with code.
If I change something, there's a commit sitting there. I can see the diff, go back to an older version or just revert it.
But with the agent, I had somehow treated the prompt and config as if they weren't really part of the application. Since then I've been thinking that maybe they should be.
If the prompt, tools, memory/config and other settings can all change how an agent behaves, it makes sense that those things should probably have some kind of version history too.
I've come across a few projects looking at this from different angles, including GitAgent from Lyzr for managing agent definitions through a more version-controlled workflow.
There are also tools like Langfuse that help with the other side of the problem, figuring out what actually happened when an agent ran.
Still trying to figure out what the right setup is here.
Want to know how are you handling this for agents that are actually running in production?
Do you version the whole agent setup, just the prompts, or something else?
I'm more interested in what you're actually doing today than the “ideal” setup.
[link] [留言]