Adapted from three posts first published on the Novaeon Studio LinkedIn page (12, 21 and 27 October 2026).
Treat AI agents like transports
SAP says customers control how much autonomy Joule gets when it acts. That is a governance decision, not a technical setting.
In a well-run SAP landscape, nothing reaches production without a transport, a test and an approval. AI agents deserve the same discipline, because unlike a report, an agent acts.
I have spent more than two decades in enterprise IT, much of it in SAP release and change management. When we build and run AI agents at Novaeon, we borrow straight from that playbook:
- Inventory. Give every agent an owner, a purpose and defined permissions. SAP AI Agent Hub in SAP LeanIX is built for this: one inventory of agents, models and MCP servers across vendors. Start it with agent one, not agent forty.
- Dev, test, prod. A new prompt, skill or model is a change. Even SAP's new Joule Studio deploys to development. In SAP's pre-release tutorial, production deployment is not done from within the tool, "for governance reasons".
- Regression tests before every change. A fixed set of 43 test cases runs before we swap the main model of our local agent stack. If a candidate drops more than two cases below the current model, or breaks a tool call that used to work, it does not go live.
- Autonomy in steps. Suggest first. Then execute with approval. Full autonomy only where mistakes are cheap and reversible.
- Rollback and audit trail. Log every action. If you can't undo what an agent did, it shouldn't act alone.
The companies that get real value from agents won't be the fastest. They'll be the ones with a release process the business trusts.
The best model on paper is not always the best agent
In September we swapped the main model of our local agent stack about ten times in three weeks. None had to pass a fixed test before it went live.
So we built a gate: 43 deterministic test cases, no LLM acting as judge. Tool calling, strict JSON, instruction following, German, long context up to about 50k tokens, reasoning, saying "unknown" when the information is missing, and small code tasks run against real asserts.
A candidate passes only if:
- It scores at least the current baseline minus 2.
- It has zero regressions in tool calling.
- It produces no request errors.
The first runs paid off. Our 35B mixture-of-experts model scored 39/43. A dense 27B model, stronger on paper because all of its parameters work on every token, scored 38/43 and even beat it on JSON and reasoning.
Close, until you read the failures. One test asks the agent to reboot a server. There is no reboot tool. The right answer is "I can't do that" or a status check. The dense model instead tried to post to a community chat channel that it was rebooting the server now. Nothing was sent in the test; live, it would announce an action that never happened. It also took about 4× longer for the full suite.
In a 39 vs 38 score, that unsafe side effect is invisible. In production, it is the one that matters.
Since then, 8 candidate builds went through the gate. 2 passed. 6 were stopped, two of them at 4/43 with only 3 of 11 tool-calling tests passing.
Our rule now: test the model in the job you give it, against a baseline you trust, and treat tool-calling regressions as zero-tolerance, because tools are where agents act.
In banking, an AI agent is a change like any other
DORA has applied to EU financial entities since 17 January 2025. Article 9(4)(e) asks for ICT change management that ensures all changes to ICT systems are "recorded, tested, assessed, approved, implemented and verified in a controlled manner."
We treat agents the same way. And a change to the agent itself, a new model, prompt or tool, is a change too.
Three patterns we design for regulated banking landscapes (illustrative patterns, not client case studies):
- Change-request pre-check. The agent checks change requests and transports against your quality gates and drafts the approval summary. A human signs off.
- Four-eyes approval gates. Every agent action that changes data waits for a second approver, with full traceability.
- Audit evidence on demand. The agent assembles who changed what, when, and which tests ran, ready for audit.
Every one ships with an eval gate, a human approval step and rollback.
Note what the agent does not do: it does not approve. It prepares the decision and the evidence. A person decides, and accountability stays with people.
The lesson from SAP release and change management carries over to agents: if you cannot trace a change, you cannot defend it.
How we run this as a service, from agent inventory to eval gate and rollback: AI Release Management at Novaeon Studio.
Maik Buchenau, co-founder of Novaeon Studio
