04 · Offer
AI Release Management
ChaRM for AI agents: every change versioned, tested, approved — and reversible. An agent changes behaviour when its prompt, model, tools or data change. We give your agents the discipline of SAP change and release management: an inventory, versions, a fixed eval gate, a risk rating, an approval — and a way back.
The problem
Agents rarely fail on day one. They fail after the third model swap, a changed prompt or a new tool nobody tested — and then nobody can say which version ran, who approved it or how to go back. In a well-run SAP landscape nothing reaches production without a transport, a test and an approval. Agents act, so they deserve the same discipline.
What we deliver
- An inventory of every agent, model, skill and MCP server — registered, owned and versioned
- A fixed eval gate every change must pass: deterministic test cases, prompt-injection and tool tests, weighted by severity — one regression in a critical category blocks the release
- A risk rating per agent, including EU AI Act classification and documentation a works council can review
- A release record for every change: what changed, test evidence, decision, named approver and rollback target
- An append-only audit log: each entry carries the hash of the one before, so editing or deleting an earlier entry shows up on verification
- Calibrated LLM judges wherever a model grades output: their verdicts count only once they agree closely enough with human ratings
- Rollback: a known-good release to return to, tested before you need it
- Operations: session traces, cost per action, drift monitoring and a release calendar — on your own Git-based pipeline or designed for SAP Cloud ALM and AI Agent Hub
- Trace checks: a delegated agent task counts as done only when its recorded tool calls show the required steps succeeded — not because the agent says so
How it runs
Audit → Build → Retainer: the same line for every offer, tailored to this one. You decide after each phase whether to continue.
- 01
Audit
An inventory of what runs today — agents, models, prompts, tools, owners — and a gap check against release discipline: what is versioned, tested, approved and reversible.
- 02
Build
The release process itself: the eval gate with your real cases, risk rating, approval workflow, audit trail and rollback — wired into your pipeline.
- 03
Retainer
We run the gate with you: every model swap and prompt change goes through it, and the test set grows with every incident.
Matching industries
Where this offer meets ground we know from the inside.
Questions
How do you know an LLM judge is right?
We measure it before we trust it. Where an LLM grades output, we compare its verdicts with human ratings of the same cases using Cohen’s kappa, a standard agreement score that corrects for chance. Its verdict counts only at 0.7 or above. We are calibrating our own judge this way now; there is no result to report yet. Our 43 deterministic gate cases do not use an LLM judge.
What does “AI release management” mean in practice?
Every change to an agent — prompt, model, tool, data source — goes through versioning, a fixed eval baseline, a risk rating and an approval, with a known-good release to roll back to. Like ChaRM, for agents.
Why is a model swap a release?
Because the same prompt behaves differently on a different model. In our own gate, one model under test tried to announce a server reboot it had no tool for. A fixed test set catches that before production, not after.
Do we need SAP for this?
No. The discipline comes from SAP change and release management, but the process runs on any stack — on your own Git-based pipeline or alongside your existing change management.
Does this help with the EU AI Act?
It lays the groundwork: an inventory, a risk classification per agent, documented tests and approvals, and an audit trail. Whether a use case is high-risk is assessed case by case; obligations for high-risk uses are scheduled to apply from 2 December 2027.
Talk it through in 30 minutes.
Bring one process and your stack. We come prepared and tell you honestly whether an agent pays off — and which offer is the right start.
Or write to us: office@novaeon.studio