I gave a talk at In The Loop called The Agent Is Half the System (it’s on my speaking page too), and the short version is that the thing you’re actually shipping is an agent someone can run in production, and the control plane around it is what decides whether they get to keep it. This post is the rubric from the talk, with enough method that you can run it on whatever you’re already using or evaluating, including n8n.
I scored ten platforms against the six tests in September 2026 to check the talk’s claims were valid, and I’m publishing the questions and the method rather than my scores. The AI world, and related docs, change often enough that my scores would be out of date quickly, so you’re better off running it yourself.
The six tests
Each capability reduces to one question with a yes-or-no shape. If you find yourself writing a paragraph in answer, the honest score is probably partial.
| Capability | The test |
|---|---|
| Identity and access | Can you scope access per tool call, or only per agent? |
| Spend governance | Can you cap spend per run and watch it in real time? |
| Step-level policy | Can you update policy centrally without redeploying every agent? |
| Observability | Can you get traces into your own backend, or only their dashboard? |
| Audit and replay | Can you replay a decision deterministically six months later? |
| Human oversight | Can you require approval at a named step, or only at the boundary? |
The scale, and why partial isn’t a criticism
There are four scores, and the difference between the first two is the one worth slowing down for.
Yes. The platform provides it and enforces it, and a vendor page says so. Nobody on your team has to write the control; you configure it and the platform refuses to do the thing you said no to.
Partial. You can get there with your own code or config on top, or the platform does a coarser version than the question asks: per agent instead of per tool call, a monthly budget instead of a per-run cap, approval at publish time instead of mid-run, a webhook of outcomes instead of traces. Most useful engineering lives here. The difference from a yes is that your team builds the control and keeps it working.
No. The vendor’s own documentation describes something that fails the test, or describes the absence. “In-flight requests are allowed to finish when the balance runs out” is a documented no on spend caps. “Replay re-executes nodes and may return different results” is a documented no on deterministic replay.
Undocumented. You couldn’t find a vendor page either way. Record it as that, not as a no, and then ask the vendor for the page, because your auditor will want to see it too.
What counts as evidence
I only cited pages the vendor owns: product docs, help centre, changelog, release notes and their official blog. Analyst reports, community posts, partner blogs and conference talks (mine included) were useful for finding the right page, but never as the source, so anyone can check a cell against the vendor’s own words. I also noted any gate on a feature, like an enterprise tier, a paid add-on or a preview label, because a yes that’s only in preview is a different answer when procurement asks.
I scored what’s shipping today. Where a product had been renamed, deprecated or migrated, I scored whatever the vendor currently points customers at, and kept the old name alongside it.
Where the vendor’s word and the test’s word differ
These are the places where I found myself about to give a yes and the doc said otherwise.
Replay. Almost every platform has a button called replay, retry or time travel, and nearly all of them re-run the step, which calls the model again and may produce a different answer. The test asks whether you can re-materialise the original decision from stored inputs, tool results and model outputs without calling the model at all. Read the doc for the word “re-execute”, and check the retention period while you’re there, because a replay that’s only possible for thirty days doesn’t answer a question asked in month six.
Spend. Account budgets, billing alerts, monthly credit pools and plan quotas all exist, and none of them are a per-run cap. The test asks whether a single run has a ceiling and stops when it hits it. Check the unit as well: a cap in tokens or in list-price dollars is real, and it’s still not your contracted spend.
Identity. A different credential per tool, chosen when you build the agent, is design-time configuration. The test asks whether something evaluates each tool call at runtime against who’s asking and what they passed. Look for words like “intercepts”, “evaluates each request”, “per invocation”.
Policy. A rule compiled into the agent’s code or config changes when you redeploy. The test asks whether the rule lives somewhere central and applies to agents that are already running. Watch for docs that say a policy template applies to new agents only.
Observability. A dashboard, however good, still keeps your traces in the vendor’s product. The test asks whether you can point traces at your own collector, ideally in OpenTelemetry. CSV downloads and terminal-outcome webhooks are partial.
Oversight. Approval before an agent is published, or before a final output goes out, is a boundary gate. The test asks whether you can pick a step or a tool mid-run and have the run pause there.
Running it yourself
One platform takes an hour or two if you know its docs, and a morning if you don’t. Score the six cells, write the one-sentence justification that quotes the doc, paste the URL and the date, and note the gate. Do it for the thing you run today before you do it for the thing you’re evaluating, because the comparison is only useful if your own row is honest. When I scored my own row I gave it two yeses, two partials and two nos, and the two nos are on the columns the talk argues matter most.