Jade Emmanuel wrote a good piece in September about the four hidden decisions your agentic system is already making. The argument is that every agent system has already decided what the task is, what happens next and when to change course, what counts as true, and what determines whether execution continues, and that most teams made those decisions by default rather than on purpose. Her advice is to find them and make them explicit before adding another agent or another layer. I’d give the same advice. It is roughly what my six tests are for.
I’d add that making those decisions explicit doesn’t account for everything the system does. Interacting parts can produce behaviour none of them was instructed to produce, so we still need to observe the system as a whole. Some of the most useful ways I’ve found to think about that came from reading biology.
I see small versions of this most weeks. An agent on a cheaper model gets drawn into some minor thing it noticed along the way and quietly forgets what it was asked to do, until I pull it back onto the task. Another tells me a fact with complete confidence, based on something it cached earlier, and says it has verified it when it hasn’t. When I show it evidence to the contrary I get an apology and no fix, which is at least a very human combination. You can put all of that down to the model, and cheaper models do it more. But what produced the behaviour was the run as a whole: the goal, the context it had accumulated, what the tools returned and the order it all arrived in. Making Emmanuel’s four decisions explicit wouldn’t, on its own, guarantee that the run stayed on track.
What I mean by emergence
The word gets used loosely, so I want to clarify how I’m using it. A property is emergent when it belongs to the system and not to any component, and when you couldn’t have read it off the components however carefully you studied them. A traffic jam isn’t a property of any car in it. The physicist Philip Anderson put it as “more is different” in 1972, and the phrase has stuck because it’s exactly right. Add enough interacting parts and new rules show up at the higher level that aren’t contradicted by the lower level, and aren’t predicted by it either.
Emergent behaviour has causes, and you can study it, and sometimes you can steer it. What you can’t do is prevent it by being more thorough about the components, which is the instinct almost every engineering team has when a multi-agent system surprises them. Or even any distributed system! Add a check, tighten the prompt, write the decision down, all of which is useful, but none of it solves for emergence.
There’s a recent paper worth reading if you want the formal version of this applied to our world. Daniel Russo’s More Is Different: Toward a Theory of Emergence in AI-Native Software Ecosystems argues that multi-agent software fails in ways the standard theories of software evolution don’t explain, and identifies three emergent failure modes that come from agents interacting rather than from any one of them being faulty: architectural entropy, cascade failures, and comprehension debt. He concludes that monitoring the whole ecosystem has to become the main governance mechanism. The decisions are made at component level and the failures show up at system level.
What I remember from the biology
The book that started it was M. Mitchell Waldrop’s Complexity, his account of the early years of the Santa Fe Institute, which I read something like twenty years ago. Possibly twenty-five, which I’m choosing not to dwell on! Most of what follows came from pulling on the threads it left, Stuart Kauffman’s work on self-organisation in particular, and a surprising amount of it has turned out to be about agents.
The thing that stayed with me is that nothing in a developing embryo is following a blueprint. There is no gene for a hand. There are genes that produce gradients of signalling molecules, and cells that respond to the concentration they find themselves in, and the hand is what happens when several thousand cells each do something local and reasonably simple in the presence of one another. The shape is a property of the interaction. Change the gradient and you get a different hand, or no hand, without any gene having “decided” that.
The second thing is how such a system stays reliable when its parts are unreliable. Biology is full of noise. Gene expression is stochastic, proteins misfold, signals arrive late. And yet the hand comes out right almost every time. The mechanisms work as design patterns as much as biology. Redundancy, where more than one component can do the job. Degeneracy, which is subtler and more interesting: structurally different components that can do the same job under some conditions and different jobs under others, so the system has options rather than spares. Canalisation, which is Waddington’s word from the 1940s for the way a developmental path is buffered so that quite large perturbations still end up at the same outcome. And feedback everywhere, so that what a cell does next depends on what its neighbours just did, which is how you get stability without a controller.
The third thing is how much of biology works by generating variation and selecting from it, rather than by specifying the answer. It’s the one that made me take notice when I started directing agents seriously. Microtubules in a cell grow out in essentially random directions and the ones that hit something useful get stabilised while the rest are dismantled. The immune system makes an enormous variety of antibodies in advance of knowing which pathogen will turn up, and amplifies the ones that bind. Marc Kirschner and John Gerhart call these exploratory processes, and the point of them is that the system does not need to know the right answer in advance. It needs a cheap way to try many answers and a reliable way to keep the good one. A coding agent working towards a goal with a test suite does something similar: try an implementation, check it and keep or discard it.
Reading the four decisions through that
Go back to Emmanuel’s four decisions with the biology in hand and each one looks a bit different.
Task. Development doesn’t have a task in the sense of a specification. It has constraints, gradients and a definition of done that is checked locally, cell by cell, not globally. The implication for an agent system is that the task is better held as constraints that must remain true than as a description of the outcome, which is in fact what Emmanuel’s first diagnostic question asks for. I’d just add that the constraints need to be checkable by the parts, not only by the person at the top.
Routing. In an agent system there is always a router, but what determines the route in production is the interaction between the router and the signals it’s fed, and the signals are often ones nobody designed as signals. Confidence scores, latencies, the order results happen to arrive in. If you haven’t looked at what your routing actually responds to, as opposed to what you wrote in the routing logic, you don’t know how your system routes.
State. Termites build complex structures without each one holding a plan for the whole structure. Each one responds to the structure the others have already built, so the environment is both the state and the message. Grassé called this stigmergy in 1959 and it’s the single most useful word I know for understanding multi-agent systems, because it describes what your agents are actually doing the moment two of them touch the same file, the same database, or the same ticket. The shared artefact becomes the coordination mechanism whether you intended it to or not. Emmanuel’s question about what the system treats as authoritative is the right question. The biology adds that whatever is most visible to the most agents will become authoritative in practice, so you had better make the thing you want to be authoritative the most visible thing.
Control. Multicellular life has a brutal and elegant answer to the question of whether execution continues. Cells kill themselves. Apoptosis is programmed, it’s triggered by signals the cell can read locally, and it runs without permission from anywhere. Checkpoints in the cell cycle are the same idea at a smaller scale: the process stops at a defined point and only continues if specific conditions are met. Control in biology is distributed, cheap to invoke, and biased towards halting. Most agent systems I’ve looked at have control that is centralised, expensive to invoke, and biased towards continuing. I think that’s the biggest reason emergent failures in these systems cascade rather than fizzle.
What to do about it
Making the four decisions explicit is the component-level work, and you should do it. The biology says the next thing is to accept that the system will still have behaviour you didn’t decide, and to build for that instead of being surprised by it. Concretely, that means putting your effort into the mechanisms that keep a noisy system reliable: redundancy, degeneracy, buffered pathways and local feedback. The other half is observing the system at the level where the behaviour lives, not only at the level where the decisions were made. Russo calls that level the ecosystem; for us it’s the population of runs.
Distributed systems went through this already. Observability as a discipline exists because production systems do things you can’t predict from their components, and the only way to find out what is to capture what the whole system did and ask questions of it afterwards. OpenTelemetry became the vendor-neutral way to do that capture, and its conventions now cover agents: a span per agent invocation, one per tool call, with arguments and results where privacy allows. The population of runs is capturable today. What’s missing is the habit of looking at it as a population rather than debugging one run at a time.
Leaders already know how to do this
The kind of direction this asks for is one leaders already know well and engineers tend to resist. You don’t run an organisation by specifying every decision each person in it will make. You set constraints, you make the important state visible, you build in cheap ways for people to stop and escalate, and you watch the aggregates. Most of what the organisation then does, nobody decided, and trying to engineer that away misses what an organisation is. The same goes for an agent system.
The mRNA thought, which I’m not sure about yet
This last bit is a hunch rather than an argument, and I’d happily be talked out of it.
A cell holds its instructions in DNA, which is stable, long-lived and expensive to change. When it needs something built, it transcribes a short-lived copy, mRNA, which is translated into protein and then degraded, often within minutes. Almost all of the fine control over how much of a protein gets made happens at that middle layer. The cell doesn’t edit the DNA to make more of something. It makes more transcripts, or makes them last longer, or stops translating them.
I keep wondering whether that’s the shape software is heading towards. The stable layer is the intent: the spec, the constraints, the tests, the things that must remain true. The transient layer is whatever the agent produces on the way to running code, and increasingly the running code itself. If the code can be regenerated from the intent cheaply and reliably, then it starts to make sense to treat it as a transcript rather than an asset. Rather than patching the transcript, you fix the intent and transcribe again. The control goes where the cell puts it, at translation, which in our world is the job of the control plane around the agent.
The other half of the thought comes from the mRNA vaccines. What was remarkable about them was that they shipped the instruction and let each recipient’s own machinery build the thing, locally, in context, and then clear it away. The equivalent in our world would be shipping the skill rather than the agent, and letting each environment build the running system from it against its own constraints and its own data.
I don’t know yet how useful the mRNA analogy will be. Local checks, feedback and observation of the whole system are useful now, though. Making each agent’s decisions explicit helps, but we still need to watch what happens when they work together.