You wire up a media pipeline out of AI agents. One agent transcribes and pulls the highlights, another reframes the footage for each channel, a third color-grades to brand, a fourth writes captions and a thumbnail. You test each one on its own and each one is great. Then you run them end to end and the output is subtly wrong: the captions describe a shot that got reframed out, the color is off from what the brief asked for, the thumbnail is from a moment that no longer exists in the cut.
Here is what most teams run into. The agents are fine. The place it breaks is the seam between them, the moment one agent hands its work to the next. Full disclosure before we go further: Layermetry is one of the off-the-shelf layers that tries to hold state across that seam, so we will lay out both paths, building it and adopting it, and let you decide.
The seam is the failure point, not the agent
When people debug a flaky pipeline, they reach for the agent first. They swap a model, rewrite a prompt, add a retry. Sometimes that helps. Often it does not, because the thing that actually broke happened in the gap between two agents that were each doing their job correctly.
An agent only knows what you hand it
An agent is an LLM given hands, a model wired to tools it can call so it can do work, not just describe it. The part that is easy to overlook is that each agent is stateless. It does not remember the last run, and it does not automatically see what the previous agent saw. It knows exactly what you put in front of it and nothing else. So everything one agent learned, every decision it made and every reason behind it, has to be carried across the seam on purpose, or it is gone.
Context loss compounds at every handoff
That carrying is where it goes wrong, because the easy way to do it is to summarize. Agent one finishes, writes a paragraph describing what it did, and hands that paragraph to agent two. The summary is lossy by definition, so small misreadings stack up across each pass. A 2026 multi-agent failure playbook lands on the same conclusion from the other direction. It finds that reliability in these systems rarely breaks at the core algorithms, it breaks at the seams, where agents hand off tasks and coordinate (multi-agent orchestration failure playbook, 2026). The failure is in the transfer, not the agent.
The same erosion hits the asset itself
Media makes this worse, because the asset degrades too. Every time a clip is re-encoded or re-rendered on its way through a pass, you lose a little fidelity. This is the idea of generation loss. Repeated lossy compression and decompression accumulate artifacts, so quality drops progressively with each successive pass, which is why the standard advice is to decode once and make all edits on that copy rather than re-encode at every step (Generation loss, Wikipedia). So you get a double erosion at the seam. The context degrades because it was summarized, and the pixels degrade because they were re-processed.
Statelessness plus no shared contract is what breaks the handoff
So why is this gap so reliably the weak spot? Three things line up against you, and they are structural, not bad luck.
No memory means nothing carries unless you carry it
The first is statelessness, which we covered. Even the teams building the most advanced systems treat this as a first-class problem. In its write-up of a production multi-agent research system, a leading AI lab notes that subagents each operate with their own context window, and that the lead agent has to deliberately save its plan to external memory, because if the context window exceeds 200,000 tokens it gets truncated and the plan would otherwise be lost (a leading AI lab, 2025). Their blunt summary of the failure mode is that agents are stateful and errors compound. Nothing carries across the seam by default. You have to make it carry.
No shared place to point at means everyone passes copies
The second is the lack of shared, addressable state. If there is no single place where the asset and its history live, every agent has to be handed its own copy of everything, and every handoff is another lossy transcription. The fix the standards world landed on is to stop stuffing everything into each message and instead expose the work as resources an agent can request on demand. That is a core idea in the Model Context Protocol, where resources are context and data the model fetches when it needs them, rather than content copied into every prompt (Model Context Protocol spec). A shared address means agents point at the same asset instead of mailing each other photocopies.
No typed contract means the next agent has to guess
The third is the missing contract. When agent one returns prose, agent two has to parse and interpret it, and interpretation drifts. This is the gap that agent-to-agent standards exist to close. The Agent2Agent protocol, donated to the Linux Foundation in 2025 to give it neutral governance (open-protocol announcement, 2025), is built around exactly this. As one 2026 protocol guide puts it, the only thing A2A cares about is the contract between agent A and agent B, where a Task carries a goal and typed artifacts that the responding agent produces (A2A protocol guide, 2026). A typed artifact is the opposite of a paragraph. The next agent reads it, it does not guess at it.
A good handoff passes a reference, not a retelling
Put those three together and the recipe for a handoff that holds is clear. The result one agent returns has to be the exact thing the next agent can read.
Shared addressable state, so nobody hands over a copy
Give the pipeline one place where the asset and its edit history live, and a stable address for it. Agents then operate on the same object and pass a reference to it, not a re-encoded duplicate. The asset stops degrading a little at every pass, because it is not being re-transcribed at every pass. This is also why production systems lean on external memory and progress files rather than trying to cram the whole history into the next prompt (a leading AI lab, 2025).
Typed contracts, so the result is machine-readable
Make each agent return a structured, typed result, not a description. A typed result is something the next agent can act on directly: which regions were reframed, the exact color profile applied, the timestamps that survived the cut. When the output is typed, the receiving agent reads facts instead of inferring them, and the drift that comes from re-interpretation disappears. This is what the artifact in an A2A Task is for, and what a resource in MCP gives you (Model Context Protocol spec).
Observability, so you can see what crossed each seam
Last, you need to see the seams. When a pipeline goes subtly wrong, you want to look at exactly what crossed from agent one to agent two and find where the meaning got lost. Without tracing across handoffs, you are back to swapping models and hoping. And this is not a one-time setup. Change one agent or one model and the whole flow can behave differently, so the observability to catch it is a standing cost.
Build the layer yourself, or adopt one that already holds the seam
Here is the honest decision, because there are two real paths and each has a genuine cost.
You can build the shared-state layer yourself. You get total control and no dependency on anyone, and if the pipeline plumbing is itself your product, this is the right call. The price is that shared addressable state, typed contracts between every pair of agents, and the cross-seam observability are real infrastructure, and once you build them you maintain them forever, including every time a model or an agent changes underneath you.
Or you adopt a surface that already holds state across the seam. Something like Layermetry keeps the asset and its edit history in one addressable place, returns typed results an agent can read, and lets the agents in your pipeline point at the same object instead of re-summarizing it down the line. It also ships native editing tools and lets you ship your own alongside them, so it is extensible rather than a closed box. The trade is speed and lower maintenance against a dependency on that layer.
Either way you are not betting on a fringe idea. The whole industry is converging on shared, addressable state and typed contracts between agents, from the open agent-to-agent standard now under neutral governance to the protocols that expose work as resources instead of copies. The only real question is how much of that plumbing you want to own.
FAQ
Why does my multi-agent pipeline fail when each agent works fine alone?
Because the failure usually lives in the handoff, not the agent. Each agent is stateless, so it knows only what you pass into it. When the next agent gets a summary instead of the real state, small assumptions and approximations stack up across the seam. Industry write-ups in 2026 describe most agent failures as orchestration and context-transfer problems at handoff points, not model capability problems. Fix the seam before you swap the agent.
How do I stop context and asset quality from degrading between agents?
Stop re-describing the work and start pointing at it. Give the pipeline one shared, addressable place where the asset and its edit history live, and have each agent return a typed result the next agent can read directly instead of a paragraph it has to re-interpret. That way the handoff passes a reference, not a lossy copy, so the asset stops degrading a little at every pass.
Should I build the shared-state layer myself or adopt one?
It depends on whether that plumbing is your product. Building it yourself gives you total control and no dependency, but shared addressable state, typed contracts, and the observability to see what crossed each seam are real, ongoing infrastructure you then maintain forever. Adopting a surface that already holds state across the seam is faster and lower-maintenance, with a dependency on that layer. Pick based on where you want your effort to go.
The seam is the design decision
Here is the whole thing in one line: the agent you pick is rarely what makes your pipeline reliable. The seam between your agents is. Give that seam shared addressable state and typed results one agent can read and the next can act on, and a chain of agents holds together. Leave it as a summary mailed from one stateless agent to the next, and the work erodes a little at every pass until the output is quietly wrong.
That is a choice about your tooling, not a model you wait for. Build the layer that holds the seam yourself if that plumbing is your product, or adopt one and spend your effort on what makes you different. If you want to see what the off-the-shelf version looks like, the operations and how state is held are in our docs.