Across 116 official MCP servers, one 2026 study found agents reach a median of only 19% of available operations. That study spans servers of every kind, not media APIs specifically, but media APIs are one of the cases where the same gap bites hardest, and it is the gap to check first if you are building or evaluating one for AI agents. The other 81% is left out not because it breaks, but because it was never designed to be called by a machine. An AI agent media API is the contract between an autonomous agent and a media-processing backend. It is the surface the agent calls to resize images, encode video, run diffusion inference, or extract captions without a human in the loop. That gap is structural, and closing it comes down to five specific design decisions. The agent primitives spec covers each one in detail.
TL;DR
- Across 116 official MCP servers of every kind, agents expose a median of 19% of available operations. Media APIs are an especially affected case, and the design choices below close most of the gap.
- Agents need strict JSON Schema responses with
additionalProperties: false. A major AI lab's own evals show 100% schema adherence with strict mode versus under 40% on the same JSON-schema benchmark without it. - Long media operations require async-first job semantics: a durable task handle, a poll mechanism, and five lifecycle states (MCP Tasks, 2025-11-25 spec).
- RFC 9457 Problem Details (
application/problem+json) is the floor for machine-recoverable errors. A plain HTTP 500 gives an agent nothing to branch on. - From 2 August 2026, EU AI Act Article 50(2) requires AI-generated content marked in a machine-readable format. Penalties reach up to €15,000,000 or 3% of worldwide annual turnover under Article 99 Tier 2.
Most media APIs were built for humans, and an agent-ready surface is a different design
The gap between a human-friendly API and an agent-ready one is structural, not superficial. In one 2025 industry survey of the API landscape, 24% of developers said they actively design APIs with AI agents in mind, which leaves a lot of headroom. Most production API surfaces are still tuned for a developer reading documentation and making deliberate choices, rather than for a planner model running a tool-selection loop at inference time.
Before walking through the gap, here is the whole checklist an agent actually needs from a media API, at a glance.
The 81% that gets left out is left out on purpose
The load-bearing number in that gap comes from the REST-to-MCP study (arxiv 2507.16044). Across the 116 official MCP servers it analysed, 88.6% are fully or partially REST-backed, and each server exposes a median of 19% of the operations available on the underlying API. That is a deliberate pattern, not an accident. Builders wrapping REST APIs for agent use leave out most of the surface on purpose, because much of it is too granular, too ambiguous, or too stateful to be called reliably by an LLM planner.
A media API is where this filter bites hardest
For a media API, this filter bites especially hard. A typical image-processing REST surface has separate endpoints for resize, crop, convert, watermark, and caption generation, each with its own parameter vocabulary. An agent planner facing that surface has to select the right endpoint, translate its intent into the right parameters, and handle a different error schema per endpoint. Each extra branch is another place a call can go wrong, so the operations that survive into the agent surface tend to be the ones that were clear enough to call reliably.
The implication is direct. A media API built for agents is not a subset of the human-facing surface. It is a reorganisation.
Strict JSON Schema is the floor that makes a response parseable every time
An agent cannot recover from a response it cannot parse, and it cannot reliably call a tool whose input shape is ambiguous. This is why strict structured outputs are not an optimisation but the baseline requirement for any AI agent media API.
Strict mode takes schema adherence from under 40% to 100% on the benchmark
One major AI lab's function-calling specification requires additionalProperties: false and all fields marked required for strict mode. Optional fields use a type: ["string", "null"] union rather than an absent key. That lab's own evals show strict structured outputs reaching 100% adherence on a complex JSON-schema benchmark, where the same model family without strict mode scored under 40% on that benchmark. The figure measures schema adherence on that eval, not a universal call-failure rate, but the direction is clear: strict mode is what makes the response shape dependable. A leading model provider's API takes a complementary step, generating a unique id for every function call so results map back deterministically even when parallel tool calls are in flight.
Consolidation is the flip side of a parseable schema
The same discipline that makes a response parseable is what makes the request surface consolidatable. The diagram below shows what that looks like spatially: many granular REST endpoints on the left, one consolidated agent tool on the right. The 19% survival rate is not loss. It is regrouping.
Fewer tools also means fewer tokens and higher accuracy
Tool consolidation compounds the benefit. A leading AI lab's 2026 advanced tool-use research reported that a tool-search approach cut tool-definition token usage by 85% while preserving 95% of the context window, and lifted its frontier model's accuracy from 49% to 74% on internal tool-use evaluations when applied to large tool libraries. The takeaway is practical. A consolidated transform_media tool with an action parameter costs fewer tokens per call and tends to be selected more accurately than six granular endpoints competing for the planner's attention. Consolidating the request surface this way also sets up deterministic polling and error handling, which the sections on async semantics and RFC 9457 errors cover below.
Agents cannot wait on the phone: async-first job semantics
Synchronous HTTP was designed for requests that complete in milliseconds. Video encoding, diffusion inference, and batch image transforms are not those requests. A connection that times out mid-operation gives an agent nothing to recover from.
Submit, get a handle, poll for a terminal state
The correct pattern is submit-and-poll. The agent submits a job, receives a durable task handle, and polls for a terminal state. The MCP Tasks extension (2025-11-25 spec revision) is the canonical example. A task handle carries a taskId, status, ttl, pollInterval, and timestamps. The five lifecycle states are: working, input_required, completed, failed, and cancelled. The tasks/get method is authoritative for polling. tasks/result blocks until a terminal state is reached.
A major lab's background mode follows the same submit-and-poll model
A major AI lab's background mode for its responses API follows the same model. Setting background: true returns a polling handle keyed by response ID, with queued, in_progress, and terminal states. Data retention runs roughly 10 minutes, cancellation is idempotent, and the mode is not compatible with Zero Data Retention projects. Both designs share one constraint, which is that the agent must not hold an open HTTP connection for the duration of a long operation.
Idempotency at the submission layer is standard practice. Treating task submission as idempotent means a network retry does not queue a duplicate encode job, a real concern when encoding runs cost money and time.
Errors must be machine-actionable, not human-readable
An HTTP 500 with a prose message gives an agent nothing to branch on. It can retry the request blindly, escalate to a human, or abort the task. None of those is the right default. The agent needs to know the error class before it can decide.
RFC 9457's type URI is the key an agent branches on
RFC 9457 Problem Details (published July 2023, superseding RFC 7807) is the standard floor. The content type is application/problem+json. The five standard fields are: type (a URI identifying the error class), title (human-readable summary), status (HTTP status code), detail (instance-specific explanation), and instance (a URI for this particular occurrence). The type URI is the machine-parseable key. An agent can branch on https://api.example.com/errors/rate-limit-exceeded without parsing prose.
Extension fields let one error serve agents and humans
Extension members carry recovery hints. A retry_after field tells the agent when to try again. A suggestions array proposes corrective parameter values. Because RFC 9457 requires that unrecognized extension members be ignored, adding agent-specific fields does not break existing human-facing clients. The same error response serves both audiences.
Recoverable errors are what keep long tool-call chains stable
A 2026 production MCP study by Srinivasan (arxiv 2603.13417) identifies structured error semantics as one of three missing production primitives alongside identity propagation and adaptive tool budgeting, drawing on real enterprise deployment experience. A separate martingale analysis of MCP tool-call chains (arxiv 2602.13320) found that error in sequential tool calls accumulates gradually, with high-probability deviations bounded by O(sqrt(T)) rather than compounding exponentially, and that re-grounding roughly every nine steps is enough for error control. Recoverable, machine-actionable errors are how chains stay within that bound. See the audit-trail schema in /docs for a worked example of how governance systems enforce this discipline.
Agent identity and provenance are not afterthoughts
A static API key tells you nothing about who is acting
A static API key has no scope, no budget rail, and no delegation chain. It cannot tell you which agent made a call, on behalf of which user, within which workflow. For a media API that may process hundreds of assets per hour on behalf of many different users, that gap is an audit and compliance liability.
Agent-native auth carries a delegation chain and a spend rail
Two complementary mechanisms address it. On the identity side, an IETF draft (filed May 2025) proposes an act claim in OAuth 2.0 tokens and a requested_actor parameter in authorization requests, creating an auditable delegation chain from user to client application to AI agent. A major cloud provider's agent-identity service is a shipping implementation that handles both inbound auth (who may call your agent) and outbound auth (what the agent may call on the user's behalf), with a secure token vault encrypted at rest. On the spend side, a managed AI gateway opened dollar-denominated spend limits in beta on 5 June 2026, tracking and capping actual spend per agent ID, model, or provider, with fixed or rolling time windows.
Provenance has a hard deadline: 2 August 2026
On the provenance side, the deadline is imminent. From 2 August 2026, EU AI Act Article 50(2) requires AI-generated content to be marked in a machine-readable format detectable as artificially generated. A grace period to 2 December 2026 covers pre-existing systems. Violations fall under Article 99 Tier 2, with penalties up to €15,000,000 or 3% of worldwide annual turnover.
The Act mandates machine-readable marking but does not prescribe a specific technique. C2PA (Coalition for Content Provenance and Authenticity) manifests are one way to satisfy the marking requirement, and recent C2PA revisions (2.3, published February 2026) have extended the specification's coverage. Depending on deployment context, additional mechanisms such as watermarking or logging may also be needed, so review implementations against the Commission's draft guidelines published in May 2026.
Provenance has to be embedded at generation time
A media API that does not embed provenance at generation time forces every downstream caller to retrofit it. Layermetry treats provenance embedding as a generation-time primitive precisely because retrofitting at scale is not a workable solution. For a deep dive into how identity and provenance interact in complex workflows, see the governance controls reference.
FAQ
Do AI agents need synchronous or asynchronous media API calls?
Async-first. Long media operations such as video encoding and AI generation exceed normal connection timeouts, so the agent should submit a job, receive a durable task handle, and poll for a terminal state rather than holding an open connection. The MCP Tasks extension (2025-11-25 spec) defines non-terminal states such as working and input_required before a terminal completed, failed, or cancelled result, and a major AI lab's background-mode API follows the same submit-and-poll model with queued and in_progress states.
How should a media API signal errors so an AI agent can recover on its own?
Use RFC 9457 Problem Details with the content type application/problem+json. The type URI acts as a machine-parseable error class the agent can branch on, and extension members can carry fields like retry_after and suggestions. Because unrecognized extensions must be ignored, you can add agent-specific recovery hints without breaking existing clients, letting the agent adjust parameters and retry deterministically instead of re-prompting a person.
What does EU AI Act Article 50 require for AI-generated media by August 2026?
From 2 August 2026, Article 50(2) requires AI-generated content to be marked in a machine-readable format detectable as artificially generated, with a grace period to 2 December 2026 for pre-existing systems. Violations fall under Article 99 Tier 2, with penalties up to 15,000,000 EUR or 3 percent of worldwide annual turnover. The Act mandates machine-readable marking but does not prescribe a single technique, so C2PA manifests are one way to satisfy it rather than a guaranteed complete solution on their own.
These five requirements are a stack: remove one and the whole surface breaks
The five requirements in this post form a stack. Strict JSON Schema makes the response parseable. Tool consolidation makes the request surface usable. Async job semantics make long operations reliable. RFC 9457 errors make failure recoverable. Agent identity and provenance make the system auditable and legally defensible. An API missing any one of these will be filtered out of the 19% agents can reach, or it will surface in a production incident at the worst time. The design work is front-loaded, but so is the payoff: once these primitives are in place, agents can call the surface autonomously and recover without human intervention. If you want to see exactly what that looks like wired together, the primitives reference in /docs maps each pattern to a concrete implementation.