What AI Agents Need to Run Media Edits Reliably: The Tool Contract Underneath

A transcode times out at the four-minute mark, your agent retries, and now you are paying for the same render twice. Or the render fails and all the agent gets back is a generic 500, so it has no idea whether to try a different format, skip the file, or ask a human. Anyone who has pointed an AI agent at real media work has hit some version of this. The model is smart enough to decide what edit to make. The thing underneath it, the part that actually runs the crop or the transcode, keeps failing in ways the agent can't recover from.

This post is about that underneath part. Not "here is a schema you should go build," but a tour of what the contract between an agent and a media operation actually has to handle, and why each gap on the list costs real money or breaks the agent's loop. Full disclosure: Layermetry is one of the off-the-shelf layers that ships this contract, so at the end we will lay out both paths, building it yourself versus adopting a layer, with the honest cost of each, and let you decide. This is a deeper companion to our piece on adding AI editing to your SaaS. That one covers the agent-drivable surface. This one zooms in on the single contract a media tool call has to honor.

Why a media edit is the hard case, not the easy one

The tool every AI tutorial starts with is a weather lookup: one quick call, an instant text answer, safe to run as many times as you like. A media edit is the opposite of that on every axis, and the gap is exactly where agents fall over.

A transcode is long-running, so a synchronous call times out

A real transcode is not instant. A major cloud provider's managed transcoding service moves a single job through at least three internal phases: analysing the input, doing the codec work, and uploading the output. A four-minute source clip at high bitrate can spend several minutes in the codec phase alone. An agent that calls and waits for the answer on the same connection will time out long before the file is ready, and a timeout looks, to the agent, like a failure worth retrying. Which leads straight to the next problem.

Retrying a media call charges you again

A weather lookup is safe to retry because asking twice costs nothing. A transcode is not. Submit the same render request twice with no deduplication and you get two jobs, two bills, and two output files that differ only in their timestamp metadata. The agent didn't do anything wrong, it just retried a call that looked like it failed. Without a guard built into the contract, every retry is a real charge.

A media job can half-succeed

A failed weather call is simply failed. A render is messier. A job can finish some output renditions while others error, for example the 720p H.264 version succeeds while an AV1 version fails on a source the codec doesn't like. A plain done-or-failed answer throws that nuance away. The agent needs to see per-output status to decide whether to retry just the failing rendition, ship the partial result, or stop. Without it, one bad output marks the whole job a failure and the good work gets discarded.

The result is a file, not a sentence

A weather call returns a string the model can read directly. A transcode returns an MP4 sitting at a CDN URL, sometimes with side-car files. The answer can't be the file itself, it has to be a reference the agent can hand to the next step. Get this wrong and the agent is holding a result it can't pass downstream.

Four properties, four ways the easy-case contract breaks. Here is the lifecycle a media-aware contract has to model instead.

The contract is one schema that works across every agent spec

Here's the encouraging part before the hard part. The three specs that dominate how teams expose tools to AI agents today all reduce to the same handful of fields, so the contract you define is portable, not three contracts.

Three specs, one shared core

One major AI lab's tool definition asks for three things: a name, a plain-text description, and an input_schema, which is a JSON Schema object. That lab's models also support a strict mode that guarantees the inputs always match the schema exactly. Another lab's function calling calls the same object parameters instead of input_schema, and in strict mode it asks you to mark every property required and forbid extra ones, so the model is guaranteed to produce conforming output. The Model Context Protocol (MCP), the open standard for connecting AI applications to external tool servers, calls it inputSchema and defaults to JSON Schema when you don't specify a version.

Different field names, same idea underneath. A contract written once carries across all three with little more than a rename.

What that contract looks like for a transcode

Here is the shape of a transcode_video contract, written to the strictest common subset so it survives all three specs. Read it as evidence of how much the contract has to carry, not a copy-paste recipe.

{
  "name": "transcode_video",
  "description": "Transcode a video file to a specified format and resolution. Returns a job handle for async polling. Non-idempotent by default; supply idempotency_key to deduplicate retries.",
  "input_schema": {
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "type": "object",
    "additionalProperties": false,
    "properties": {
      "source": {
        "type": "string",
        "format": "uri",
        "description": "Publicly accessible URI of the source video file."
      },
      "output_format": {
        "type": "string",
        "enum": ["mp4", "webm", "gif"],
        "description": "Target container format. mp4=H.264/AAC, webm=VP9/Opus, gif=palette-quantised."
      },
      "resolution": {
        "type": "string",
        "enum": ["3840x2160", "1920x1080", "1280x720", "854x480", "640x360"],
        "description": "Output resolution as WxH. Source aspect ratio is preserved; black bars added if needed."
      },
      "fit": {
        "type": "string",
        "enum": ["contain", "cover", "fill"],
        "default": "contain",
        "description": "How to handle aspect-ratio mismatch between source and target resolution."
      },
      "idempotency_key": {
        "type": "string",
        "format": "uuid",
        "description": "UUID v4. Pass the same key on retry to collapse into the original job instead of starting a new render. Max 255 characters."
      }
    },
    "required": ["source", "output_format", "resolution", "idempotency_key"]
  },
  "annotations": {
    "destructiveHint": false,
    "idempotentHint": false,
    "readOnlyHint": false
  },
  "_async_job_handle_response": {
    "description": "What the tool returns on successful submission (done: false = job accepted, not yet complete).",
    "example": {
      "job_id": "job_01j9x7kz3p8q4m5n6r7s8t9u",
      "done": false,
      "metadata": {
        "phase": "PROBING",
        "submitted_at": "2026-06-19T10:23:45Z"
      }
    }
  },
  "_error_response_rfc9457": {
    "description": "RFC 9457 Problem Details returned when the tool invocation fails (MCP: isError: true).",
    "example": {
      "type": "https://layermetry.com/errors/unsupported-codec",
      "title": "Unsupported codec combination",
      "status": 422,
      "detail": "Source codec AV1 is not supported for gif output_format. Use mp4 or webm.",
      "instance": "/jobs/job_01j9x7kz3p8q4m5n6r7s8t9u"
    }
  }
}

The details that look small and aren't

A few choices in there are doing real work. The output_format and resolution fields use enum rather than free-text strings. JSON Schema validates that the value is one of the declared options, so the model physically can't hand you a codec you don't support. That one keyword removes a whole class of agent error.

The idempotency_key is the field generic schemas leave out, and it is the one that saves you money. The annotations block records that the source file is read-only and the operation isn't safe to repeat blindly, which is exactly the kind of metadata an agent reads before deciding whether a retry is safe. Our agent primitives reference in /docs covers the governance and identity patterns these definitions anchor.

The three additions that make a schema something an agent can depend on

A generic tool schema gets you a name, a description, and typed inputs. A media contract needs three more things, and each one maps directly to a way agents fail in production.

An idempotency key, so a retry doesn't bill you twice

The retry-charges-you-twice problem from the top of the post is fixed here. The idempotency_key field, a version-4 UUID, is passed through to the underlying request as an Idempotency-Key header. The payments world standardised this pattern because a duplicate charge is unacceptable. The underlying request stores the status code and body of the first execution and returns that same response on any retry within roughly 24 hours, so a retried submit collapses into the original job instead of starting a second render. The key applies to submit-style requests. Reads and deletes are already safe to repeat. Worth noting on the standards front, an IETF effort to standardise this header (draft-ietf-httpapi-idempotency-key-header) expired in April 2026 without becoming an RFC, so the shipped header convention remains the load-bearing reference rather than a formal spec.

An async job handle, so a long render doesn't time out

The timeout problem is fixed by refusing to wait on one call. The submit step returns a job ID right away. A second step, get_job_status, takes that ID and returns an operation object in the AIP-151 long-running-operations shape from a widely used API-design standard: a done boolean, plus response on success or error on failure, plus metadata for progress. A major cloud provider's managed transcoding service follows a similar job-status shape in production, cycling through SUBMITTED, PROGRESSING, and COMPLETE or ERROR, with the probing, transcoding, and uploading sub-phases visible in the status updates.

One precision note for anyone implementing it. AIP-151 defines this operation-object shape at the RPC level rather than prescribing an HTTP status code, so it is the citable reference for the object you return. The common "202 Accepted plus a Location header to poll" is a separate, informal HTTP convention with no single normative source, so treat the two as different layers rather than the same standard. On the forward-looking side, the MCP release candidate available now adds a Tasks extension for long-running tools, with the final specification scheduled for 2026-07-28. Until that ships, the two-step submit-then-poll pattern is the recommended approach on the current stable MCP spec.

REQUEST → JOB → RESULT LIFECYCLE

One submit and one poll tool sharing a job resource; a retried submit with the same idempotency key collapses into the original job.

A structured error, so a failure is a branch, not a dead end

The generic-500 problem from the opener is the last one to fix, and it is the difference between an agent that recovers and one that gives up. A failure that returns a bare 500 tells the model nothing. It can't choose a different codec, skip the failing output, or escalate, because it has no idea what went wrong.

RFC 9457 Problem Details gives the failure a readable shape instead. Five fields carry it: a type URI naming the error class, a short title, the status code, a detail explaining this specific occurrence, and an instance URI for this problem. Custom fields are allowed, so you can add a job_id or a retryable flag.

The payoff lands when you pair it with MCP's error model. MCP separates protocol errors (malformed request, unknown tool) from tool-execution errors, which it returns in the result body with isError: true, and the spec says those execution errors should carry "actionable feedback that language models can use to self-correct and retry." Put an RFC 9457 body inside an isError result and the model gets a real signal: the type says it was an unsupported codec, the detail says why, and the agent retries with a corrected format as a deterministic recovery path instead of a guess. That structure is what gives an agent a path to reason through a failure instead of stalling on it.

So do you build this contract, or adopt a layer that ships it?

Here is the decision this whole post has been circling, laid out honestly. The contract above is real, documented, and reachable. The question is whether the ongoing cost of owning it is where you want your team's effort going.

Building it yourself

You get total control and zero dependency, and the pieces are public: JSON Schema for the inputs, AIP-151 for the job lifecycle, the idempotency-key convention for retries, RFC 9457 for errors. A capable team can stand up a first version of the transcode_video contract.

The cost is everything that comes after the first version. The async lifecycle, partial-failure handling, idempotency storage, structured errors, and the observability to see what your agents actually did all have to keep working as the specs move (the MCP Tasks extension landing, the idempotency draft expiring) and as the models you call change underneath you. That last point is the quiet one: swap a model or adjust one operation and the whole agent flow can behave differently, so the observability to catch it isn't a one-time setup, it is a standing cost. You are signing up to maintain editing infrastructure indefinitely, which is effort not spent on whatever actually differentiates your product.

Adopting a layer that ships it

You trade some control for speed and lower maintenance. The contract, the job lifecycle, the error envelope, and the observability come built and maintained, so your effort goes into your agentic workflow and your models instead of the plumbing underneath them. Layermetry is one such layer, and the thing worth knowing is that it is extensible rather than lock-in: you ship your own tools alongside its native editing ones, so adopting the contract doesn't cap what you can build on top of it.

Neither path is the "right" one. If owning every layer is core to your product, build it. If the editing contract is plumbing you'd rather not maintain forever, adopt a layer and spend the saved time on your edge. The decision is yours. This post just lays out what each side costs.

FAQ

Why does an AI agent trigger the same expensive render twice?

Because a media submit is not idempotent by default. When a call times out or the network blips, the agent retries, and without a deduplication key the second call starts a second render. That means two jobs, two bills, and two slightly different files. The fix is an idempotency key passed on submit, which the underlying request carries through as an Idempotency-Key header so a retry collapses into the original job instead of starting a new one. This is a well-known pattern in payments, where a duplicate charge is unacceptable, and the same logic applies to a render you pay for per minute.

How does an AI agent know when a long-running transcode is finished?

Not by waiting on the same call. A four-minute clip can spend minutes transcoding, so a synchronous call times out. The contract has to split the work in two: a submit step that returns a job handle immediately, and a separate poll step that reports a done flag plus either a result on success or an error on failure. This is the long-running-operation shape the AIP-151 API-design standard describes, and a major cloud provider's managed transcoding service follows a similar shape in production, moving a job through SUBMITTED, PROGRESSING, and COMPLETE or ERROR.

Should we build the agent-driving contract ourselves or adopt a layer that ships it?

Both are valid. Building it yourself gives you total control and no dependency, and the schema is well documented, so a first version is reachable. The ongoing cost is the part teams underestimate: idempotency, async job lifecycle, partial-failure handling, structured errors, and observability that has to keep working as specs and models change. Adopting a layer that ships this contract trades some control for speed and lower maintenance, and lets you spend your effort on your agentic workflow instead of the plumbing. Layermetry is one such layer, and it is extensible, so you can ship your own tools alongside its native ones rather than being locked in.

The reason agents fail on media edits isn't that the models aren't smart enough. It's that the contract underneath them was designed for weather lookups, not renders that run for minutes, charge on retry, and half-succeed. Get the contract right and the four failure modes go away: enums block hallucinated codecs, the idempotency key kills duplicate renders, the two-step async pattern prevents timeouts, and structured errors give the model a way back instead of a dead end. The same shape applies to resize, crop, caption, or any operation where duration and partial failure are real. Whether you build that contract or adopt a layer that ships it, the agent primitives in /docs show how these definitions anchor a full media-editing architecture.