How Do You Know if Your AI Agent's Work Is Actually Any Good?

Your agent finishes a batch overnight. A hundred thumbnails, a stack of vertical cutdowns, captions on every clip. You open the folder in the morning, and the work looks fine. Then the real question lands and you have no clean way to answer it. Is this good? Is it on-brief? Would the brand team sign off, or are you about to push a hundred slightly-off assets in front of customers?

This is the part of running an agent that nobody has fully solved. We have gotten good at making agents produce creative work. Grading it, deciding whether it is actually good, is still the open problem. Full disclosure: Layermetry builds an AI media-editing layer where this question shows up every day, so we will lay out the approaches honestly, including where each one falls short, and let you decide how much to trust a machine with the judgment.

"Good" is the hardest thing to define

The trouble starts before you pick a tool. You cannot measure what you have not defined, and "good creative work" resists definition in a way "correct code" does not.

A right answer versus a good answer

When an agent writes code, there is usually a right answer. The test passes or it fails. Creative work has no single right answer. A caption can be punchy or warm or plain, and all three can be good for different briefs. So the question is never "did it match the answer," because there is no answer to match. The question is "is this one of the many good outcomes, and is it the right one for this brief and this brand."

On-brief, on-brand, and good are three separate tests

Split "good" into the three things people actually mean, because an output can pass one and fail another.

• On-brief: did it do the job asked. Right aspect ratio, the requested crop, logo present, caption under the limit.
• On-brand: does it match the voice and visual identity. Brand guidelines try to capture this across hundreds of pages and never quite go far enough, because being on-brand is hundreds of small decisions shaped by context and taste.
• Good: would a skilled person be glad to put their name on it. This is the slippery one, and where every shortcut breaks down.

On-brief is mostly mechanical and you can check it automatically. On-brand is partly checkable and partly taste. Good is almost entirely taste. Any honest plan treats these as separate gates rather than one score, because the cheap, reliable checks live at one end and the expensive, human ones live at the other.

Why the simple metrics let you down

The first instinct is to reach for a number. It feels rigorous, it scales, and it gives you a dashboard. For creative output, the number usually lies to you.

Overlap metrics were built for a different job

The classic text metrics, BLEU and ROUGE, were designed decades ago for machine translation and summarization, tasks where one reference answer is roughly correct and you score how close the output lands to it. Pointed at creative work, they measure word overlap with a reference, which is exactly the wrong thing. As one 2026 teardown put it, the behaviors people find most impressive, genuine understanding and creative problem-solving, are the ones these metrics punish most severely. Rewarding closeness to one reference rewards the bland average. That is backwards for anything creative.

A green dashboard can still be wrong

The deeper trap is a precise number for something the metric is not actually measuring. You can ship a pipeline where every output scores 0.82 and half are off-brand, because the score never looked at brand. The honest framing from the 2026 guidance is to tie your measures to the outcome you care about, not to a convenient proxy. If the goal is "assets the brand team would approve," the only ground truth is whether they approve them, and your job is to build the cheapest reliable approximation of that judgment.

The approaches that actually work, and what each one costs

No single method grades creative work end to end. What works in practice is a stack, where each layer handles what it is good at and hands the rest up. Here are the four that matter, cheapest first.

Machine-readable acceptance criteria catch the obvious failures

Start with the checks a computer can make for free. Write down the conditions the output must meet as something a program can test: the dimensions, the safe zones, the character limits, the required label, the banned terms. This is the discipline software teams now call eval-driven development, where you define success and encode it as evaluations before you trust the system to deliver. Teams that work this way catch more issues before production because failures surface immediately instead of in a customer's feed. Acceptance criteria will never tell you if something is beautiful. They reliably tell you if it is broken, and that alone removes most bad outputs at zero marginal cost.

A rubric turns fuzzy taste into something repeatable

For the judgments a checklist cannot make, the next step up is a rubric. Instead of asking "is this good," you decompose quality into named criteria, score each one, and force the evaluation to justify itself. Recent work favors rubric-based scoring precisely because handing out a single coarse number is unstable and biased, while breaking quality into interpretable criteria is far steadier. A rubric does two useful things. It makes two reviewers grade the same way, and it makes the grade explainable, so when something scores low you see which criterion failed. You can apply a rubric by hand, or hand it to a machine to apply at scale, which leads to the next layer.

An AI judge scales the rubric, with real caveats

The pattern that took over in 2026 is the AI judge: a separate model reads each output, applies your rubric, and returns a score with reasons. It scales, it is consistent, and on structured criteria it can track human reviewers reasonably well. How well depends on details most people never tune. One 2026 study found that even the scoring scale matters, with human-LLM agreement landing strongest on a 0-5 scale across six benchmarks, though it varied by model and task and was never perfect. That is useful, and it is also where honesty matters most.

The caveat is that these judges carry biases that have nothing to do with quality. Studies through early 2026 found the verdict shifts with length, position, and formatting, not just substance, so a wordier or better-formatted output can win on looks alone. On a dedicated bias benchmark, the same research found even frontier models exceeding 50% error rates. The takeaway is not to throw the judge away. Use it for what it is good at, applying clear mechanical criteria at volume, and keep it away from the final call on taste, where its blind spots do the most damage.

An evaluator-optimizer loop lets the agent fix its own work first

The fourth piece is not a way to grade, it is a way to raise the floor before grading. In an evaluator-optimizer loop, one role generates the output, a separate role critiques it against explicit criteria, and the feedback goes back to the generator to try again, repeating until the work clears the bar or hits a retry limit. The roles stay separate on purpose, because generating and judging are different cognitive jobs and one model doing both at once tends to do both worse.

Two rules keep the loop honest. The criteria must be explicit and checkable, or it never knows when to stop, and there must be a retry limit, or a generator and evaluator that disagree will burn budget forever. A loop like this will not invent taste the agent never had, but it catches the self-evident misses before a human sees them. We go deeper in our piece on structured feedback for media pipelines.

Where humans stay in the loop, and why that is fine

Stack all four and you have removed the broken outputs, scored the checkable criteria, and let the agent self-correct. What remains is the part the machine genuinely cannot do, and the honest answer is to keep a human on it.

Spot checks make human attention affordable

You do not need a person to inspect every asset. The practical pattern that has settled in is sampling. Review a small percentage of outputs, often in the single digits, and track the correction rate. If corrections stay low, widen what the agent handles unsupervised. If they climb, tighten the gate. Random sampling gives an unbiased read on quality, and pointing review at the cases where your automated judge was least confident spends that attention where it pays off most. This keeps a human on quality without putting one on every item.

The taste call is a feature, not a failure

Keeping a person on the subjective call is not a gap in your tooling, it is the correct design. 2026 surveys still find audiences trust human-touched work more, with one finding nearly 78% of consumers trust video featuring real people over content they believe was AI-made. The judgments machines miss, tone, brand voice, and whether something simply feels right, are exactly the ones worth a person's time.

The goal is not to remove the human. It is to make sure the human spends their judgment on the hundred decisions that need taste, not the ten thousand that needed a checklist. The human holds the creative vision and the final taste call. Everything mechanical underneath is what you automate or hand off.

So how should you actually set this up

Here is the whole thing in one line: there is no single grade for creative work, so build a stack of cheap checks under a thin layer of human taste, and be honest about which layer is doing the real judging.

Build it yourself or adopt a layer that ships it

You have two honest paths, and neither is wrong. You can build this stack in-house, wiring up the acceptance criteria, rubric, AI judge, optimizer loop, and sampling dashboard yourself. That wins on total control, no dependency, and a harness tuned exactly to your brand. The cost is real, ongoing engineering, and it is never a one-time setup. Change one model or prompt and the whole pipeline can behave differently, so the evaluation to catch that drift is a standing cost, not a project you finish. There is more in our note on why media observability is never one-and-done.

Or you adopt a layer that ships the evaluation scaffolding with the editing, so your team focuses on the rubric and brand judgment that are yours to own rather than rebuilding the harness. That wins on speed and lower maintenance, and the cost is a dependency you do not fully control. Layermetry is one such layer. We are a fact-sharing company, so we would rather you make this call with both paths in front of you. If you are weighing the two more broadly, our build-versus-adopt breakdown walks the same decision for the editor, and our primer on agents and frontier models grounds the terms if any of this was new. The docs go further.

Whichever path you take, the principle holds. Grading an agent's creative work is not solved, and anyone selling you a single magic score is selling you the dashboard, not the judgment. Build the cheap checks, sample the rest, and keep a person on taste.

FAQ

Can an AI judge reliably grade an AI agent's creative work?

Partly. An AI judge with a clear rubric can track human reviewers reasonably well on structured criteria, and a 2026 study found human-LLM agreement was strongest on a 0-5 scale, though it varied by model and task. But the same judges show verbosity, position, and self-preference biases that come from how a response looks rather than what it says, and frontier models exceeded 50% error rates on challenging bias benchmarks. Treat an AI judge as a cheap first pass on mechanical, checkable criteria, not as the final word on taste.

Why don't automated metrics work for creative output?

Metrics like BLEU and ROUGE were built for machine translation and summarization, where one reference answer is roughly correct. Creative work has many valid answers, so a metric that scores closeness to one reference punishes originality and rewards the bland average. They measure word overlap, not whether the work is on-brief, on-brand, or good.

What should I keep human in agent creative evaluation?

Keep humans on the subjective judgments automation cannot capture: tone, brand voice, taste, and whether the work feels right. A common practical pattern is sampling a small percentage of outputs for human review and tracking the correction rate. If corrections stay low you can widen what the agent handles unsupervised, and if they rise you tighten the gate. Automate the mechanical checks, sample the rest, and keep a human on the part that needs judgment.