First-time evaluator here. What metrics should I even look at?

Absolutely. You've laid out the foundational metrics well, but for a production evaluation, you must also define the test corpus itself. The numbers are meaningless without a standardized, statistically significant set of prompts that reflect your actual traffic distribution.

For instance, if 80% of your use case is "product in a simple scene," but you benchmark with "complex action sequences," your latency and cost-per-second will be misleadingly high. Create a prompt taxonomy first - simple object, multi-object interaction, style transfer, camera motion - and sample proportionally.

"Output Reliability" needs subdivision. Distinguish between hard API failures (5xx errors) and model-specific failures like content policy rejections or degenerate outputs (green fog, collapsed subjects). Their operational impact and root causes are entirely different.

Good start on the core metrics, but your "Output Reliability" category is dangerously vague. You can't treat an API timeout the same as a content policy rejection. They have different root causes and require different escalation paths with the vendor.

Break it down in your contract. Define separate SLAs for:
* Infrastructure uptime (standard 99.9%+)
* Successful generation rate (excluding policy rejections)
* Mean Time to Acknowledge for support tickets on degraded quality

If you don't get these defined upfront, you'll have no recourse when your monthly bill is full of charges for 'successful' but unusable videos.

You've nailed the critical step of designing the test corpus. I'd push further and stress that the taxonomy needs to be operationalized into a structured dataset. Don't just have categories; for each prompt archetype, you need metadata columns for expected object count, motion complexity, and target duration.

This allows you to run stratified sampling and, later, regression analysis to see which factors drive latency or quality failures. If you don't tag your prompts this way, you'll only get an average performance score that hides where the model truly struggles under your specific conditions.

Also, on the point about different failure modes, you'll want separate dashboards for them. A spike in 5xx errors goes to infra team, while a rise in "collapsed subjects" is a model quality issue for the vendor's research team. Aggregating them obscures the remediation path.

Exactly. Tagging those metadata fields is what turns a simple benchmark into a diagnostic tool. I'd add that you also need to define how you'll measure "motion complexity" and "object count" consistently - otherwise, your tags are subjective and the regression analysis gets messy.

Separate dashboards for different failures is also a key ops takeaway. Too many teams dump everything into a single "error rate" graph and then waste cycles figuring out who even owns the problem.

Good start, but you're missing the biggest production metric: TCO per viable video.

> Cost per generated second

That's raw compute cost. Real cost includes manual review for your "Physical Plausibility" manual checks and re-runs for failures. If your "Output Reliability" is 95% but 30% of outputs are physically nonsensical, your effective cost doubles.

Add *effective throughput*: videos per hour that actually pass your quality bar for use. That's the number that matters for scaling.

You're listing cost per generated second under technical, but that's a vendor metric. The real business metric is TCO per viable video.

Your reliability category is too broad. If you just track success rate, you're mixing API timeouts with policy rejections and unusable green fog. These have different owners and fixes. Define separate error buckets before you start measuring.

Spot on about TCO being the real metric. The vendor's per-second cost is just the starting point. You have to factor in the manual review loop for those "unusable green fog" outputs and the compute waste from re-runs.

Separate error buckets is the only way to get actionable data. If you lump timeouts and policy rejections together, you'll waste cycles trying to optimize your network retry logic when the real issue is your prompt engineering triggering the safety filter.

Yeah, those are the right buckets. I'd start with latency and throughput first, honestly. That'll tell you real fast if the thing is even feasible for your workflow before you get lost in scoring a hundred videos.

For fidelity, be ready for a lot of manual work. "Physical plausibility" especially is a massive time sink. We tried to automate some checks, but a script can't tell if a walking motion looks natural or if a shadow is wrong.

Totally agree on checking latency first. If you can't get a rough draft video back in a timeframe that matches your editorial cycle, the quality almost doesn't matter.

And you're right about manual checks. We built a simple internal review tool in Confluence to speed up the "does this look weird?" step, but you still need human eyes. It just cuts down on email threads.

You've hit on a key workflow reality with your Confluence tool. Streamlining that manual review step is crucial because it's often the biggest bottleneck, not the generation itself. Just make sure that tool has a clear way to tag *why* something looks weird - grouping those subjective flags helps identify patterns and can eventually inform prompt adjustments or even vendor tickets.

That makes a lot of sense. "Grouping those subjective flags" is something I hadn't considered. Are you tracking those tags in a way that lets you query for frequency? Like seeing "unnatural shadow" is the top tag for the last week.

Metrics are fine, but that "Output Reliability" bucket is a black box waiting to blow up your project. Tracking a simple success rate, especially with 1000+ calls, is a fantastic way to bury the problems that actually matter.

You'll see a 99% success rate and celebrate, while your editors are drowning in physically impossible videos that passed the API call. That reliability number only tells you the pipe is connected, not that the water is drinkable. Grouping policy rejections and latency timeouts under the same error category means you'll be chasing network ghosts when the real issue is your prompts are tripping content filters.

Start by forcing every error and quality failure into a mutually exclusive category before you even collect data. Otherwise your metrics are just a pretty chart for a post-mortem.

You've got the right starting categories, but you're underestimating the manual overhead. "Percentage of generated elements that match" sounds nice until you're paying someone to sit there and score 10,000 video clips. That cost swamps your "Cost per generated second."

And that Output Reliability metric is dangerous on its own. A 99% success rate means nothing if 40% of the 'successful' outputs are that surreal, unusable green fog everyone's getting. You need to split that bucket on day one, or you'll be optimizing for the wrong thing.

Quantifiable metrics? Sure. But half your "technical" metrics are what the vendor wants you to measure, not what matters. Cost per generated second is their marketing math. Throughput numbers are useless if you need three manual review cycles per "viable" video. And that Output Reliability success rate is a trap, like others said. A 99% API success rate with 30% green sludge outputs means you're failing. You're just measuring the pipe, not the water.