Methodology
How we test
The battery
The prompts are not chosen to look good. They are chosen because they break models, and because a human can verify the result by eye: either the sign says OPEN 24H or it does not, either there are five fingers or there are not.
| Category | Prompts |
|---|---|
| Hands and fingers | 4 |
| Human faces in motion | 4 |
| Liquid physics | 4 |
| On-screen text | 4 |
| Multi-shot continuity | 4 |
| Lip-sync and dialogue | 4 |
| Camera movement | 4 |
| Animal anatomy | 4 |
| Style consistency | 4 |
| Image to video | 4 |
| Animation | 4 |
The battery is frozen. A new model gets exactly the battery every previous model got. If a prompt ever has to change, that creates a new battery version rather than silently rewriting history.
Animation and 3D
Most of the battery asks for photographic footage, but many people use these models for cartoons, anime and CG. The animation set adds 4 prompts that an animator would recognise: a 2D walk cycle, squash and stretch, an anime sword draw and a 3D feature film look. The judge checks the craft, not just the look: feet that plant without sliding, a loop that closes, volume kept through squash and stretch, follow-through on hair and cloth, and a character that stays on model.
The set is scored on its own animation board and does not count towards the overall ranking, which stays on the core 40 prompts. Two views split the results by look: Realism (the photographic prompts) and Animation and 3D (the animation set plus the anime and claymation tests of the core battery).
Scoring
A vision model evaluates each clip against the original prompt without being told which model produced it. It scores five dimensions from 0 to 5:
- Prompt fidelity: did it make what was asked for?
- Visual quality: resolution, artefacts, coherence.
- Physics and motion: does the world behave?
- Subject and scene consistency: does it hold together over time?
- Audio and sync: only where the model generates audio. Models without audio are not penalised; the dimension is simply excluded from their average.
Each clip is scored once. A single automated pass can be wrong in ways that sound confident, so every score below 4 out of 5 gets a second check: real frames pulled from the actual clip and looked at directly against the written finding, not just the judge's own summary of itself. A finding that does not hold up gets corrected before it is ever published, and the same check applies again any time an existing score is revisited. Score tables also publish a 95% confidence interval computed from that model's own runs, since four clips is not many samples and a wide interval is the honest reflection of that, not a flaw to hide.
Preferred vs not preferred, from our own data
Every dimension below is a real pair pulled straight from the benchmark, not a staged example. The caption is the actual judge note that shipped with the score.
Physics and motion
preferred MiniMax H3
Pigeon takeoff. Correct crouch, wing extension and push-off from the ledge, clean separation from stone.
not preferred Google Veo 3.1
Ice melting in whisky. FAILURE MODE: a detached liquid stream materializes out of thin air and pools outside the glass; condensation droplets stay static instead of forming.
Subject and scene consistency
preferred Google Veo 3.1
Cafe, three shots. Coat, cup and table grain match exactly across the cut.
not preferred MiniMax Hailuo 02
Rotating mug. FAILURE MODE: mug stays largely stationary instead of rotating; handle subtly morphs and widens rather than traveling.
Audio and sync
preferred MiniMax H3
Sung line. Audio present, sustained melody matches the sung phonemes.
not preferred MiniMax Hailuo 02
Bearded man, long line. FAILURE MODE: no audio track; mouth stays mostly closed with minimal jaw movement, doesn't attempt to articulate the line.
Visual quality
preferred Google Veo 3.1
1970s film look. Genuine period look: grain, warm color grading, halation around the sun.
not preferred Google Veo 3.1
Bakery sign. FAILURE MODE: text garbled across signs (duplicated Bread, Fresh Fresh, Fresh Bread Breakfast) instead of FRESH BREAD DAILY.
Prompt fidelity
preferred MiniMax H3
Counting fingers. Five normal digits, but all the fingers open at the same time instead of one by one.
not preferred MiniMax Hailuo 02
Bakery sign. FAILURE MODE: header reads BRRAANTOP BREAD and side signs misspell DAILY as DALY, instead of FRESH BREAD DAILY.
Cost, and the number nobody publishes
Sticker price per generation is close to meaningless. What matters is how many attempts it took to get a clip you would actually use. We record that as attempts to usable and it is the figure that decides what a model really costs you.
What the scores do not cover
Read the numbers for what they measure, not more than that.
- One judge, not a panel. Every clip is scored by a single vision model, not by a panel of working filmmakers or editors. That is a real limit on what a score can tell you, not a detail we gloss over.
- Four clips per category, not forty. The overall score for a model pools all 40 prompts, but a single capability, hands or lip-sync or camera moves, rests on only 4 of them. The 95% CI on each board exists specifically to keep that visible.
- Deliberately adversarial prompts, not average ones. The battery targets known failure modes (fingers, physics, camera continuity) on purpose. A model can score lower here than it feels in ordinary use, because ordinary use rarely asks for the thing that breaks it.
- Not a measure of safety, licensing terms, or fitness for your specific job. A high score means the model handled this battery well. It says nothing about content policy, usage rights, or whether it is the right tool for what you are actually making.
- One access route per model. Each model is tested through a single provider. A model's own native API, or a different platform reselling it, could behave differently; the platform pages track pricing and access separately from the score itself.
What we do not do
- We do not accept payment for placement, ever.
- We do not blend in other people’s rankings and present the result as our own measurement.
- We do not publish a price we have not verified ourselves. Unverified third-party figures are labelled as such.
- We do not hide the failures. A model that fails a prompt gets a recorded failure, not a quiet retry until it works.
Conflicts of interest
Some links to platforms earn us a commission and are marked. Model makers do not pay us and cannot see results before publication. If the best-scoring model earns us nothing, it still finishes first. Full disclosure.