news

Artificial Analysis ships a Decision Index for the models being sold as cheap AI judges, and all four of its evaluations are text

Version 1.0, published October 13, scores classification, tool calling, knowledge, hallucination and reasoning, and never asks a model to look at an image or a frame of video.

By Priya Shenoy · 14 October 2026

Artificial Analysis published Decision Index v1.0 on October 13, 2026, the first version of an independent scorecard for decision models, the single-call systems that return a probability for each allowed answer and that several vendors are now selling as a cheaper substitute for an LLM judge. Version 1.0 combines four evaluations spanning classification, tool calling, knowledge, hallucination and reasoning. Every one of them is text. Nothing in the suite asks a model to look at an image or a frame of video, which is the one thing a judge has to do before it can score generated media.

A model answers every question in one call, and the highest probability wins

The protocol is the interesting part, because it is what separates this category from prompt-and-parse evaluation. Artificial Analysis sends the state and the questions to a model's decision API, and for a choice the model returns the option it picks plus a probability for every option, with scoring applied to the option it assigns the highest probability. For a yes or no question the model returns the probability of yes, and the answer is counted as yes when that probability reaches 0.5. Language models are run on the same tasks through a fixed system prompt with a JSON reply enforced by structured outputs, so the comparison between a 27B decision head and a frontier LLM is at least running on the same questions.

AA-Omniscience-Decision, HLE-MultiChoice, When2Call and WildGuardTest

Those are the four evaluations in v1.0. AA-Omniscience-Decision uses 500 private AA-Omniscience questions, each one a choice among five options: four candidate answers, one of them correct, plus an explicit "I don't know". It carries 25% of the index, split evenly between accuracy at 12.5% and non-hallucination rate at 12.5%, which is a reasonable way to stop a model from buying points with confident guesses. The remaining three cover multiple-choice knowledge, deciding when a tool call is warranted, and safety classification. None of them is perceptual.

Clef accepts images. The index that would rank it does not use any

This is where the gap bites for anyone evaluating generated media. Cloudflare's Clef models take image input, while other entrants in the same category, including AWS's Strands Decider and Comet's Opik judge, cannot read a picture at all. A buyer choosing between them for an automated media pipeline gets a single index number that was produced entirely without pixels, so it cannot distinguish the vision-capable option from the text-only one on the task that matters.

In our own benchmark, the hands category is the clearest example of what a media judge is actually being asked to do: count fingers, track a knife, and decide whether the motion survives the cut.

Scoring that is a per-category problem, not a single scalar, which is how our model scores are reported.

Two different boards are now called a Decision Index

There is a naming problem worth checking before anyone compares numbers. Perplexity's model card for pplx-decider-v1.1-27b reports an overall Decision Index score of 61.56, up from 56.4 for v1, broken into Knowledge at 48.18, Language at 69.45, Retrieval at 61.26, Tools at 78.88 and Arts at 44.66. Those five categories are not the four evaluations in Artificial Analysis v1.0, so the two scores come from different suites that share a name. A 61.56 and a number from the Artificial Analysis board are not interchangeable, and vendor cards citing one of them do not settle a position on the other.

The thing to watch is whether a later version adds a perceptual evaluation. Until it does, a high Decision Index score is evidence about structured text decisions and says nothing about whether a model can look at a generated clip and tell you what is wrong with it.

Sources: Artificial Analysis, Artificial Analysis Articles, Perplexity model card.

Priya Shenoy: Tracks what AI video actually costs across the platforms that resell access to the same handful of models. Treats a pricing page as a claim, not a fact, until someone checks it.