news

DecisionPivot scores 40 automated judges across 40,033 items, and 19 of them land under a guessing baseline

StartLux's new diagnostic benchmark covers seven decision roles used inside LLM systems, all of them text, with contrast pairs that differ by a single answer-changing detail.

By Priya Shenoy · 21 October 2026

The small classifier models being sold as cheap automated judges now have a benchmark designed to break their headline numbers apart, and the first results are not flattering. DecisionPivot, published by StartLux on October 20, 2026, spans 48 tasks and 40,033 items across seven decision roles, evaluates 40 decision models plus 14 general LLMs, and reports that 19 of the 40 decision models fall below a 16.1% uniform-guess baseline on overall pair accuracy, with 30 below that line at difficulty level 3. The paper notes those totals include failures caused by input restrictions, so they measure input compatibility as well as decision quality.

Contrast pairs that hinge on one changed detail

The core track holds 7,800 items from 13 generated task families, organised into contrast groups whose answers were computed and verified by independently implemented solvers. Most are built as pairs differing in one answer-changing detail, and a pair counts only when the model gets both sides right, which is why pair accuracy drops so far below the item accuracy these models usually advertise. A separate reference track adds 32,233 items converted from 35 public test sets, keeping each one's original evaluation metric. The seven roles cover answerability, guardrails, output verification, agent tool checks, reasoning classification, record reconciliation and constraint-based matching, and scoring goes beyond accuracy to probability ordering, calibration error over 15 bins, accuracy and coverage at confidence 0.90, and retrospective selective coverage at a 5% error rate.

That metric set matters for anyone pointing a judge at generated media, because a grader that is confidently wrong is worse than one that abstains. Nothing in the 40,033 items is an image or a clip, so none of it speaks to whether these same models can score output like this.

The hosted reference in the lineup is jev-1.13.0, the judge classifier that returns a probability rather than prose, and the companion StartLux-Decision report positions its models against the public subset of Decision Index 0.3. The whole adjacent field of cheap judges is text-shaped, and DecisionPivot extends that shape rather than breaking it.

Stable totals, moving answers

The sharpest finding is about reproducibility. Two 27B models sitting within 2.6 reference points of each other scored 70.1% and 44.2% on contrast-group accuracy, a spread an aggregate table would hide. A repeat run of the 27B AutoThink model produced 87.2% pair accuracy against 86.9% on the first pass while 5.6% of individual answers changed underneath, and the authors write that "stable aggregate scores can mask changes in individual predictions." Reasoning effort moves things too: on a separate 1,296-item subset, GLM-5.3 goes from 81.5% contrast-group accuracy at its lowest setting to 93.5% at a higher one, with mean reasoning length rising from 121 to 265 tokens per item. Rankings across core versions 1.0 and 1.1 stayed correlated at Spearman 0.986 over 20 configurations.

The team that built the benchmark also built two of the entrants

Two StartLux-Decision-27B variants sit in the 40-model field, and the paper's own limitations section flags that the benchmark and those models come from the same team, marking independent evaluation as worth having. The authors also removed 3,016 reference items, 8.5% of that track, after detecting them in their training files, and did so before computing the headline rankings. The dataset, released on Hugging Face as version 1.1, is the former DecideBench under a new name with the same items, scorer and result tables, with the 13 generated core families under CC BY 4.0 and 95% percentile bootstrap intervals computed from 1,000 resamples. StartLux's model page states DecisionPivot was not used for any data, checkpoint or configuration choice.

Sources: arXiv, Hugging Face, StartLux.

Priya Shenoy: Tracks what AI video actually costs across the platforms that resell access to the same handful of models. Treats a pricing page as a claim, not a fact, until someone checks it.