A benchmark for the model that decides
which tools to call, and in what order.
A planner model turns a question into a chain of tool calls, where the output of one step feeds the next. A single accuracy number says little about it. Does it survive a catalog full of look-alike tools? Does it still wire things correctly five steps deep? And when it fails, is it the reasoning or just the output format?
A benchmark whose headline metric is the wiring: is every step fed by the right output of the right previous step. Answers are read tolerantly, so a formatting slip is reported as a diagnostic and never hides a correct plan. Catalogs scale up to 100 tools to add realistic noise. Run settings are fixed (temperature, seeds, timeouts, one retry) so two campaigns can be compared. It showed that deep chains were the breaking point, and traced the cause to the training data, where real deep chains made up less than 1% of examples, and not to the reward as first assumed.