James Booth
Back to A decision model against a chat model for agent routing

Method for finding 1 of 3

The Jev decision model scored 52 of 52 (lower bound 0.931, p95 389 ms). gpt-4o scored 48 of 52 (p95 2,222 ms), and 1 injection got through.

A held-out bench of 52 asks, scored with Wilson lower bounds, plus prompt-injection probes.

Basis: Held-out bench of 52 asks, 2026-09-22

Back to A decision model against a chat model for agent routing

Method for finding 2 of 3

In production Jev cost about 18 times less per decision, measured on only 12 decisions.

A held-out bench of 52 asks, scored with Wilson lower bounds, plus prompt-injection probes.

Basis: 12 production decisions, 2026-09-22

Back to A decision model against a chat model for agent routing

Method for finding 3 of 3

Splitting one question into atomic questions raised accuracy from 62.6% to 95%.

A held-out bench of 52 asks, scored with Wilson lower bounds, plus prompt-injection probes.

Basis: An earlier test, recorded 2026-09-21