The model was at fault in 3 of 358 failures (0.8%).
358 of 359 failures from the agency's error vault were extracted, anonymised, classified and mapped to the MAST and OWASP taxonomies. A refute pass then checked the 250 failures coded silent or wrong signal against their own notes. 7 were recoded as loud.
Basis: Version 1.0 corpus, pinned at vault commit d152f8a
Version 1.0, pinned 2026-09-23. Basis: 358 of 359 failure notes classified, recorded 2026-06-14 to 2026-08-21, from the error vault at commit d152f8a. Data CC BY 4.0.
49.2% of failures were silent. The system reported success.
358 of 359 failures from the agency's error vault were extracted, anonymised, classified and mapped to the MAST and OWASP taxonomies. A refute pass then checked the 250 failures coded silent or wrong signal against their own notes. 7 were recoded as loud.
Basis: Version 1.0 corpus, pinned at vault commit d152f8a
What the system reported when it failed
Silent
176, 49.2%
Loud
107, 29.9%
Wrong signal
67, 18.7%
Quiet
8, 2.2%
Silent. Reported success. Exit zero, HTTP 200, a green check.
Loud. Raised an error a human saw, describing the real problem.
Wrong signal. Raised an error describing a different problem, so the signal misdirected.
Quiet. Raised an error into somewhere nobody was reading.
Version 1.0, pinned 2026-09-23. Basis: 358 of 359 failure notes classified, recorded 2026-06-14 to 2026-08-21, from the error vault at commit d152f8a. Data CC BY 4.0.
339 prevention rules were written, and 0 were enforced.
358 of 359 failures from the agency's error vault were extracted, anonymised, classified and mapped to the MAST and OWASP taxonomies. A refute pass then checked the 250 failures coded silent or wrong signal against their own notes. 7 were recoded as loud.
Basis: Version 1.0 corpus, pinned at vault commit d152f8a
358 of 359 failures from the agency's error vault were extracted, anonymised, classified and mapped to the MAST and OWASP taxonomies. A refute pass then checked the 250 failures coded silent or wrong signal against their own notes. 7 were recoded as loud.