James Booth
Back to Research

The Agent Failure Index

  • Published 21 August 2026
  • Advizr
  • Version 1.0
  • Bylined James Booth

What breaks in production AI agents, and did anyone notice?

Findings

Open a finding to see how it was measured.

  1. The model was at fault in 3 of 358 failures (0.8%).
  2. 49.2% of failures were silent. The system reported success.
  3. 339 prevention rules were written, and 0 were enforced.
  4. The MAST taxonomy covers 3.9% of the failures.

First published by Advizr.

Back to The Agent Failure Index

Method for finding 1 of 4

The model was at fault in 3 of 358 failures (0.8%).

358 of 359 failures from the agency's error vault were extracted, anonymised, classified and mapped to the MAST and OWASP taxonomies. A refute pass then checked the 250 failures coded silent or wrong signal against their own notes. 7 were recoded as loud.

Basis: Version 1.0 corpus, pinned at vault commit d152f8a

Failures by layer

Verification 117
Application 81
Infrastructure 66
Data 33
Third party 32
Orchestration 26
Model 3
Model: The model produced a wrong output.

Source: advizr.ca/research/data/agent-failure-index.json

Version 1.0, pinned 2026-09-23. Basis: 358 of 359 failure notes classified, recorded 2026-06-14 to 2026-08-21, from the error vault at commit d152f8a. Data CC BY 4.0.

Back to The Agent Failure Index

Method for finding 2 of 4

49.2% of failures were silent. The system reported success.

358 of 359 failures from the agency's error vault were extracted, anonymised, classified and mapped to the MAST and OWASP taxonomies. A refute pass then checked the 250 failures coded silent or wrong signal against their own notes. 7 were recoded as loud.

Basis: Version 1.0 corpus, pinned at vault commit d152f8a

What the system reported when it failed

Silent 176, 49.2%
Loud 107, 29.9%
Wrong signal 67, 18.7%
Quiet 8, 2.2%
  • Silent. Reported success. Exit zero, HTTP 200, a green check.
  • Loud. Raised an error a human saw, describing the real problem.
  • Wrong signal. Raised an error describing a different problem, so the signal misdirected.
  • Quiet. Raised an error into somewhere nobody was reading.

Source: advizr.ca/research/data/agent-failure-index.json

Version 1.0, pinned 2026-09-23. Basis: 358 of 359 failure notes classified, recorded 2026-06-14 to 2026-08-21, from the error vault at commit d152f8a. Data CC BY 4.0.

Back to The Agent Failure Index

Method for finding 3 of 4

339 prevention rules were written, and 0 were enforced.

358 of 359 failures from the agency's error vault were extracted, anonymised, classified and mapped to the MAST and OWASP taxonomies. A refute pass then checked the 250 failures coded silent or wrong signal against their own notes. 7 were recoded as loud.

Basis: Version 1.0 corpus, pinned at vault commit d152f8a

Back to The Agent Failure Index

Method for finding 4 of 4

The MAST taxonomy covers 3.9% of the failures.

358 of 359 failures from the agency's error vault were extracted, anonymised, classified and mapped to the MAST and OWASP taxonomies. A refute pass then checked the 250 failures coded silent or wrong signal against their own notes. 7 were recoded as loud.

Basis: Crosswalk of the version 1.0 corpus