[Talk Testing] What did your AI agent actually verify?

Dear Community,

“Tests passed” is a weak handoff when an AI agent changes code. Which tests ran? What did they cover? What could the agent not check?

Software Testing Weekly’s latest issue points to Will Larson’s verification pattern. His team requires a per-PR record of checks and gaps before a draft becomes ready for review.

Here is a small experiment inspired by that approach. For your next AI-assisted change, ask for:

  1. The behavior that changed and the expected result.
  2. The exact checks, their results, and links to logs or reports.
  3. The checks that did not run, with the reason.
  4. The risks that still need human review.

For a checkout retry fix, a unit test might pass while duplicate-charge behavior remains untested against a payment sandbox. Record that gap explicitly; do not translate it into “verified.”

Check the evidence yourself. An agent-written verification file is a review aid, not independent proof. Use repeated gaps to decide which fixtures, environments, or checks to improve next.

What evidence do you require before you trust an AI-authored test or code change?

Source: Self-reinforcing quality / adopting verification pattern | Irrational Exuberance

1 Like