Module: 9 · Duration: ~45 min
[SLIDE 1] Module Nine: Verification and Feedback Loops. Forty-five minutes. Adding verification improves output quality two to three times — Boris Cherny, Claude Code founder. That is the empirical justification for the entire module.
[SLIDE 2] The Cherny claim. Adding verification — an extra LLM call, a test-suite run, a model-judged check — improves output quality two to three times. The cost is real; the quality multiplier is larger. This module is about how to verify, how much to verify, and when to stop retrying.
[SLIDE 3] Four verification methods. Computed — linter, type checker, test suite, build system — deterministic and cheap; the default. Visual — Playwright screenshot, pixel diff — high reliability for UI tasks. Model-judged — an independent subagent evaluates the output — flexible, catches semantic errors, but expensive (an extra LLM call). Human — highest reliability, highest latency; the last resort.
[SLIDE 4] Tight versus staged versus loose. Tight: verify after each tool call, correct immediately; highest quality, highest cost. Staged: verify at checkpoints — post-plan, post-implementation, post-test; balanced. Loose: verify at task end, restart if failed; lowest cost, most rework on failure.
[SLIDE 5] Retry budget and flaky tests. Without a retry budget, a flaky test traps the agent in a verify-fail-retry loop — a variant of Module Seven's stuck loop. Flaky handling: run three times, majority vote — two of three pass is a pass. Or quarantine flaky tests as warnings, not failures.
[SLIDE 6] Verification as a security control. Beyond quality: verification answers "does the output do what was intended?" — which is also "did the agent do what it was supposed to, not what an injection told it to?" A computed verification catches an injected code change because it breaks tests. A model-judged verification catches semantic deviation. Verification is Module Six's layered defense, not just quality assurance.
[SLIDE 7] The three-tier pipeline as a node graph. Agent output. Tier one, computed — lint plus compile; fail returns to the model. Tier two, model-judged — an independent subagent; fail returns to the model. Tier three, human — optional for low-risk, required for destructive and external. Cheapest and deterministic first; expensive and flexible second; human last resort. The retry budget caps how many times the model can retry.
[SLIDE 8] Four takeaways. Verification improves quality two to three times, per Cherny; the cost is worth it. Four methods — computed default, visual for UI, model-judged for semantic, human last resort. Retry budget prevents flaky-test stuck loops. Verification is a security control — catches injected deviations; defense in depth. Next: Module Ten — Observability and Debugging.
# Teaching Script — Module 9: Verification & Feedback Loops **Module**: 9 · **Duration**: ~45 min [SLIDE 1] Module Nine: Verification and Feedback Loops. Forty-five minutes. Adding verification improves output quality two to three times — Boris Cherny, Claude Code founder. That is the empirical justification for the entire module. [SLIDE 2] The Cherny claim. Adding verification — an extra LLM call, a test-suite run, a model-judged check — improves output quality two to three times. The cost is real; the quality multiplier is larger. This module is about how to verify, how much to verify, and when to stop retrying. [SLIDE 3] Four verification methods. Computed — linter, type checker, test suite, build system — deterministic and cheap; the default. Visual — Playwright screenshot, pixel diff — high reliability for UI tasks. Model-judged — an independent subagent evaluates the output — flexible, catches semantic errors, but expensive (an extra LLM call). Human — highest reliability, highest latency; the last resort. [SLIDE 4] Tight versus staged versus loose. Tight: verify after each tool call, correct immediately; highest quality, highest cost. Staged: verify at checkpoints — post-plan, post-implementation, post-test; balanced. Loose: verify at task end, restart if failed; lowest cost, most rework on failure. [SLIDE 5] Retry budget and flaky tests. Without a retry budget, a flaky test traps the agent in a verify-fail-retry loop — a variant of Module Seven's stuck loop. Flaky handling: run three times, majority vote — two of three pass is a pass. Or quarantine flaky tests as warnings, not failures. [SLIDE 6] Verification as a security control. Beyond quality: verification answers "does the output do what was intended?" — which is also "did the agent do what it was supposed to, not what an injection told it to?" A computed verification catches an injected code change because it breaks tests. A model-judged verification catches semantic deviation. Verification is Module Six's layered defense, not just quality assurance. [SLIDE 7] The three-tier pipeline as a node graph. Agent output. Tier one, computed — lint plus compile; fail returns to the model. Tier two, model-judged — an independent subagent; fail returns to the model. Tier three, human — optional for low-risk, required for destructive and external. Cheapest and deterministic first; expensive and flexible second; human last resort. The retry budget caps how many times the model can retry. [SLIDE 8] Four takeaways. Verification improves quality two to three times, per Cherny; the cost is worth it. Four methods — computed default, visual for UI, model-judged for semantic, human last resort. Retry budget prevents flaky-test stuck loops. Verification is a security control — catches injected deviations; defense in depth. Next: Module Ten — Observability and Debugging.