def computed_verify(repo_dir: str) -> VerificationResult:
    """Tier 1: deterministic checks. Always run first; cheap and precise."""
    checks = [
        ("lint",     lambda: run(["ruff", "check", "."], cwd=repo_dir)),
        ("types",    lambda: run(["pyright"], cwd=repo_dir)),
        ("tests",    lambda: run(["pytest", "-x", "--tb=short"], cwd=repo_dir)),
        ("build",    lambda: run(["python", "-m", "build"], cwd=repo_dir)),
    ]
    failures = []
    for name, check in checks:
        result = check()
        if result.returncode != 0:
            failures.append(CheckFailure(name=name, output=result.stderr[-4000:]))
    if failures:
        return VerificationResult(ok=False, tier=1, failures=failures)
    return VerificationResult(ok=True, tier=1)

interface JudgeInput {
  task: string;                  // original goal
  successCriteria: string[];     // "done and correct" checklist
  artifact: string;              // the primary agent's output
  computedResult: VerificationResult;   // Tier 1 verdict, for context
}

interface JudgeVerdict {
  pass: boolean;
  score: number;                 // 0–1, calibrated confidence
  criteriaMet: string[];
  criteriaMissed: string[];
  reasoning: string;             // why it passed/failed — fed back to the primary agent
  severity: "blocker" | "major" | "minor";
}

const codeReviewAgent: Subagent = {
  name: "code_review",
  description: "Independent reviewer. Evaluates a code change against explicit success criteria. Does NOT modify code — returns a verdict only.",
  run: async (input: JudgeInput): Promise<JudgeVerdict> => {
    const response = await model.complete([
      { role: "system", content: JUDGE_PROMPT },        // no shared context with primary
      { role: "user",   content: JSON.stringify(input) },
    ]);
    return parseJudgeVerdict(response);
  },
};

{
  "pass": false,
  "score": 0.6,
  "criteriaMet": ["(1) max 5 attempts per window"],
  "criteriaMissed": [
    "(2) returns 429 — code returns 403 instead",
    "(3) reset-on-success — counter never resets"
  ],
  "reasoning": "The rate limiter increments correctly and caps at 5, but uses 403 (Forbidden) where the spec requires 429 (Too Many Requests), and the success-path does not clear the counter.",
  "severity": "major"
}

Let p = probability a single verify-fix cycle resolves the failure.
Expected cycles to resolve = 1 / p.
If p = 0.5 (even chance each cycle), expected = 2 cycles.
If p = 0.2 (stubborn failure), expected = 5 cycles.

Budget N should be set to: the smallest N where (1 - p)^N is acceptably small.
  p = 0.5, N = 5  →  P(still failing after 5) = 0.5^5 = 3%   ← acceptable
  p = 0.2, N = 5  →  P(still failing after 5) = 0.8^5 = 33%  ← not acceptable; raise N or escalate
  p = 0.2, N = 10 →  P(still failing after 10) = 0.8^10 = 11%

def verification_loop(artifact, max_cycles=5, min_improvement=0.05) -> VerificationOutcome:
    history: list[float] = []
    for cycle in range(max_cycles):
        verdict = judge(task, criteria, artifact)
        history.append(verdict.score)
        if verdict.pass:
            return VerificationOutcome(ok=True, cycles=cycle, artifact=artifact)
        # Stagnation detection: no meaningful improvement over last 2 cycles.
        if len(history) >= 3 and history[-1] - history[-3] < min_improvement:
            return VerificationOutcome(
                ok=False, reason="score_stagnant", history=history, artifact=artifact
            )
        artifact = repair(artifact, verdict)   # feed the verdict back to the primary agent
    return VerificationOutcome(ok=False, reason="budget_exhausted", history=history, artifact=artifact)

# Module 9 — Verification and Feedback Loops

**Course**: Master Course · **Module**: 9 · **Duration**: 45 min · **Prerequisites**: Modules 1–8

> *Adding verification improves output quality 2–3× (Boris Cherny, Claude Code founder).*

---

## Learning Objectives

1. Choose among the 4 verification methods (computed, visual, model-judged, human) by reliability and cost.
2. Design a verification loop with a retry budget and flaky-test handling.
3. State verification as a security control (does the output do what was intended?).

---

# 9.1 — Verification Methods

*What can be verified, and who/what does the verifying? The choice is a reliability/cost tradeoff, and the right answer is almost always a tiered combination rather than any single method.*

## The four methods

Every verification — every "did the agent's output do the right thing?" check — falls into one of four buckets. They differ in reliability (how often the verdict is correct) and cost (latency, tokens, human attention). No single method is sufficient for production; the right design layers them.

| Method | Description | Reliability | Cost |
| --- | --- | --- | --- |
| **Computed** | Linter, type checker, test suite, build system | Deterministic | Low latency |
| **Visual** | Playwright screenshot; pixel diff; browser check | High for UI | Medium latency |
| **Model-judged** | Independent subagent evaluates the primary agent's output | Flexible; catches semantic errors | High cost (extra LLM call) |
| **Human-in-loop** | Human reviews output before commit | Highest reliability | High latency |
| **No verification** | Ship and hope | — | — |

The "no verification" row is included because it is the *default* of a naive harness. An agent that produces output and returns it without any check is shipping on hope. The compounding math from Module 7.1 applies: an unverified 50-step process at 99% per-step reliability is 60% likely to be correct end-to-end. Verification is what lifts that number toward production-grade.

## The Cherny claim, precisely

Boris Cherny (Claude Code creator), in his talk on how he actually uses Claude Code: *"Give Claude a way to verify its work. If Claude has that feedback loop, it will 2–3x the quality of the final result."* He frames it as the single most important tip — "probably the most important thing" — and pairs it with a concrete question to ask before any non-trivial task: *what does 'done and correct' look like?*

This is a **practitioner estimate from a talk, not a peer-reviewed benchmark** — treat the 2–3× as an order-of-magnitude claim, not a precise constant. But the direction is well-supported by the verification literature (see 9.3 on self-refinement and reward models): giving a generator access to verifiable feedback reliably improves output quality, with reported gains in the same range. The mechanism is the same in all cases: without feedback, errors compound silently (Module 7.1); with feedback, errors are caught and corrected before they propagate. The 2–3× is the empirical size of that correction effect for agentic coding tasks, per Cherny's experience.

The cost is real — a verification call is an extra LLM call (model-judged) or a tool execution (computed). Cherny's claim is that the quality multiplier exceeds the cost multiplier, which is the same logic that justifies every retry, checkpoint, and observability payload in this course.

## Tight vs staged vs loose loops

How often to verify is a separate decision from *which* method. Three loop shapes:

- **Tight loop**: verify after each tool call; correct immediately. Highest quality; highest verification cost. Appropriate for high-stakes, low-latency-tolerance tasks (e.g., code that ships on commit).
- **Staged loop**: verify at defined checkpoints (post-plan, post-implementation, post-test). Balanced. The default for most agentic coding tasks.
- **Loose loop**: verify after full task completion; restart if failed. Lowest verification cost; most rework on failure. Appropriate for cheap-to-retry, exploratory tasks.

The mistake is choosing the loop shape by gut. The right heuristic: **match verification frequency to the cost of an undetected error propagating.** In a tight loop, an error is caught within one tool call — propagation cost is one step. In a loose loop, an error at step 3 propagates through steps 4–50 before the final verification catches it — propagation cost is the entire task. Module 7.1's compounding math quantifies that cost; the loop shape is the control.

---

# 9.2 — The Three-Tier Verification Model

*In practice, production harnesses layer the methods rather than pick one. The three-tier design is the reference pattern.*

## Tier 1: Computed verification (the gate)

The first tier is deterministic and cheap: a linter, a type checker, a test suite, a build. It runs on every significant change and produces a binary verdict — pass or fail, with a specific error message. This is the foundation because it is **free feedback**: no LLM call, no human time, near-zero latency.

```python
def computed_verify(repo_dir: str) -> VerificationResult:
    """Tier 1: deterministic checks. Always run first; cheap and precise."""
    checks = [
        ("lint",     lambda: run(["ruff", "check", "."], cwd=repo_dir)),
        ("types",    lambda: run(["pyright"], cwd=repo_dir)),
        ("tests",    lambda: run(["pytest", "-x", "--tb=short"], cwd=repo_dir)),
        ("build",    lambda: run(["python", "-m", "build"], cwd=repo_dir)),
    ]
    failures = []
    for name, check in checks:
        result = check()
        if result.returncode != 0:
            failures.append(CheckFailure(name=name, output=result.stderr[-4000:]))
    if failures:
        return VerificationResult(ok=False, tier=1, failures=failures)
    return VerificationResult(ok=True, tier=1)
```

Computed verification catches the errors a machine *can* catch deterministically: syntax, types, import errors, test regressions, build failures. It cannot catch semantic errors — the function type-checks but does the wrong thing; the test passes but tests the wrong behavior. Those require Tier 2.

The verdict is fed back to the agent as a tool result. The agent sees "pytest failed: test_auth.ts:42 — expected 200, got 401" and can self-correct on the next turn. This is Module 2.2's LLM-recoverable error pattern in action: the computed verifier produces the structured error the loop returns to the model.

## Tier 2: Model-judged verification (the semantic check)

The second tier is an **independent subagent** (Module 1.3, agents-as-tools) that evaluates the primary agent's output against the task's success criteria. This catches what computed checks cannot: does the code actually solve the problem? Does it match the intent? Is it secure? Is it well-structured?

```typescript
interface JudgeInput {
  task: string;                  // original goal
  successCriteria: string[];     // "done and correct" checklist
  artifact: string;              // the primary agent's output
  computedResult: VerificationResult;   // Tier 1 verdict, for context
}

interface JudgeVerdict {
  pass: boolean;
  score: number;                 // 0–1, calibrated confidence
  criteriaMet: string[];
  criteriaMissed: string[];
  reasoning: string;             // why it passed/failed — fed back to the primary agent
  severity: "blocker" | "major" | "minor";
}

const codeReviewAgent: Subagent = {
  name: "code_review",
  description: "Independent reviewer. Evaluates a code change against explicit success criteria. Does NOT modify code — returns a verdict only.",
  run: async (input: JudgeInput): Promise<JudgeVerdict> => {
    const response = await model.complete([
      { role: "system", content: JUDGE_PROMPT },        // no shared context with primary
      { role: "user",   content: JSON.stringify(input) },
    ]);
    return parseJudgeVerdict(response);
  },
};
```

Three properties make this work.

**Independence.** The judge subagent has *no shared context* with the primary agent. It receives the task, the criteria, and the artifact — nothing else. This is the analogue of a code review by an engineer who did not write the code. A judge that shares the primary's context inherits the primary's blind spots; an independent judge can catch them. This is why it is a separate subagent call (Tier 2) and not just a "review your own work" prompt (which empirically underperforms — see 9.3 on self-refinement limits).

**Explicit criteria.** Cherny's "what does done and correct look like?" question is operationalized as a checklist. The judge evaluates against *the list*, not a vague sense of quality. `"Does auth.ts correctly validate JWT expiry?"` is checkable; `"is the code good?"` is not. The criteria turn a subjective judgment into a structured verdict (`criteriaMet`, `criteriaMissed`).

**Calibrated score.** The verdict includes a numeric `score` that the harness can threshold on. A loose harness accepts score > 0.7; a strict one requires > 0.95. The score is also what makes the retry budget (9.4) tractable — you retry while the score is improving and stop when it plateaus.

### Worked example: model-judged verification

Primary agent is asked: *"Add rate limiting to the login endpoint. Done and correct = (1) max 5 attempts per 15 min per IP, (2) returns 429 after the limit, (3) resets on successful login."* The agent edits `auth.ts` and reports done.

**Tier 1 (computed)**: `pytest` passes, `pyright` clean, build succeeds. Gate open — but Tier 1 cannot tell whether the limit is actually 5, whether the window is 15 min, or whether reset-on-success works. Those are semantic.

**Tier 2 (model-judged)**: the judge subagent receives the task, the three criteria, and the diff. It returns:

```json
{
  "pass": false,
  "score": 0.6,
  "criteriaMet": ["(1) max 5 attempts per window"],
  "criteriaMissed": [
    "(2) returns 429 — code returns 403 instead",
    "(3) reset-on-success — counter never resets"
  ],
  "reasoning": "The rate limiter increments correctly and caps at 5, but uses 403 (Forbidden) where the spec requires 429 (Too Many Requests), and the success-path does not clear the counter.",
  "severity": "major"
}
```

The primary agent receives this verdict as a tool result and corrects the two missed criteria. **Tier 2 caught what Tier 1 structurally cannot** — the code compiled and tested green, but it did not match the intent. This is the 2–3× quality gain: the semantic error was caught and fixed before the task was reported complete, instead of shipping as a silent bug.

## Tier 3: Human-in-the-loop (the final gate)

The third tier is a human review, typically before an irreversible action (commit to main, deploy, send email, merge PR). Highest reliability, highest latency, scarcest resource. Tier 3 is not "verify everything by hand" — it is "verify the small set of decisions where a wrong outcome is expensive enough to justify human attention."

Module 6.2's `interrupt()` is the mechanism: the harness checkpoints (Module 8), surfaces the artifact and the Tier 1/Tier 2 verdicts to the human, and waits. The human approves, rejects, or requests changes. This is the approval-gate stop condition from Module 1.2, implemented at the verification layer.

The tiered design exists precisely to *minimize* Tier 3 load. Tiers 1 and 2 catch the bulk of errors cheaply and automatically; the human only sees artifacts that passed both. A harness that escalates everything to Tier 3 causes alert fatigue (Module 6.2); a harness that escalates nothing to Tier 3 ships unreviewed high-stakes changes. **The threshold between Tier 2 and Tier 3 is the risk-tolerance dial of the whole system.**

---

# 9.3 — LLM-as-Judge Mechanics

*Model-judged verification is powerful but has known failure modes. The mechanics determine whether it helps or hurts.*

## Why model-judged verification works

The generator-verifier gap is the core finding of the self-refinement and self-reward literature. Huang et al. (2023, *"Large Language Models Cannot Self-Correct Their Reasoning Yet"*, arXiv:2310.01798) showed that a model correcting *itself* without external signal often *degrades* performance — it talks itself into worse answers. But Yuan et al. (2024, *"Self-Rewarding Language Models"*, arXiv:2401.10020) and the broader LLM-as-a-judge line (Zheng et al., 2023, *"Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena"*, arXiv:2306.05685) showed that a model judging an *independently generated* artifact — with explicit criteria — approaches human-judgment reliability on many tasks.

The distinction is load-bearing: **self-correction without signal fails; independent verification with criteria works.** This is why the Tier 2 design in 9.2 uses a separate subagent with no shared context and an explicit checklist. Get either of those wrong (shared context, or vague criteria) and you regress to the self-correction failure mode.

## Known biases of LLM judges

LLM-as-judge is not a perfect oracle. Three documented biases to design around:

- **Position bias.** When asked to compare two options, the judge prefers whichever is presented first. Cure: evaluate each option independently (absolute scoring), or run the comparison in both orders and check for agreement.
- **Verbosity bias.** Longer answers score higher, independent of quality. Cure: normalize for length, or instruct the judge to penalize unnecessary length.
- **Self-preference.** A judge tends to prefer outputs that look like its own training distribution. Cure: use a different model family for the judge than for the generator (a GPT-class judge over a Claude-class generator, or vice versa), so stylistic preferences do not align.

Zheng et al. (arXiv:2306.05685) quantify these biases and the mitigations; the takeaway for harness engineering is that a naive "ask the model if it's good" judge will inherit all three. The structured `JudgeVerdict` in 9.2 (absolute scoring against explicit criteria, with reasoning) is the mitigation pattern.

## Cost calibration

A model-judged verification call is one extra LLM call per verified artifact — call it ~$0.01–0.05 at current pricing for a modest artifact. The Cherny 2–3× quality claim implies the verification pays for itself whenever the cost of a shipping bug exceeds a few dollars of extra inference. For code that ships, that bar is essentially always met. For throwaway exploration, it may not be — which is why the tight/staged/loose choice (9.1) gates when Tier 2 runs.

---

# 9.4 — Verification Loop Engineering

*The retry budget, flaky-test handling, and verification as a security control.*

## Retry budget math

A verification loop without a budget is Module 7's stuck loop in disguise: the agent fails verification, retries, fails again, retries forever — burning the entire budget on one un-fixable failure. The retry budget is the hard cap: **max N verification failures before giving up.**

The math for choosing N:

```
Let p = probability a single verify-fix cycle resolves the failure.
Expected cycles to resolve = 1 / p.
If p = 0.5 (even chance each cycle), expected = 2 cycles.
If p = 0.2 (stubborn failure), expected = 5 cycles.

Budget N should be set to: the smallest N where (1 - p)^N is acceptably small.
  p = 0.5, N = 5  →  P(still failing after 5) = 0.5^5 = 3%   ← acceptable
  p = 0.2, N = 5  →  P(still failing after 5) = 0.8^5 = 33%  ← not acceptable; raise N or escalate
  p = 0.2, N = 10 →  P(still failing after 10) = 0.8^10 = 11%
```

The discipline: **if after N cycles the failure persists, the agent is not making progress and further retries are wasted.** The correct response is graceful degradation (Module 7.4): report the failure, surface it to the human (Tier 3), and preserve the partial work. This is the verification-layer analogue of the circuit breaker (Module 7.3).

A subtler signal is the *score trajectory*. If the judge's score is climbing across retries (0.4 → 0.55 → 0.7 → 0.85), the agent is converging and a few more cycles are worthwhile. If the score is oscillating or flat (0.6 → 0.62 → 0.59 → 0.61), the agent is stuck and the budget should cut in. **Track the score delta, not just the count.**

```python
def verification_loop(artifact, max_cycles=5, min_improvement=0.05) -> VerificationOutcome:
    history: list[float] = []
    for cycle in range(max_cycles):
        verdict = judge(task, criteria, artifact)
        history.append(verdict.score)
        if verdict.pass:
            return VerificationOutcome(ok=True, cycles=cycle, artifact=artifact)
        # Stagnation detection: no meaningful improvement over last 2 cycles.
        if len(history) >= 3 and history[-1] - history[-3] < min_improvement:
            return VerificationOutcome(
                ok=False, reason="score_stagnant", history=history, artifact=artifact
            )
        artifact = repair(artifact, verdict)   # feed the verdict back to the primary agent
    return VerificationOutcome(ok=False, reason="budget_exhausted", history=history, artifact=artifact)
```

## Flaky test handling

Don't let a flaky test trap the agent. If a test passes intermittently, the agent may retry endlessly against a verifier that is itself unreliable. Patterns:

- **Majority vote.** Run the test 3 times; treat as pass if it passes 2/3. A single flake no longer fails the gate; a genuine failure still fails 3/3.
- **Quarantine.** Move known-flaky tests out of the verification gate into a "warnings" channel. They surface to the human but do not block the agent. The gate is only as reliable as its least-reliable required test.
- **Attribution.** If a test fails, check whether it failed in the *same way* on the prior commit (pre-agent). A test that was already red is not the agent's regression; gating on it traps the agent in fixing a pre-existing bug.

Flaky-test handling is the verification analogue of Module 7.2's transient-error retry: a noisy verifier is treated like a noisy tool — de-noised (majority vote), quarantined, or attributed, never retried blindly.

## Verification as a security control

Beyond quality: verification answers *"does the output do what was intended?"* — which is also *"did the agent do what it was supposed to, not what an injection told it to?"* This dual role makes verification a security control, not just a quality gate.

A computed verification (test suite) catches an injected code change that breaks tests. A model-judged verification catches semantic deviation — the agent was told to send the report to `finance@company.com` but the code sends it to `attacker@evil.com`; the judge, evaluating against the criteria, flags the mismatch. Verification is defense in depth (Module 6's layered stack): even if an injection slips past the input controls, it must also slip past the verifier.

This is why the verifier must be **independent** (9.2): a verifier that shares the primary agent's compromised context will ratify the compromised output. Independence is both a quality measure and a security measure. Course 2 Module S08 covers the offensive counterpart — attacks specifically designed to evade the verifier — and independence is the primary defense there too.

---

## Anti-Patterns

### Verification without a retry budget
The agent fails verification and retries forever. Cure: the retry budget and stagnation detection (9.4).

### The self-reviewing agent
The primary agent reviews its own work in its own context. Self-correction without signal degrades quality (Huang et al., arXiv:2310.01798). Cure: an independent judge subagent with explicit criteria.

### The vague criteria
Judge asked "is this good?" Verbosity bias and self-preference dominate. Cure: explicit, checkable success criteria (the "done and correct" checklist).

### The flaky gate
A verification gate that includes a known-flaky test. The agent retries against noise. Cure: majority vote, quarantine, or attribution (9.4).

### Everything-to-Tier-3
Every artifact escalated to human review. Alert fatigue; the human rubber-stamps. Cure: tier the verification; escalate only what passes Tiers 1 and 2 and is high-stakes.

---

## Key Terms

| Term | Definition |
| --- | --- |
| **Computed verification** | Deterministic checks: linter, type checker, test suite, build |
| **Model-judged verification** | Independent subagent evaluates output against explicit criteria |
| **LLM-as-judge** | Using an LLM to evaluate an independently generated artifact |
| **Three-tier model** | Computed → model-judged → human; layered for cost/risk balance |
| **Retry budget** | Max N verification failures before giving up |
| **Stagnation detection** | Cut the budget when the judge score stops improving |
| **Tight/staged/loose loop** | Verify after each call / at checkpoints / at task end |
| **Cherny claim** | Verification improves output quality 2–3× (practitioner estimate) |
| **Position/verbosity/self-preference bias** | Known LLM-judge failure modes; mitigated by criteria + independence |

---

## Lab Exercise

See `07-lab-spec.md`. Add a computed verification step (pytest) to a custom harness loop. Build a 3-tier verification loop (computed → model-judged → human) with a retry budget and stagnation detection.

---

## References

1. **Boris Cherny / Claude Code** — the 2–3× verification quality claim. From his talk on how he uses Claude Code; surfaced via his X post (x.com/bcherny/status/2007179861115511237) and summarized in *"How the Creator of Claude Code Actually Uses Claude Code"* (Push to Prod). A practitioner estimate, not a peer-reviewed benchmark.
2. **Huang et al. (2023)** — *Large Language Models Cannot Self-Correct Their Reasoning Yet*. arXiv:2310.01798. The case against self-correction without external signal.
3. **Yuan et al. (2024)** — *Self-Rewarding Language Models*. arXiv:2401.10020. Independent generation + explicit reward beats self-correction.
4. **Zheng et al. (2023)** — *Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena*. arXiv:2306.05685. LLM-judge reliability, biases (position, verbosity, self-preference), and mitigations.
5. **Playwright documentation** — visual verification via screenshot/pixel-diff.
6. **Module 1.3** — agents-as-tools; the subagent pattern the Tier 2 judge uses.
7. **Module 6** — verification as a defense-in-depth layer.
8. **Module 6.2** — `interrupt()` for the Tier 3 human gate; alert fatigue.
9. **Module 7** — retry budgets relate to stuck-loop detection; graceful degradation on budget exhaustion.
10. **Module 8** — checkpointing enables the Tier 3 pause/resume.
11. **Course 2 Module S08** — offensive attacks against the verifier (the counterpart to this module's defenses).