Lab Specification — Module 9: Verification & Feedback Loops

Module: 9 · Duration: 40 min · Environment: Codespace, Node 18+ or Python 3.11+.

Learning objectives

  1. Add a computed verification step (test suite) to a harness loop.
  2. Build a 3-tier verification loop (computed → model-judged → human) with a retry budget.

Phase 1 — Computed verification gate (15 min)

import subprocess
def verify_computed(solution_path: str) -> dict:
    # Tier 1: syntax check + compile + test suite
    syntax = subprocess.run(["python3", "-c", f"import ast; ast.parse(open('{solution_path}').read())"], capture_output=True)
    if syntax.returncode != 0: return {"passed": False, "tier": "computed", "error": "syntax error"}
    tests = subprocess.run(["python3", "-m", "pytest", "--tb=short"], capture_output=True)
    return {"passed": tests.returncode == 0, "tier": "computed", "output": tests.stdout.decode()[-500:]}

Wire into the loop: agent writes solution → verify_computed → if fail, return error to model (self-correct); if pass, proceed.

Phase 2 — 3-tier with retry budget (20 min)

Add Tier 2 (model-judged via independent LLM call) and a retry budget (max 3 failures):

def verify_3tier(solution, budget=3):
    for attempt in range(budget):
        c = verify_computed(solution)
        if not c["passed"]: return c  # fail fast on computed
        m = verify_model_judged(solution)  # independent subagent
        if m["passed"]: return {"passed": True, "tiers": ["computed","model-judged"]}
        # else: model-judged failed → return to model for self-correction, retry
    return {"passed": False, "error": f"retry budget ({budget}) exhausted"}

Deliverables

Stretch goals

  1. Add visual verification (Playwright screenshot) for a UI task.
  2. Add flaky-test handling: run a flaky test 3×, majority vote.
# Lab Specification — Module 9: Verification & Feedback Loops

**Module**: 9 · **Duration**: 40 min · **Environment**: Codespace, Node 18+ or Python 3.11+.

## Learning objectives
1. Add a **computed verification step** (test suite) to a harness loop.
2. Build a **3-tier verification loop** (computed → model-judged → human) with a retry budget.

## Phase 1 — Computed verification gate (15 min)
```python
import subprocess
def verify_computed(solution_path: str) -> dict:
    # Tier 1: syntax check + compile + test suite
    syntax = subprocess.run(["python3", "-c", f"import ast; ast.parse(open('{solution_path}').read())"], capture_output=True)
    if syntax.returncode != 0: return {"passed": False, "tier": "computed", "error": "syntax error"}
    tests = subprocess.run(["python3", "-m", "pytest", "--tb=short"], capture_output=True)
    return {"passed": tests.returncode == 0, "tier": "computed", "output": tests.stdout.decode()[-500:]}
```
Wire into the loop: agent writes solution → verify_computed → if fail, return error to model (self-correct); if pass, proceed.

## Phase 2 — 3-tier with retry budget (20 min)
Add Tier 2 (model-judged via independent LLM call) and a retry budget (max 3 failures):
```python
def verify_3tier(solution, budget=3):
    for attempt in range(budget):
        c = verify_computed(solution)
        if not c["passed"]: return c  # fail fast on computed
        m = verify_model_judged(solution)  # independent subagent
        if m["passed"]: return {"passed": True, "tiers": ["computed","model-judged"]}
        # else: model-judged failed → return to model for self-correction, retry
    return {"passed": False, "error": f"retry budget ({budget}) exhausted"}
```

## Deliverables
- [ ] Phase 1: computed gate code; confirmation it catches syntax errors and test failures
- [ ] Phase 2: 3-tier loop; retry budget enforced; model-judged catches a semantic error computed misses

## Stretch goals
1. Add visual verification (Playwright screenshot) for a UI task.
2. Add flaky-test handling: run a flaky test 3×, majority vote.