harness-digest · 2026-W45
Week 45from 2 Nov 2026
Most of this week's items argue that agent reliability comes from the scaffolding around the model rather than the model itself, and that the scaffolding only counts if it is enforced. Rules in AGENTS.md drift unless CI checks them, tool guards can quietly fail open, and agents declare victory early unless a hook runs the real test. The same idea shows up in the unattended-loop harness, where a second agent must pass the work before anything ships.
- scans
- 2
- passed the judge
- 21
- carried over
- 4
- picked
- 7
47 web searches, 2.0M tokens read. Every run writes a log: see them on GitHub.
9/10
learn-harness-engineering
Agents fail on real repos from a weak environment, not a weak model; this course teaches a file-based harness covering instructions, state, verification, scope and session lifecycle, with projects and templates.
How it works
- Add AGENTS.md as a short map to deeper docs, rather than one giant instruction file.
- Keep feature_list.json as machine-readable scope so the agent works on exactly one unfinished feature.
- Run init.sh at session start to install, verify and health-check before any work begins.
- Log progress in a markdown file and commit clean state so the next session resumes exactly.
- Count only passing tests, lint and type checks as evidence; the agent records proof before claiming done.
Steal this: Before your next multi-session agent task, drop in AGENTS.md, init.sh, feature_list.json and a progress log, and require the agent to update them before stopping.
Go deeper if you're already running coding agents on real repos and want templates, a harness audit script and projects comparing weak versus strong harnesses.
“The MODEL decides what code to write. The HARNESS governs when, where, and how it writes it.”
9/10
Stop trusting your AGENTS.md: the drift-check that makes it a contract
A lean AGENTS.md is still just prose that quietly goes stale, so tag each rule with a marker and run a CI script that fails when rules and checks fall out of sync.
How it works
- Put an invisible HTML marker like <!--check: single-lockfile--> next to each rule in AGENTS.md, so the prose stays readable.
- A small Node or Python script extracts marker ids and maps each to a verification function returning pass, fail, or offending files.
- Two-way binding: a marker with no mapped check fails, and a check with no marker fails.
- Run it as its own GitHub workflow on pull requests and pushes, exiting 1 on any drift.
- Make checks inspect real code, such as actual imports, rather than only config keywords, and update rules in the same PR as the change.
Steal this: Before your next AGENTS.md edit, add a check marker to each rule and write a short script that fails CI when any marker lacks a mapped check, or the reverse.
Go deeper if you're tired of agents quietly violating conventions in your instruction file and want a concrete CI pattern plus the failure modes of building it.
“A rule that nothing can break isn't a rule; it's a suggestion the next agent will politely decline.”
8/10
markhazlett/agent-harness: Portable coding agent harness
Coding agents drift into destructive or sloppy habits, so this repo wraps one in hooks that block risky commands, skills with fixed rules, and a weekly plan-build-ship workflow.
How it works
- A bash-guard hook blocks commits on main, --no-verify, sed -i on source, and rm -rf on source dirs.
- A protected-files hook stops edits to .env, hook scripts, settings.json and the lockfile, guarding the guards.
- Session-start and stop hooks inject branch and handoff notes, then run tests and typecheck if source changed.
- Verification skills carry a one-sentence Iron Law plus a Rationalization Table of real excuses the model used under pressure.
- A read-only validator agent runs tests and lint, and /orchestrate loops builders and validator up to 3 cycles.
Steal this: The next time an agent talks itself out of a test or gate, record its exact excuse in a Rationalization Table inside the relevant skill, next to a one-line Iron Law.
Go deeper if you're building your own hooks, skills and guardrails and want a complete, working example to copy from.
“Hooks are the one exception (they catch destructive shell commands), and they too are configurable.”
8/10
loop-harness: scheduled agent loops with worktree isolation and a second-agent verification gate
Unattended agents risk shipping unchecked work, so this harness has the primary agent only stage outputs in an isolated worktree, and a skeptical second agent must pass them before anything is pushed or posted.
How it works
- A scheduler wakes each loop on a cadence and gives a Claude session a task-specific skill file.
- Write loops run in a fresh git worktree on their own branch, never touching your checkout.
- The primary agent can't push or post; it stages commits, a PR body, or outbox action files.
- A second read-mostly claude session inspects the diff, runs cheap checks, and must print VERDICT: PASS.
- Each skill ends with a RESULT line whose stable item IDs feed a state file for deduplication, so reruns are safe.
Steal this: Before your next unattended agent job, split it into a stage step that writes outputs to files and a separate verifier session that must print an explicit PASS before anything ships.
Go deeper if you're running agents on a schedule or in CI and want a small shell-based template for isolation, verification gates and idempotent state.
“You don't prompt Claude ; loops do.”
8/10
How We Optimized Browser Use for Deep Agents
Browser agents burn context and turns because tools return huge page dumps one step at a time; code-mode tools, compact snapshots and trace-driven fixes cut cost and time while accuracy held.
How it works
- Let the agent write one code snippet that chains many browser steps, so intermediate page content stays in variables, not context.
- Return a trimmed accessibility tree with IDs on interactive elements instead of raw HTML or screenshots.
- Trim output at the source: capping snapshots after the fact just made agents refetch the page.
- Read traces to spot waste, such as fixed 8-second sleeps, then end sleeps once the page is quiet for 500 ms.
- Run evals on every change before keeping it, comparing setups side by side as experiments.
Steal this: The next time your agent's tool returns bulky output, read a few traces for repeated waste, change the tool to return less or batch steps, and rerun your evals before keeping the change.
Go deeper if you're building or tuning tools for a long-running agent and want measured before-and-after numbers on cost, turns and accuracy.
“Traces show where the agent wastes steps and tokens, and evals tell you whether a change helped.”
8/10
Your agent tool guard probably fails open. One minute to tell.
Agent tool guards often fail open because the orchestrator proceeds unless told to block; probing the guard with crash, silent, and odd-input cases shows which paths actually refuse a call.
How it works
- Exit code 2 is the only blocking code in Claude Code hooks; crashes, exit 1 and a missing script all let the call proceed.
- Make the guard raise once, print nothing once, and exit 2 once, then check the transcript for which refused.
- Pipe variant command strings into the hook: quoted paths, git -C, quoted subcommand names, and empty input, printing each exit code.
- Put the deny where the side effect is decided, since settings edits only gate operations that haven't resolved policy yet.
- Turn each permission layer off in turn to find which one actually decides, and fail closed whenever the guard can't reach a verdict.
Steal this: Before you trust your next PreToolUse hook, pipe the five sample command strings into it and confirm that crash, empty input and a quoted-path variant all exit 2.
Go deeper if you're running coding agents with hooks or deny rules and want to know whether those guards really stop anything.
“A guard that "crashes loudly" is a guard that fails open loudly.”
8/10
Claude Code Stop Hook: Block Early Exit Without Loops
Agents often declare done too early. A Stop hook runs your real check, such as tests passing, blocks with a specific reason while it fails, and gives up after a capped number of retries.
How it works
- Register a command hook under Stop in .claude/settings.json; Stop takes no matcher, so it fires on every finished turn.
- The script reads session_id and stop_hook_active from stdin, then runs npm test and checks that the output file exists.
- On failure it prints JSON with decision block and a reason containing the last 30 lines of test output.
- A per-session counter file caps retries at about 3, then exits with a systemMessage and logs GAVE_UP for cron to grep.
- Build the JSON with jq, send test output to a file not stdout, and pipe sample input in by hand to test.
Steal this: Before your next unattended or overnight run, copy the verify-before-stop.sh script, swap in your real done check, set a retry cap of 3, and add a grep for GAVE_UP in your wrapper.
Go deeper if you're running Claude Code unattended in cron or CI and need a deterministic, budget-capped done check, or you're debugging a Stop hook that won't fire.
“A Stop hook lets a script that you control decide whether the work is actually done.”
What the judge turned down this week (11)
Verdicts were only logged from partway through this week, so this list is incomplete.
- Why human insight is a strategic advantage for building better agents
Calibrating an LLM judge against expert human scores on full threads is portable, but it is covered only at a high level with no code, rubric or before-and-after, and it wraps a LangSmith feature launch; the card's '20 threads' step isn't on the page.
- Building Legal Agents When You Can't See the Data - Jacob Lauritzen, CTO of Legora
A podcast page with only show notes and chapter titles: simulated test worlds and grouping paraphrased traces into 10–20 failure types are named as topics but not taught, so the card's technique isn't on the page and it's nowhere near the 8+ a bonus listen needs.
- New context engineering challenges in late 2026
The post lists open context-engineering problems for always-on agents and explicitly says there are no general solutions, so it teaches no technique; the card's 'Steal this' step of enumerating ways an agent could game a goal and adding a check for each is not in the page, which only poses the question.
- LLMs are not superhuman at reading and writing neural network weights
An opinion piece rebutting a paper's superhuman claims; the weight-norm heatmap and steering/abliteration steps are interpretability research, not agentic-engineering harness techniques, so the card's "steal this" stretches the page into something it doesn't teach.
- Qwen 3.8 27B on my laptop can interpret model weights, too
This is a capability check on a research paper (can a local model read raw weights?), not agentic engineering craft; the only liftable trick is running a prompt URL through the llm CLI with -f, which doesn't help anyone shape or verify agent output.
- See the agents, set the terms: new visibility and control for personal agents
This is a Cloudflare product launch about site owners controlling personal-agent traffic, not about building agent harnesses; the per-agent analytics and allow/block policies only work inside Cloudflare's dashboard, so there's no portable agentic-engineering technique.
- F3 Lightning: How Meta Accelerates Ads ML Features to Production at Scale
A solid infrastructure write-up on per-feature releases, content-derived node IDs and startup-time execution planning for ML feature serving, but it teaches nothing about shaping AI agents, so it is off-topic for agentic engineering, as the card itself admits.
- Learn the rules: The hacker mindset
A career-profile podcast about hacker culture and DARPA's Cyber Grand Challenge with no concrete agent-harness technique; the card's 'steal this' is generic leadership advice, not something the episode teaches about shaping agents.
- The U.S. Census Bureau and “Super Intelligence”
An analysis of Census survey wording changes with no agent technique; the card's own idea admits as much, and its 'steal this' is invented rather than taught by the page.
- Audience capture
An opinion essay on proxy metrics and audience capture in blogging with no agentic engineering technique, so it fails the teaching-moment test (the card itself admits it).
- LangSmith BYOC is now available on AWS, Azure, and Google Cloud
A product launch announcing LangSmith BYOC on three clouds, with no agent technique: the control/data plane split and migration steps only apply inside the vendor's product.
This issue as markdown, as the harness committed it.