# focuschain-ab — does an append-only focus chain improve OSWorld task success?

Date: 2026-07-04 · Model: Sonnet 4.6 via OpenRouter · Bench: OSWorld, 40-task
integrity-filtered pack (20 medium / 20 long), max_steps=50, 960×540, batched
actions, no thinking, prompt caching on, num_envs=12 on a dedicated m5.2xlarge
runner.

**Question.** The cache-aware-context experiment found the best scores in its
notes arm (P3), but that arm also dropped screenshots — confounded. Here we
isolate the notes: **CONTROL = keep-everything. TREATMENT = keep-everything +
append-only `task_notes`** (model fills a required tool-schema field each call;
the harness echoes the latest note at the END of the next user turn; history is
never edited). Branch `focuschain-ab` (commit `b66b99dd`), off
`cache-aware-context`.

## TL;DR

1. **No overall effect.** Paired per-task delta +0.051 (95% bootstrap CI
   [−0.046, +0.151], n=40 tasks × 2 runs/arm). Treatment 24.0 vs control 21.9
   points of 40.
2. **The effect that exists is on MEDIUM tasks, not long ones** — the opposite
   of the "notes = long-horizon memory" intuition. Medium: **+0.178/task, CI
   [+0.050, +0.328]** (treatment won both replicates, 16/17 vs 14/11.9). Long:
   −0.075, CI [−0.175, +0.025] — and the 24-task long-pack extension
   replicated the negative lean (−0.098 [−0.318, +0.122]); pooled 42-task
   long-horizon Δ = **−0.087 [−0.210, +0.033]**.
3. **Why (from trace forensics):** notes act as an *obligation/progress
   ledger* — "need to save", "intro done, body done, now conclusion" — which
   pays off on ordinary multi-step tasks. On long tasks that value is offset by
   a real failure mode: **plan fixation** (append-only notes re-anchoring a
   failing approach for 40+ steps) plus env-flake/step-limit noise that notes
   cannot fix.
4. **The mechanism is free and harness-clean:** hit rate 0.921 vs 0.928, zero
   context edits in 160 task-runs, cost $0.410 vs $0.415/task (treatment used
   **12.5% fewer steps**; its +16% output tokens are the notes themselves).
5. **Benchmark noise dominates single runs:** 8/40 tasks flipped outcome
   between the two *identical* treatment runs (control replicates: 23 vs 20.9).
   Any single-run OSWorld comparison of ±4 points is noise-compatible.

## Design

- Pack: `trials/shared/osworld-40-pack/` — from `test_nogdrive.json`, stratified
  by Pointer's published sonnet-4-6 step counts (medium 9–15, long ≥16),
  excluding all tasks flagged by the benchmark-integrity audit (broken/drifting
  evaluators, gate false positives) and all 29 infeasible-type tasks.
- 4 serialized blocks, ABBA: control-1 → treatment-1 → treatment-2 → control-2,
  40 tasks each, same runner, same num_envs.
- Smoke test (2 tasks, treatment): PASS — zero edits, notes on every tool call;
  the only cache invalidations were >5-min TTL expiries during slow env
  commands (both arms suffer these equally; see steps table in
  `results/focuschain-ab-smoke/`).
- Analysis pre-specified: per-task mean score (2 runs averaged), paired across
  arms; bootstrap CI (10k, task resampling); McNemar exact on binarized means;
  strata; $/task; cache hit rate as the tamper detector.

## Per-block results

| Block | Score /40 | medium /20 | long /20 | Steps | Hit rate | TTL inval | Cost |
|---|---:|---:|---:|---:|---:|---:|---:|
| control-1 | 23.00 | 14.0 | 9.0 | 1078 | 0.927 | 37 | $15.72 |
| treatment-1 | 22.00 | 16.0 | 6.0 | 1007 | 0.923 | 28 | $16.86 |
| treatment-2 | 26.00 | 17.0 | 9.0 | 968 | 0.919 | 25 | $15.92 |
| control-2 | 20.89 | 11.9 | 9.0 | 1180 | 0.929 | 34 | $17.51 |

Zero policy edits/compactions/trims in all 160 task-runs (by construction in
both arms; verified per-step from usage telemetry). Cost accounting matches
OpenRouter upstream cost exactly (same rule as cache-aware-context:
`summarize_results.py`).

## Paired analysis (2 runs/arm averaged per task)

| Stratum | Δ/task (T−C) | 95% CI | McNemar (C-only/T-only wins) |
|---|---:|---|---|
| Overall (40) | +0.051 | [−0.046, +0.151] | 1 / 2, p=1.0 |
| Medium (20) | **+0.178** | **[+0.050, +0.328]** | 0 / 2, p=0.5 |
| Long (20) | −0.075 | [−0.175, +0.025] | 1 / 0, p=1.0 |

Per-run sensitivity: c1-vs-t1 discordant 5/4 (p=1.0); c2-vs-t2 discordant 2/7
(p=0.18). McNemar is underpowered after averaging (most tasks concordant); the
bootstrap CI is the operative test. Two strata ⇒ two comparisons: treat the
medium result as strong-suggestive, not definitive.

## Trace forensics — the 9 run-1 discordant tasks

Three parallel analyst passes over both arms' full trajectories
(control-1 vs treatment-1):

**Treatment wins (4):**
- chrome/2ae9ba84 — NOTES-CAUSED: note carried "Need to confirm/save the
  change"; treatment pressed Enter+verified; control typed the value, declared
  DONE uncommitted, and the evaluator's Chrome restart wiped it.
- multi_apps/9219480b (tetris bug) — NOTES-CAUSED (med conf.): both arms found
  the identical fix and botched the first edit identically; treatment's note
  pinned the fix spec + "Will use Find & Replace" and executed; control
  abandoned the same idea and thrashed through 4 mechanisms to step 50.
- libreoffice_writer/b21acd93 — notes-as-progress-ledger: treatment recovered
  from the same dialog-timing env quirk in 1 step each time (note: "Intro done.
  Body done. Now third paragraph"), finished in 18 steps; control re-derived
  state repeatedly, ended at step 50 with a modal dialog open and never saved.
- libreoffice_calc/37608790 — STOCHASTIC (formulas vs hand-retyping 21 rows
  with 2 typos + premature DONE); notes only aided treatment's mid-task
  recovery.

**Treatment losses (5):**
- thunderbird/7b1e1ff9 — **NOTES-CAUSED FIXATION** (the one genuine anti-notes
  case): from step 7 the notes prescribed a dev-console/tabmail route and
  re-anchored it for 43 steps (verbatim failed JS carried in notes); control
  bailed on the same idea at step 9 and won via Help → Troubleshooting.
- multi_apps/a82b78bb — ENV: VM booted in a different window state; terminal
  corrupted every keystroke (~21 steps to launch Chrome). Notes were flawless
  (4-author URL ledger held 50 steps); ran out of steps.
- vs_code/6ed0a554 — ENV: same UI freeze hit both arms; control lucked into the
  only working path (dock right-click).
- vs_code/930fdb3b — STOCHASTIC: GUI keybinding editor structurally cannot emit
  the `when: terminalFocus` clause the evaluator requires; control hand-wrote
  JSON.
- vs_code/c6bf789c — STOCHASTIC premature DONE: typed config without
  select-all → `}{` invalid JSON; misread the final screenshot ("has been
  saved") with an error badge visible.

Pattern: notes' upside = commitment/ledger (2–3 wins); downside = fixation
(1 loss); the rest is benchmark noise. Notes were factually accurate in every
trace examined — no wrong-fact poisoning observed.

## Mechanism — trace-verified (added after long-run forensics)

Statistical signature on the 128 long-task runs (64/arm):

| | treatment | control | Fisher p |
|---|---:|---:|---|
| DONE declared but task NOT solved | **23 (36%)** | 6 (9%) | **0.0006** |
| Ran to the 50-step cap | 14 | 29 | 0.008 |
| DONE precision | 0.52 | 0.82 | — |
| Correct DONEs | 25 | 27 | — |
| Mean actions per tool call | 3.43 | 2.57 | — |
| Blind batches ≥10 actions | 3.8% | 1.7% | — |

The notes do not reduce genuine completions; they **quadruple false ones** and
embolden larger blind action batches. Score deltas alone are noise-compatible
(permutation p=0.21) — the behavioral shift is not.

**The verified failure mode is a write-once completion ledger.** A success
claim enters `task_notes` at the moment of action — often before observing the
result — is echoed back every subsequent turn, and from then on (a) verification
effort is allocated only to items the ledger says are open, and (b) the final
check certifies against the ledger's own target rather than re-deriving from
the instruction. Control agents, having no ledger, re-derive completion from
pixels every turn and catch identical environment traps. Decisive cases
(adversarially verified, rival explanations checked):

- impress/841b50aa: both arms typed "APP" into the wrong text frame. Control's
  next step: "it appears in a text frame in the middle of the slide rather
  than in the notes" → undo, fix, verify, score 1. Treatment's step-5 note:
  "Note 'APP' added successfully (confirmed in status bar)" — repeated as fact
  in every note thereafter, DONE at 14, score 0.
- writer/adf5e2c3 (both treatment runs): note flipped from "Need to check if
  there's a duplicate [14] and fix it" (step 7) to "format looks OK" (step 8)
  with no fix; the certified line persisted verbatim 11 steps; final
  verification only targeted the ledger's one open item.
- calc/7a4e4bc8: the step-7 note compressed the instruction into a wrong
  column target, dropping the "don't touch irrelevant regions" constraint;
  ~20 notes repeat it; the final check verified "matches the target" — the
  note's target, not the instruction's.
- gimp/2a729ded: notes asserted "looks great / DONE visually" for an image
  with transparency holes; the one verification loop chased only the note's
  single open item. Control made the same selection error and caught it on
  screen.

Not everything fit the pattern (honesty tally, 8 solvable premature-DONE cases
examined): 4 decisively note-certified, 1 partial (plan lock-in + strict
evaluator), 3 other (blind-batch XML corruption an on-screen check could not
catch; GUI-path strategy flip; screen misread). Control also premature-DONEs —
just 4× less often (and one control loss was itself a confident wrong DONE).

**The verified upside is the same ledger working forward:** obligation
carrying ("need to confirm/save" → the save happens), method pinning (note
fixed "use Insert > Pivot Table" at step 1 while control hand-typed a fake
pivot and confidently failed), and checklist recovery (treatment accidentally
reordered slides, then rebuilt state from its note ledger in 9 steps while
control thrashed 30 steps to the cap on the same trap). On medium tasks the
ledger stays short and the upside dominates (+0.178/task, CI excludes zero);
on long tasks the accumulated "done list" becomes self-certification and the
net flips slightly negative.

**Design implication — the harm is plausibly implementation-specific.** Four
choices in THIS focus chain map 1:1 onto the verified failure signatures:
(1) the schema asks for "what is done" → collects status claims;
(2) notes are written in the SAME tool call as the actions they describe →
claims are born pre-observation ("added successfully" written while typing);
(3) the field is required every call → cheapest valid output is restating the
prior note (staleness, "format looks OK" ×11 steps, wrong target ×20);
(4) the ~120-word cap compresses instructions lossily → constraints drop out
("don't touch irrelevant regions"), and the final check runs against the
compressed target. The prompt even nudges against re-verification ("avoid
re-deriving facts").

**Pre-registered P3b** (next A/B, ~$35 on the 42-long-task set, same branch):
- Observation-gated claims: substeps may move to `verified_done` only in a
  step AFTER a screenshot confirmed them; schema splits
  `verified_done` / `did_but_unverified` / `remaining`.
- Remaining-first framing: primary field = "what remains + exact next check",
  not an accomplishment list.
- Optional field (write only on state change) to kill copy-forward staleness.
- Verbatim constraint slot: instruction constraints copied, never paraphrased.
Expectation management: ~half the long-task failures here were env flakes,
evaluator strictness, or step-cap censoring no note format can fix; P3b's real
test is 150+ step horizons, beyond OSWorld's task supply.

## focus_chain_v2 validation run (2026-07-05) — the engineered fix DID NOT restore precision

Commit `2d8ca929`: structured ledger (constraints verbatim / remaining-first /
next_check / verified_done with observation-gated tense rule) + harness
DONE-gate (no-tool-call completion rejected once while `remaining` non-empty,
converted to a forced fresh screenshot + feedback). Smoke (5 hand-picked
tasks, $1.48) validated the machinery: writer/adf5e2c3 — failed by v1 twice
via premature DONE — was SOLVED (ledger held the pending item 9 extra steps);
gate fired exactly once and produced the correct action (forced save); no
nagging on empty ledgers.

Full run: 2 blocks × 40 tasks (results/focuschain-v2/), same config/pack/
runner recipe, paired vs the original control and v1 blocks:

| per block | control | v1 | v2 |
|---|---:|---:|---:|
| Score /40 (task-avg) | 21.9 | 24.0 | 22.0 |
| Steps | 1129 | 987 | 1014 |
| Wall (task-hours) | 11.8 | 10.8 | 11.4 |
| $ | 16.62 | 16.39 | 18.35 |
| Long DONE-right / DONE-wrong | 15 / 4 | 14 / 15 | 16 / 15 |
| **Long DONE-precision** | **0.79** | 0.48 | **0.52** |
| Long cap-hits | 21 | 9 | 8 |

Paired deltas: v2−control overall +0.001 [−0.111,+0.114]; long −0.025
[−0.175,+0.125]. v2−v1 overall −0.050 [−0.138,+0.037]. All null.

**Pre-registered verdict: prediction 1 (speed preserved) HELD — 1014
steps/block, −10% vs control. Prediction 2 (DONE-precision ≥ 0.75) FAILED —
0.52, statistically indistinguishable from v1's 0.48.** The gate fired only
5 times in 80 tasks (log-counted; a telemetry bug kept `done_gate_fired`
context_events out of traj usage rows — behavior unaffected): models obey the
schema, "verify" items on a screenshot glance they then misjudge
(status-bar-style misreads), the ledger empties, and the gate — which can
only audit the declaration against the model's own ledger — correctly waves
through a wrong belief. v2 converted "stopped without looking" into "looked
carelessly, then stopped."

**Program conclusion (366 task-runs):** a focus chain is a termination
accelerator — it reliably trades control's unproductive grinding (21 cap-hits
→ 8) for confident early conclusion at ~10–13% fewer steps, equal accuracy,
equal cost. Its false-completion cost is NOT a schema defect: observation
gating, tense discipline, and completion gating do not reduce it, because the
binding constraint is the model's screenshot-verification QUALITY, not the
note structure. Scaffold-level interventions on context cannot buy back
termination precision; that likely requires better perception/verification
behavior in the model itself (or an external verifier). This closes the
focus-chain line on this benchmark.

## Noise quantification

- Same-config flips: treatment-1 vs treatment-2 = **8/40 tasks**; control
  replicates differ by 2.1 points. Cross-arm run-1 discordance (9/40) is the
  same order — most single-run "effects" are flips.
- TTL-expiry invalidations (>5-min steps, both arms): 124 across 160 runs;
  zero caused by history edits.

## Cost

Treatment $32.78 vs control $33.23 for 80 task-runs each — **the focus chain
is cost-neutral** (notes' +69k output tokens ≈ +$1.0, offset by −283 steps).
Total mission spend incl. smoke + extension: **≈$90.6** of the $120 cap
(per-trajectory upstream costs; the shared OpenRouter key makes account-level
deltas unusable — another experiment ran concurrently).

## Ops log

- 21:32–00:12Z: runner provisioned (m5.2xlarge); worker SG lacked
  runner→worker ingress (only controller SG allowed) — added self-referencing
  rules on 5000/5910/22/9222. Smoke PASS. Blocks started 00:11Z.
- 06:16Z: external stop/restart of the `fc-orchestrator` systemd unit (shared
  controller); restarted copy collided with the live block unit and aborted.
  Recovered via idempotent DONE-marker takeover (`fc2-finalize`); ~6 min gap,
  no data loss.
- 09:10Z: control-L deadline-stopped at 22/24 per plan.
- Cleanup: all 26 worker VMs self-terminated via the manager's TTL schedules
  (verified); runner terminated 09:22Z. Final fleet = controller only.
  Zero leftovers.

## Extension: focuschain-ab-long (pre-registered mid-run)

Motivated by the long-stratum power problem: OSWorld contains only **41
structurally-long feasible unflagged tasks** (both published Pointer models
needed ≥20 single-action steps — the both-model minimum filters thrash-length).
`trials/shared/osworld-long-pack-candidates/candidates.csv` lists them;
`long-pack-24.json` = the 24 longest not already in the main pack (10 calc,
8 impress, 6 multi_apps; structural length 22–45). Config identical to the
main run so the 10 overlapping long tasks pool. Blocks: treatment-L →
control-L, 1 run/arm (budget-bound).

**Results** (treatment-L 24/24; control-L 22/24 — 2 tasks clipped by the
09:10Z deadline stop and excluded from pairing):

| | control-L | treatment-L |
|---|---:|---:|
| Score (22 paired) | **13.2** | **11.0** |
| Hit rate / edits | 0.938 / 0 | 0.931 / 0 |
| Cost | $10.96 | $12.82 (24 tasks) |

- Extension paired delta: **−0.098/task**, 95% CI [−0.318, +0.122];
  discordant 4 control-won vs 2 treatment-won.
- **Pooled long-horizon (42 tasks: 20 main ×2 runs + 22 ext ×1):
  Δ = −0.087/task, 95% CI [−0.210, +0.033].**

Two independent long-task samples now agree: the focus chain does **not** help
genuinely long tasks; the point estimate is mildly negative in both (main −0.075,
extension −0.098), with the pooled CI nearly excluding zero on the negative
side. Combined with the medium-stratum gain, the effect gradient is the
*inverse* of the "notes = long-horizon memory" intuition — consistent with the
trace-forensics mechanism: the ledger/commitment benefit applies at any length,
while the fixation cost (append-only notes re-anchoring a failing plan) grows
with trajectory length, and long-task failures are increasingly env/step-limit
noise that notes cannot fix.

Definitive follow-up design (pre-registered): 3+ runs/arm × all 41 candidates
(~$150): with ~20–25% per-task flip noise, replicates beat breadth. Also worth
testing: a fixation-resistant note format (e.g. required "what has NOT worked"
field, or instructing the model to rewrite rather than restate its plan when
the last 3 steps made no progress).

## Caveats

- Medium-stratum CI excludes zero but survives no multiple-comparison
  correction with margin; replicate before acting on it.
- max_steps=50 censors: 6/160 runs ended at the cap mid-path; long-task
  failures are part env/censoring, so "notes don't help long tasks" is
  entangled with "long tasks fail for reasons notes can't fix".
- 1 orchestration incident (external restart of the systemd unit at 06:16Z)
  cost ~6 min between blocks; no data loss (see orchestrator.log).
- Score threshold for McNemar binarization = mean ≥ 0.5; OSWorld partial
  credits make binarized tests coarse — bootstrap on raw scores is primary.

## Artifacts

- Runs: `results/focuschain-ab/{control-1,treatment-1,treatment-2,control-2}/`
  (+ DONE markers), smoke in `results/focuschain-ab-smoke/`.
- CSVs: `trials/focuschain-ab/summary/focuschain-ab/{results.csv,steps.csv,summary.md}`;
  `trials/focuschain-ab/results.csv` (copy); `analysis.json` (paired stats).
- Code: OSWorld branch `focuschain-ab` (worktree `repos/OSWorld-focuschain`);
  scripts in `trials/focuschain-ab/scripts/` (pack builder, run/orchestrate,
  smoke check, summarizer, analyzer).
- Packs: `trials/shared/osworld-40-pack/`, `trials/shared/osworld-long-pack-candidates/`.
