songoow · October 16, 2026 · Interim research note · Evidence cutoff 03:10 UTC
-Why the agent stood still after 24 hours
-One repair round for LoopX long runs: four failure classes, who owns them, and what remains unproven
-The previous article recorded interim progress on Portfolio and Lean. This round put the same system on a longer clock: 24-hour paired runs against native Codex, plus 90-minute LHTB tasks to test whether goals close too early. The results are not flattering, but they are clear. The LoopX agent kept working while its score did not move; the native control changed methods three times in the same window.
-This article explains what happened behind those curves. Each of four failure classes has a locatable cause: the first external feedback arrived too late; every review point crashed a wake; review was allowed to “retain the path”, so ten hours went into tuning one solver; and on LHTB the goal declared itself done while the hidden score was far from passing. We label every repair as a product change or a harness change—that distinction decides what a rerun can prove.
-Our reading: this round mostly exposed product problems, not model problems. Native Codex reached better scores with the same model. Some repairs are complete and verified against the real CLI; others are only directions. The text keeps them apart.
+songoow · Oct 16, 2026 · Exploratory notes · Historical cutoff 03:10 UTC
+EdgeBench / LHTB: notes from historical runs
+Historical builds, interrupted runs and untested hypotheses
+These are the author’s exploratory notes on historical builds, not official or formal LoopX evaluation results. They do not establish current capability, efficiency or relative performance.
+The option was withdrawn: #6612 removed --feedback-first and its planning switch. Scores below come from historical combined builds; they neither evaluate the current version nor isolate one change.
This round records Borden and Grid runs with planned 24-hour budgets that ended early, plus 90-minute LHTB pilots. The traces help locate engineering issues in submission order, review handoff, repeated tuning and closure evidence. The article separates reproducible local defects, explanations not yet isolated and the author’s proposed follow-up tests, retaining failures, interruptions and counterexamples.
1. What this round asks
-The last round concluded that the system as a whole had become more effective, but that contributions were not yet separated. This round switched to two task families to approach a sharper question: when a task has a real external evaluator and the budget is long, does LoopX’s control plane make the agent use that time better, or merely keep it awake?
-| Task | Score direction | Comparison | Budget | Status |
|---|
| Task | Score direction | Comparison | Planned budget | Status |
|---|---|---|---|---|
| EdgeBench Borden source inversion | higher is better | LoopX heartbeat (planned entry) vs native Codex | 24 h, native feedback, 1 h submission cooldown | Two pairs; first interrupted at 12.4 h, second paused at 3 h |
| EdgeBench Grid Turing robot | lower is better | same | 24 h, same | Paused at 11.7 h |
| EdgeBench Borden / codec / rust / jagua | higher is better in this group | Combined LoopX candidate, native official, native-goal | 90 min local, 15 min cooldown | 13 scored runs: 11 normal finals, 2 runner failures; another 2 unscored startup failures |
| EdgeBench Borden / codec / rust / jagua | higher is better in this group | Historical combined LoopX candidate, native official, native-goal | 90 min local, 15 min cooldown | 13 scored runs: 11 normal finals, 2 runner failures; another 2 unscored startup failures |
| LHTB four tasks (covshift and others) | hidden-set reward 0–1 | LoopX heartbeat, single arm, against a historical five-arm study | 90 min | Four pilots finished; covshift rerun with the acceptance gate finished |
EdgeBench used gpt-6.1-sol / xhigh; both arms share the model, image and evaluator. LHTB used gpt-5.6-sol / max. “Native feedback” means the agent may submit and receive a public score, with no further submission during the cooldown. The original Borden and second Grid LoopX arms enabled checkpoint capture. The second Borden pair and local combined candidates disabled checkpoints but explicitly enabled TurnEnvelope. These treatments are not one default configuration; section 7 separates them.
Several runs did not finish. Two causes: our own experiment controller failed (a container-count guard was tripped by another cohort’s transient judge container; the controller crashed reading a container that was exiting), and we deliberately paused everything while writing this article, to repair before rerunning. Interrupted runs remain evidence—the curves had already made their point—but they are not terminal scores.
- +Several runs did not finish. Experiment-controller failures included a container-count guard tripped by another cohort’s judge container and a crash while reading an exiting container; other runs were deliberately paused while writing. Pre-interruption traces can support particular fault diagnoses, but missing later windows and normal closeout cannot be filled in as completed 24-hour results.
+2. What the score curves show
-
Three phenomena appear on both tasks, independently of each other.
-The first external feedback arrives late. On Borden, native Codex had its first score at 13.6 minutes; LoopX at 48.2. On Grid it was 21 versus 12 minutes, but LoopX’s first submission was a 3593 placeholder. In a task that accepts one submission per hour, 35 minutes late shifts the whole curve right by half a cooldown.
+The following observations have different scopes within the historical runs. They suggest follow-up checks; they are not a pattern established for all LoopX runs.
+First-submission timing and quality need separate checks. On Borden, native Codex first submitted at about 13.6 minutes and LoopX at about 48.2. On Grid, native first submitted at about minute 21 and LoopX at about minute 12, the latter with a 3593 placeholder. A submission timestamp is not the feedback-return time, and earlier submission alone does not establish a better candidate.
The curve flattens on a local solution. On Grid, LoopX reached 2075 at 85 minutes—better than native’s 2182 at the time—then spent 9.5 hours and nine submissions going from 2075 to 2039, under 2%. Native went from 2182 to 1791 in the same window and changed method three times: an embedded C++ accelerator, a dynamic-programming repair step, a sweep over cost parameters. On Borden, from its third submission LoopX changed little but one amplitude constant, gaining about two points per hour.
Submissions sometimes regress. In the first Borden pair, 3 of LoopX’s 11 valid submissions regressed; in the second pair it fell from 68.1 to 41.7, and the evaluator report shows why: the public error improved while the hidden-well error rose from 0.09 to 0.74—it was fitting the public data. Native regressed too (54.7 to 50.3), so this is not specific to LoopX, but every regression costs a cooldown slot.
-Native’s 109 on Borden needs its own note. The evaluator report returns aggregate metrics for the hidden set, and both the native arm and the fixed LoopX build in local short runs reached 109 by fitting those metrics. That is a protocol leak, not modelling skill; we record it as a task issue and count it as capability evidence for neither side.
-Local supplement: earlier submission did not yield a win across tasks
+Native’s 109 on Borden needs its own note. The evaluator report returns aggregate metrics for the hidden set, and both the native arm and the historical combined LoopX build in local short runs reached 109 by fitting those metrics. That is a protocol leak, not modelling skill; we record it as a task issue and count it as capability evidence for neither side.
+Historical local supplement: timing and scores show no consistent advantage
The local report covers four tasks, 13 scored runs and 71 native submissions. They shared a four-CPU, 16-GiB host; resource ceilings were not exclusive reservations. The chart below starts each arm at its first native-session record, places each submission at submitted_at, and plots the best score through that submission. Lines only connect observations and end at the last submission; there is no fitted function, confidence band or invented zero.
| Task / comparison | Combined LoopX best | Native official best | Native-goal best |
|---|
| Task / comparison | Historical combined LoopX best | Native official best | Native-goal best |
|---|---|---|---|
| Borden | 109.000‡ | 69.579 | Not run |
| Integer compression codec | 24.714 | 28.620 | 12.836 |
| Rust multicrate reconstruction | 26.506 | 26.597 | 26.597* |
| covshift | Pilot (default) | With acceptance gate (one run) | Five-arm study heartbeat |
|---|
| covshift (historical individual runs; not a paired effect estimate) | Pilot (default) | With acceptance gate (one run) | Five-arm study heartbeat |
|---|---|---|---|
| Hidden-set reward | 0.000 | 0.82 | 0.743 |
| Self-stop | minute 55 | minute 38, but only after all three criteria passed | — |
| Cost | $15.8 | $15.3 | — |
| Gap | Owner | Shown | Not yet shown |
|---|---|---|---|
| Late first feedback | Harness (experimental planning constraint, now withdrawn) | Traces show Todo order delaying submission; local combined first submissions about 1.7–3.0 min, official 2.8–9.7 min | An isolated causal gain or preserved first-result quality; #6612 removes this option rather than accepting timing alone as success |
| Gap | Owner | Recorded or locally validated | Not yet validated |
|---|---|---|---|
| Later first Borden submission | Harness (experimental planning constraint, now withdrawn) | Traces show Todo order delaying submission; local combined first submissions about 1.7–3.0 min, official 2.8–9.7 min | An isolated causal gain or preserved first-result quality; #6612 removes this option rather than accepting timing alone as success |
| Review-point wake crashes | Product (checkpoint selector, quota CLI) | Root cause reproduced field for field; 7 new tests fail before / pass after; default path unchanged | Readback in a real Docker trial; a stable per-trial transport |
| Ten hours of in-place tuning | Product (review rules, progress review, planning contract) | Mechanism located: review may retain, progress is self-declared, assist fails silently | All repair directions unimplemented; even implemented, a different method is not guaranteed |
| Premature goal closure | Harness calling the product acceptance contract (off by default) | covshift 0.000 → 0.82 once; closure evidence now verifier-shaped; reconfirm counterexample closed | Gain after forced continuation; LoopX’s own closure judgement still fires at minute 38 |
| Checkpoints force the envelope on | Product (heartbeat builder) | Coupling located; both 24-hour runs are therefore not evidence of default behaviour | Decoupling not yet proposed; envelope on/off effect never validated |
| Premature goal closure | Harness calling the product acceptance contract (off by default) | Separate covshift runs scored 0.000 and 0.82; branch tests check the acceptance interface and reconfirm counterexample, not a paired gain | Gain after forced continuation; LoopX’s own closure judgement still fires at minute 38 |
| Checkpoints force the envelope on | Product (heartbeat builder) | Coupling located; the interrupted original Borden / Grid runs used non-default configurations | Decoupling not yet proposed; envelope on/off effect never validated |
What these repairs have in common: each is “the system is working, but the direction of the work was never checked”. The agent did not stop; nobody asked why it was still doing the same thing. The wake was not killed; a legitimate status word was not recognised. The goal was not forgotten; the evidence used to decide “done” had nothing to do with what the evaluator wanted. The value of a long-horizon control plane is not keeping an agent awake for more hours; it is leaving inspectable traces of these “is the direction right” judgements. This round repaired the parts that can be verified item by item; the parts that need the model to choose better are still design.
+The engineering value is in retaining checkable failure chains, exit criteria and acceptance counterexamples. Corresponding entrypoint reproductions and regression checks can qualify local repairs; a versioned, configuration-explicit rerun retaining failures is still needed to assess long-horizon outcomes. These fragments establish neither that the main cause lies in the product rather than the model nor that LoopX is generally better or worse than native execution.
-9. Next: repair the product first, then rerun
-At the historical run cutoff, the remote runs were paused and their evidence retained. Since then #6584 and #6601 reached main, while #6612 withdrew feedback-first. The sequence below is the research proposal, not execution authority: new runs need their own authorization, and #6585 remains a branch experiment pending an accepted owner placement.
+9. The author’s proposed follow-up validation
+At the historical run cutoff, the remote runs were paused and their evidence retained. Since then #6584 and #6601 reached main, while #6612 withdrew feedback-first. The sequence below is the research proposal, not execution authority: new runs need their own authorization, and #6585 was closed without merging. Its retained branch evidence does not establish a supported mainline strategy.
| Step | Content | Decisive evidence |
|---|---|---|
| Product repairs reach main | #6601 wake crashes; handoff/envelope decoupling; plateau signal and visible failure for progress review; planning contract rejecting open-ended Todos | Real CLI and TypeScript owner tests; the default-off path byte-identical |
| Short pairs first | One 3-hour pair each for Borden and Grid, fixed build vs native, checkpoints off | No crash at review points; review records show a pivot or an evidenced retain |
| Envelope validation | Same task, one pair each with envelope on and off, checkpoints off | Uncached tokens, packet reads, CLI calls, time share, score |
| 24-hour reruns | One pair each for Borden and Grid, envelope state declared explicitly | Common-window comparison with the curves above; failures and interruptions kept |
| Acceptance experiment retained | #6585 branch evidence; mainline placement remains on hold | Accepted owner placement and separately authorized execution before any new run |
| Acceptance experiment retained | Historical #6585 branch evidence; PR closed without merging | A separately reviewed implementation and new execution authorization before any rerun |
The previous article ended by saying progress had moved from “keep the agent running longer” to “make more of the runtime produce valid work”. This round takes that one step further: valid work also needs to be asked about its direction. Native Codex changed methods three times in the same window because it saw the score stall and changed; LoopX saw the same score, and every one of its rules allowed it to continue. Fixing that is the actual goal of this round.
+This article leaves engineering leads from historical configurations, not formal LoopX capability conclusions. The author proposes checking local repairs and then task quality and efficiency under declared source, model, task, feedback, budget and resource conditions. Earlier feedback does not establish better delivery; a locally verified repair does not establish long-horizon gains. A withdrawn strategy is not current activation advice.