diff --git a/REFERENCE.md b/REFERENCE.md index 2e89b64..c29b7ba 100644 --- a/REFERENCE.md +++ b/REFERENCE.md @@ -82,6 +82,7 @@ sections are fixed: `## Question`, `## Witness`, `## Where to look`, `## Runs`, | `Review-minutes:` | integer | a seed, not a measurement | | `Epic:` | slug | shared with the Mind — the join key across the two dashboards | | `Filed:` | `YYYY-MM-DD` | | +| `Migrated-from:` | source path or ledger anchor | the Mind prompt, review file or project-ledger entry this phase was transcribed from (phase 4 of the birth epic) | Gate refs are matched by `GATE_REF_RE`, copied verbatim from `PyAutoMind/scripts/lifecycle.py`: @@ -313,6 +314,7 @@ The human's words, verbatim. | `Reviewed-at:` | timestamp | | | `Review-minutes-actual:` | integer | | | `Follow-ups:` | comma-separated GitHub refs | `GATE_REF_RE`; **the issue is created before `rule` runs** | +| `Migrated-from:` | source path or ledger anchor | the review file or project-ledger entry a backfilled ruling was transcribed from | Body: `## Ruling` — the human's words verbatim; `## Evidence` — pointers. @@ -375,6 +377,9 @@ names a member of it. - reviewed-at: - review-minutes-actual: +### Follow-ups accepted +- + ## — - decision: accept|rerun|drop|leave-to-finish|(none) - ruled: yes|no @@ -382,6 +387,9 @@ names a member of it. ``` +`Follow-ups accepted` is optional and is `###`, not `##`, because `check` +requires every `##` to name a member of the slot's batch record. + `` ∈ `HEALTHY | SUSPECT | FAILED | RUNNING`. `(none)` with `ruled: no` is a member the human left untouched — listed, never silently dropped. `ruled: yes` requires a verb; a verb with `ruled: no` is a leaning the diff --git a/batches/2026-08-31-am.md b/batches/2026-08-31-am.md new file mode 100644 index 0000000..2fe5f29 --- /dev/null +++ b/batches/2026-08-31-am.md @@ -0,0 +1,54 @@ +# Batch 2026-08-31 am +- dispatched: 2026-08-29T23:44Z (retroactive — the members were hand-dispatched before the batch framework shipped) +- review-at: 2026-08-31T13:00Z (stated at review, not at dispatch) +- shift: overnight 29th→31st, pre-framework +- lane: local-dev +- review-minutes-planned: 121 (packet estimate; no dispatch-time plan existed) +- migrated-from: PyAutoMind/batches/2026-08-31-am.md +- members: + - failed_submissions_342008_10: phases/inference_programme/failed_submissions_342008_10.md — 342008, 342009, 342010 — 1 — ready + - smc_probe_342018: phases/inference_programme/smc_probe_342018.md — 342018 — 25 — ready + - knn_rerun_342016_7: phases/inference_programme/knn_rerun_342016_7.md — 342016 — 15 — ready + - refs_5_6_342016_56: phases/inference_programme/refs_5_6_342016_56.md — 342016 — 15 — ready + - slogdet_ab_342017: phases/inference_programme/slogdet_ab_342017.md — 342017 — 12 — ready + - delaunay_adapt_split_pl_sersic_0: phases/subhalo_validation/delaunay_adapt_split_pl_sersic_0.md — 342027 — 10 — awaiting-ruling + - delaunay_adapt_split_pl_eff_0: phases/subhalo_validation/delaunay_adapt_split_pl_eff_0.md — 342027 — 8 — running + - delaunay_adapt_split_pl_eff_1_outer: phases/subhalo_validation/delaunay_adapt_split_pl_eff_1_outer.md — 342027 — 8 — running +- collected: 2026-08-31T14:46Z +- reviewed-at: 2026-08-31T13:34 (local) +- delivered: 7/7 runs produced results (2 of them failed on their own evidence: the 342008-10 submissions and the SMC probe) +- packet: batches/packets/2026-08-31-am.html +- review: batches/reviews/2026-08-31-am.md +- review-minutes-actual: excluded — workshop session (the framework was built and polished mid-review); the first calibration point is the next batch +- notes: | + Transcribed into the Cortex grammar on 2026-09-01 (birth epic phase 4). The Mind + original stays intact at PyAutoMind/batches/2026-08-31-am.md with a `- moved-to:` + line — a verbatim history has to live somewhere live. + + Two shape differences from the Mind record, both forced by the Cortex grammar: + the Mind's single member `subhalo-pl_eff-342027_12` covered two lenses on one + array, and decision 56 gives each lens its own phase, so it appears here as two + members (`delaunay_adapt_split_pl_eff_0`, `delaunay_adapt_split_pl_eff_1_outer`) + carrying the Mind member's 8 review-minutes each. And the Mind's eighth member + `euclid-phase2-pr46` was a **dev** retrospective (PR #46, `structure-ok`); the dev + surface has its own vocabulary and its own record, so it is named here and not + transcribed as a science member. + + Retroactive record: every member was hand-dispatched on 2026-08-29 night, before + the batch framework shipped, and the Mind file was written at collection. The + review session doubled as the framework workshop (review surface v2, laptop mirror, + flexible slot, packet archive all built mid-review), which is why the minutes + figure is excluded from calibration. + + The four profiling members were REJECTED wholesale, with no ledger writes: the + human rewound the JAX profiling epic to Phase 1 the same day (autolens_profiling#200). + Do not certify refs 5/6/7, do not write the slogdet verdict, do not ingest SMC + rates — the rewound epic takes precedence over this packet's proposed follow-ups. + In the Cortex those four rejections are R-20260831-02..05, filed as `drop` + (`reject` is the dev surface's word). + + Packet defect found in use and fixed same-day: the submit modal could not close on + Pages (the fragment relied on the artifact viewer's [hidden] reset); packets are + standalone documents now. `batches/packets/2026-08-31-am.html` here is the archive + of record from now on; the Mind's copy stays at its Pages URL so the localStorage + the human's browser holds against it keeps working. diff --git a/batches/2026-08-31-pm.md b/batches/2026-08-31-pm.md new file mode 100644 index 0000000..d7d89fb --- /dev/null +++ b/batches/2026-08-31-pm.md @@ -0,0 +1,59 @@ +# Batch 2026-08-31 pm +- dispatched: 2026-08-31T18:40Z +- review-at: 2026-08-31T21:00Z ("5pm, 2.5 hours from now" — stated by the human at dispatch) +- shift: afternoon (first timed batch under the review workflow) +- lane: local-dev +- review-minutes-planned: 74 +- migrated-from: PyAutoMind/batches/2026-08-31-pm.md +- members: + - inference_refs_v1_redo: phases/inference_programme/inference_refs_v1_redo.md — 342091 — 20 — awaiting-ruling + - mge_pos_ref_reuse: phases/inference_programme/mge_pos_ref_reuse.md — 340210 — 5 — awaiting-ruling + - mge_fp64_retro_baseline: phases/inference_programme/mge_fp64_retro_baseline.md — 339070 — 4 — awaiting-ruling + - delaunay_fp64_retro_baseline: phases/inference_programme/delaunay_fp64_retro_baseline.md — 339071 — 4 — ready + - delaunay_adapt_split_pl_sersic_0: phases/subhalo_validation/delaunay_adapt_split_pl_sersic_0.md — 342027 — 10 — accepted + - delaunay_adapt_split_pl_eff_0: phases/subhalo_validation/delaunay_adapt_split_pl_eff_0.md — 342027 — 8 — awaiting-ruling + - delaunay_adapt_split_pl_eff_1_outer: phases/subhalo_validation/delaunay_adapt_split_pl_eff_1_outer.md — 342027, 342093 — 8 — awaiting-ruling + - rectangular_adapt_pl_sersic_0: phases/subhalo_validation/rectangular_adapt_pl_sersic_0.md — 342094, 342095 — 15 — awaiting-ruling +- refreshed: 2026-08-31T22:53Z — inference_refs_v1_redo pulled (first: 2 of 10 tasks ended clean, 1 running, 7 queued behind the single A100) +- refreshed: 2026-08-31T22:57Z — the subhalo members pulled (second: 342093 landed; 342094 landed, 342095 released and running; both pl_eff chains still inside subhalo[2] with ~3 h to wall) +- refreshed: 2026-09-01T13:01Z — both laptop legs pulled (third: the 342091 wave complete, 10/10 delivered clean; both pl_eff witnesses landed inside the 2-day wall) +- refreshed: 2026-09-01T17:54Z — rectangular_adapt_pl_sersic_0 witness landed (evidence_increase +0.2538, the project's first grid-derived witness) +- collected: 2026-08-31T20:57Z +- reviewed-at: 2026-09-01T15:13 (transcribed) +- delivered: 8/8 science members produced results; 3 ruled this slot, 5 carried +- packet: batches/packets/2026-08-31-pm.html +- review: batches/reviews/2026-08-31-pm.md +- review-minutes-actual: (not given) +- notes: | + Transcribed into the Cortex grammar on 2026-09-01 (birth epic phase 4). The Mind + original stays intact at PyAutoMind/batches/2026-08-31-pm.md with a `- moved-to:` + line; this record holds only the **science** half of that slot. + + The Mind slot was mixed: nine cloud dev members (autofit-resampling-info, + autofit-prodigy-49, autofit-multistart-iterations, autonerves-colab-silence, + release-smoke-env, organ-board-github-link, memory-queue-filing-gate, + numba-vs-jax-sparse, euclid-3a-prose) and two laptop science members. The dev + members have their own vocabulary (merge | tweak | reject | defer) and their own + record; they are named here and not transcribed. `review-minutes-planned:` above + is the science half of the Mind's 117 + 13. + + The two laptop members expand into the eight phases above: the Mind's + `jax-inference-phase1-refs` carried the 342091 wave plus three legacy-reuse ruling + members added to the packet by that leg (mge-pos-ref-reuse, mge-fp64-retro-baseline, + delaunay-fp64-retro-baseline), and the Mind's `subhalo-follow-up-wave` covered four + lens/pipeline runs that decision 56 gives a phase each. + + Both laptop members were read-and-record only: no RAL job was submitted, cancelled + or modified, no pipeline or library code was edited, and no ruling was written in + the session — the rulings are this file's review, filed 2026-09-01T15:13Z. + + CARRY-FORWARD, adopted at the human's direction during this slot: long-running + members do not hold a review. The slot reviews everything ready; in-flight members + are recorded carried and enter the next batch's packet automatically. Five members + are carried out of this slot — the refs wave (its mesh rows are now spent evidence + under R-20260901-03 and must be re-checked against the positions-penalty rule before + certification) and the four subhalo phases. + + The parked second `rectangular_adapt` submission stays PARKED: its gate (a clean + pl_eff witness) opened when both pl_eff witnesses landed, but which lens goes next + is a human decision that follows the rulings on the two pl_eff phases. diff --git a/batches/packets/2026-08-31-am.html b/batches/packets/2026-08-31-am.html new file mode 100644 index 0000000..0107ff6 --- /dev/null +++ b/batches/packets/2026-08-31-am.html @@ -0,0 +1,2080 @@ + + + + + +Slot Review 31 Aug 2026 + + + + + +
+ +
+

31 Aug 2026 · local-dev lane · RAL harvest + one retrospective

+

Slot Review

+

This is the review packet for the runs that came back — eight members, each with the question it was submitted to answer, its pre-registered witness, the health evidence, the numbers, and the one decision waiting on you; nothing was edited, committed, submitted or cancelled in building it. Choose a decision and leave a note on each member, then Submit review at the bottom — the review lands in PyAutoMind/batches/reviews/ for the orchestrator chat to enact.

+

Run outputs and logs are mirrored at /mnt/c/Users/Jammy/Science/inference_programme/ (pulled 2026-08-31 11:42 EDT via the autolens_profiling repo’s hpc/sync pull).

+

Generated 2026-08-31 10:46 · refreshed 2026-08-31 11:42

+

Permanent home: pyautolabs.github.io/PyAutoMind/packets/2026-08-31-am.html

+
+ +
+
8Members
+
2Failed
+
1Suspect
+
3Healthy
+
1Running
+
1Merged
+
+ 121 + Est. review-minutes + 1 + 25 + 15 + 15 + 12 + 10 + 8 + 35 + against a 45-minute slot +
+
+ + +
+
+ + +
+
+ + +
+

The minutes figure is the only calibration the review-cost estimate ever gets.

+
+ +
+

Most important finding

+

The Phase 7 SMC probe (342018) did not fail loudly — all three arms report converged: true and valid: true while sitting 106,000–163,600 nats from the Nautilus bar on the same cell; the pre-registered validity rule (λ→1.0 + final ESS ≥ 10% of particles) passes a frozen particle cloud, and the warm arms have regressed from the parked prototype's ±0.8-nat bridge (job 331058) to −74k/−76k.

+
+ +
+

Rulings needed

+
    +
  1. SMC probe — record Phase 7 as a failed probe and rewrite the validity rule, or re-run first?
    2. +
  2. refs 5 & 6 — certify against provenance despite Δmax logL of −773 / −116 nats vs a 2-nat registry tolerance?
    3. +
  3. knn task 7 — the 480-nat deficit did not close; withdraw permanently, re-run the Phase 8B Prodigy arm on the corrected target, or accept?
    4. +
  4. slogdet A/B — call the DECISIONS verdict “no difference” (Δlog Z 0.50 nats) given the arms ran on two different PyAutoFit trees?
    5. +
  5. subhalo_validation — accept pl_sersic_0's witness (−0.30, PASS) and rule on the 2×2 subhalo grid.
    6. +
  6. euclid-dr1-prep phase 2 — structure sound → phase 3 dispatchable as-is; bundle the three tidy-ups into phase 3's PR or file them separately; make euclid CI release-blocking before phase 4?
    7. +
+
+ +
+ + + +
+ +
+
+
+

autolens_profiling · inference programme

+

342008 / 342009 / 342010 — the 1-second FAILED submissions

+

342008 · 342009 · 342010

+
+
+ Failed + +
+
+
+ +
+

Question this run was submitted to answer

+

The same three questions as 342016/17/18: they are the identical submissions (sacct SubmitLine: sbatch --array=5,6,7 --requeue hpc/batch_gpu/submit_search_nautilus_inference_refs_v1_array.sh, then the slogdet A/B submit, then the SMC probe), resubmitted 78 s later as 342016/17/18.

+
+ +
+

Witness

+

n/a — submission-time failure, no science content.

+
+ +
+

Health evidence

+

sacct FAILED at 00:00:01, batch step CANCELLED, ExitCode 0:53, Reason: None. There is no .err to read: that is the failure. WorkDir=/mnt/ral/jnightin/autolens_profiling for 342008-10 vs .../hpc/batch_gpu for 342016-18, and /mnt/ral/jnightin/autolens_profiling/error does not exist — the #SBATCH -e error/error.%A_%a.err path is relative to the submit CWD, so SLURM could not open the error file and killed the batch step before Python ever started. Resubmitting from hpc/batch_gpu/ (where output/ and error/ both exist) fixed it.

+
+ +
+

Readout

+

No numbers; nothing ran. This is the known sbatch-CWD trap, third occurrence.

+
+ +
+

Ruling

+

Nothing scientific to rule on; optionally, make the submit scripts self-locating — #SBATCH -o/-e cannot be computed, so a cd $(dirname $0) guard or an absolute path is the only fix.

+
+ +
+

Your review

+
+ + + + +
+ +
+ +
+

Follow-ups

+
    +
  • [autolens_profiling]Convert every hpc/batch_gpu/submit_* -o/-e to an absolute /mnt/ral/jnightin/autolens_profiling/hpc/batch_gpu/{output,error}/… path so submit CWD stops mattering.
    • +
  • [autolens_profiling]Add a one-line “submit from hpc/batch_gpu/” banner to hpc/README / the WALL-BASIS header block.
    • +
+
+ +
+

Where to look yourself

+
+
Project root
+
/home/jammy/Code/PyAutoLabs/autolens_profiling
+
Submit scripts (the -o/-e lines)
+
/home/jammy/Code/PyAutoLabs/autolens_profiling/hpc/batch_gpu/submit_search_nautilus_inference_refs_v1_array.sh
+
The missing directory
+
RAL only/mnt/ral/jnightin/autolens_profiling/error (absent) vs /mnt/ral/jnightin/autolens_profiling/hpc/batch_gpu/error (present)
+
Open in order
+
  1. The #SBATCH -o/-e block of the submit script.
  2. ssh euclid_jump 'ls -d /mnt/ral/jnightin/autolens_profiling/error' to see it is not there.
+
+
+ + Est. review-minutes 1 +
+
+ +
+
+
+

autolens_profiling · Phase 7

+

Phase 7 SMC probe — mala_warm / hmc_warm / mala_cold

+

342018_[0,1,2]

+
+
+ Failed + +
+
+
+ +
+

Question this run was submitted to answer

+

“FIRST PROBE of af.SMC in this framework… it measures a rate, an acceptance/ESS schedule and — from the cold arm — the evidence bridge against Nautilus's 31690.42 nats on imaging/mge/hst, which is the first independent (gradient-side) check this programme has ever had on a log_evidence” (PROGRAMME.md §“Phase 7 — SMC and other high-dimensional candidates”, STATUS 2026-08-29).

+
+ +
+

Witness

+

Pre-registered in PROGRAMME.md's risk table: “SMC particle impoverishment / silent λ=1 garbage — judge by acceptance traces + ESS schedule, never 'Converged: yes'; independent-run logZ variance.” The submit script's own coded rule: valid: false on a partial tempering path or a final ESS below 10 % of the particle count. Bar to beat: Nautilus imaging/mge/hst logZ 31,690.42, maxL 31,786.80 (targets/REFS_V1_HARVEST.md); truth logL 31,521.74.

+
+ +
+

Health evidence

+

sacct COMPLETED for all three; walls 10:34 / 12:19 / 31:23 against a 2 h budget (9–26 %); .err files contain only the benign mask-padding UserWarning; 0 tracebacks in all three .out; 0 resume markers; version stamp 2026.8.17.1 in all three JSONs. Finite-log_l scan: no sample above 1e10 — but the log-likelihoods are catastrophically low, not high. No checkpoint.hdf5 (SMC does not write one; the tempering schedule in the JSON is the equivalent). PyAutoFit version stamp MISMATCH — see the cross-cutting note.

+
+ +
+

Readout

+
+ + + + + + + + + + + +
Three SMC arms against the Nautilus bar
armtasklogZmaxLn_smc_stepsλ path (last two)acceptance finalESS finalwallderived evalsms/evalconvergedvalid
mala_warm342018_0−74,306.46−74,278.1394.6e-4 → 1.03.3e-35212/256623.5 s11,77652.9truetrue
hmc_warm342018_1−75,721.88−75,695.2483.9e-4 → 1.06.1e-29210/256727.3 s82,1768.85truetrue
mala_cold342018_2−131,885.99−131,858.45164.6e-3 → 1.01.7e-179225/2561,867.3 s20,73690.1truetrue
Nautilus bar341879-era ref+31,690.42+31,786.80————1,015 s————
+
+

Three independent readings all say the same thing:

+
    +
  1. Acceptance collapses to exactly 0.0 by step 5–6 (MALA warm), step 6 (HMC warm) and step 12 (cold), and stays there. The cloud stops moving. max_log_likelihood_list is frozen from step 2 onward in the HMC arm (−75,695.24 repeated six times).
    2. +
  2. The whole evidence is one temperature jump. log_likelihood_increment_list is ≈ −3 nats per step until the last, which is −74,215 / −75,639 / −131,249. λ goes 4.6e-4 → 1.0 in a single step. That is the “silent λ=1 garbage” the PROGRAMME names.
    3. +
  3. The recorded ESS is meaningless here — 210–225 of 256 is high because a frozen, resampled cloud has near-uniform weights. The coded valid rule tests exactly this quantity and therefore passes all three.
    4. +
+
    +
  • The warm start did not land on the MAP. Both warm arms were to be centred on the completed Prodigy MGE MAP fit, yet max_log_likelihood_list[0] is −92,559 (mala_warm) / −89,777 (hmc_warm) — ~124,000 nats below that MAP. The submit header calls a warm arm that was actually cold “the one failure this must not permit”.
    • +
  • Label mismatch: the submit exports SEARCHES_SMC_MASS=prior_scaled and its header promises the artifact will record that so the row “cannot be mistaken for a covariance-informed one”; the JSON records whitening_kind: "diagonal covariance", inverse_mass_matrix_kind: "array", warm_start_source: InitializerParamStartPoints.
    • +
  • Regression vs the parked prototype: PROGRAMME.md line 29 records job 331058 — “bridge validated: warm logZ within ±0.8 nats of the Nautilus bar; cold arm −179k with 'Converged: yes'”. The cold arm here reproduces the known cold pathology (−131.9k); the warm arms have gone from ±0.8 nats to ≈ −106,000 nats.
    • +
  • Do not put these in rates.py. The derived eval counts are kernel-inconsistent (HMC 82,176 evals in 727 s = 8.85 ms/eval; MALA warm 11,776 in 623 s = 52.9 ms/eval — a 6× spread on the same likelihood), and a frozen cloud's cost is not a working run's cost.
    • +
  • Raw result JSONs, local after the pull: packet/smc_probe/hpc_hpc_a100_fp64_n256_{mala_warm,hmc_warm,mala_cold}.json. They are not in the local repo — the local autolens_profiling/results/searches/smc/ tree does not exist yet.
    • +
+
+
+ Bar chart, truth vs best-fit, MALA warm arm +
smc_probe/hpc_hpc_a100_fp64_n256_mala_warm.png — truth vs best-fit
+
+
+ Bar chart, truth vs best-fit, HMC warm arm +
smc_probe/hpc_hpc_a100_fp64_n256_hmc_warm.png — truth vs best-fit
+
+
+ Bar chart, truth vs best-fit, MALA cold arm +
smc_probe/hpc_hpc_a100_fp64_n256_mala_cold.png — truth vs best-fit
+
+
+
+ +
+

Ruling

+

Record Phase 7 as a failed probe (Gate D not callable, no rate ingested) and fix the validity rule + warm-start path before any re-run — or treat this as a run-configuration miss and re-run once first.

+
+ +
+

Your review

+
+ + + + +
+ +
+ +
+

Follow-ups

+
    +
  • [autolens_profiling]Add an acceptance floor and a final-λ-jump ceiling to the SMC valid rule — a row whose last temperature carries >90 % of the log-evidence, or whose acceptance is 0.0 for ≥2 consecutive steps, is valid: false.
    • +
  • [PyAutoFit]Investigate why an SMC warm start from a completed MAP fit begins 124,000 nats below that MAP (InitializerParamStartPoints + prior_scaled mass path).
    • +
  • [autolens_profiling]Reconcile whitening_kind with SEARCHES_SMC_MASS — the artifact must say prior_scaled when that is what was asked for.
    • +
  • [autolens_profiling]Do not add an SMC row to scripts/misc/wall/rates.py; keep the cell at source: unmeasured probe-first: yes.
    • +
  • [autolens_profiling]Re-derive likelihood_evals_derived per kernel so MALA and HMC rates are comparable.
    • +
+
+ +
+

Where to look yourself

+
+
Project root
+
/home/jammy/Code/PyAutoLabs/autolens_profiling
+
Pulled artefacts (local)
+
…/scratchpad/packet/smc_probe/ (3 JSONs + 3 PNGs)
+
Output folders
+
/mnt/c/Users/Jammy/Science/inference_programme/output/searches/smc/imaging/mge/hst/n256_mala_m5_seed0_warmebe2e28d_massprior_scaled/hpc_a100_fp64_n256_mala_warm/
+
…/n256_hmc_m5_i8_seed0_warmebe2e28d_massprior_scaled/hpc_a100_fp64_n256_hmc_warm/
+
…/n256_mala_m5_seed0_cold/hpc_a100_fp64_n256_mala_cold/ — each holds model.results, image/, files/
+
Batch logs
+
/mnt/c/Users/Jammy/Science/inference_programme/logs/output/output.342018_{0,1,2}.out
+
Scripts
+
submit /home/jammy/Code/PyAutoLabs/autolens_profiling/hpc/batch_gpu/submit_search_smc_imaging_mge_a100_hst_fp64_probe
+
entry point /home/jammy/Code/PyAutoLabs/autolens_profiling/scripts/imaging/searches/smc/mge.py
+
Ruling lands in
+
/home/jammy/Code/PyAutoLabs/autolens_profiling/results/notes/inference/DECISIONS.md (Phase 7 / Gate D)
+
the Phase 7 row of /home/jammy/Code/PyAutoLabs/autolens_profiling/results/notes/inference/PROGRAMME.md
+
Open in order
+
    +
  1. packet/smc_probe/hpc_hpc_a100_fp64_n256_mala_cold.json → the diagnostics.acceptance_rate_list / lambda_list / log_likelihood_increment_list triple.
    2. +
  2. PROGRAMME.md line 928 (the pre-registered “never 'Converged: yes'” rule) and line 29 (job 331058's ±0.8-nat warm bridge).
    3. +
  3. Compare its log_evidence against the imaging/mge/hst row in results/notes/inference/targets/REFS_V1_HARVEST.md.
    4. +
  4. The submit script's “READ THE SCHEDULE, NOT THE FLAG” header block.
    5. +
+
+
+ + Est. review-minutes 25 +
+
+ +
+
+
+

autolens_profiling · InferenceRefs_v1

+

Task 7 — knn re-run on the corrected target

+

342016_7

+
+
+ Suspect + +
+
+
+ +
+

Question this run was submitted to answer

+

“task 7 knn, positions off — 341879_7 LANDED and is CERTIFIED, but its maxL (30,077.028) sits 480 nats below a same-target_id Phase 8B Prodigy arm (30,557.03). That is the same overflow signature at lower amplitude” — re-run on the corrected AdaptSplitPower / capped-LogUniform(1e-6,1e4) target to replace the withdrawn row (submit-script header, RESUBMIT 2026-08-29; REFS_V1_HARVEST.md “FLAG — do not score against the knn reference”).

+
+ +
+

Witness

+

Certification is the three human provenance checks in REFS_V1_HARVEST.md (“Rows certified”): version stamp 2026.8.17.1, wall consistent with the cell's recorded cost, no resume marker. The registry tolerance (targets/TOLERANCES.md, knn_fp64) is Δmax logL ≤ 2 nats vs the truth bar, mean shift ≤ 0.2σ, σ-ratio in [0.8, 1.25]. The scientific witness is the 480-nat deficit: does the corrected target close it?

+
+ +
+

Health evidence

+

sacct COMPLETED, wall 58:37 of a 6 h budget; .err is one benign mask-padding UserWarning; 0 tracebacks; 0 resume markers; version stamp 2026.8.17.1 ✓; ran cold in a new identifier dir 39b5ccc228ad4b4389b1129a7e5cf605 (the predicted new hash — the old 36d5967e91391122bff82e879c619548 sits beside it untouched), so the resume-collision check in the submit header held. No checkpoint.hdf5 (search_internal removed on completion). Finite-log_l scan over samples.csv: n=34,544, max 30,062.67, min 30,021.18, 0 samples > 1e10.

+
+ +
+

Readout

+
+ + + + + + + + + + + + + + +
Withdrawn row vs the new row
withdrawn row (341879_7, legacy target)new row (342016_7, corrected target)Δ
target_idsha256:84c0d88d3032sha256:ccafb8b191bcnew target class
logZ30,010.17029,995.887−14.28
maxL30,077.02830,062.672−14.36
Δmax logL vs truth (31,521.74)−1,444.7−1,459.07−14.4 (tolerance is 2)
total wall3,343.6 s3,507.2 s+163.6
Kish ESS4,848.85,334.7+485.9
evals / ms-per-eval—66,600 / 51.80—
+
+

The corrected target did not close the 480-nat deficit — the new row is 14.4 nats lower still, putting it ~494 nats below the Phase 8B Prodigy log_reg arm's 30,557.03. Caveat that matters: the Prodigy arm was measured on the legacy sha256:84c0d88d3032 target, so it is no longer a same-target_id comparison — the deficit is now unscorable rather than confirmed. The run replaces one broken comparison with another.

+

Watch out: /home/jammy/Code/PyAutoLabs/autolens_profiling/results/searches/nautilus/imaging/knn/hst/hpc_hpc_a100_fp64_ref.json is still the old withdrawn row (30,010.17); the new one is uncommitted on RAL (git status shows it as M). Raw JSON local: packet/refs_v1/knn_hpc_a100_fp64_ref.json.

+
+
+ Bar chart, truth vs best-fit, knn reference re-run +
refs_v1/knn_hpc_a100_fp64_ref.png — truth vs best-fit, 342016_7
+
+
+
+ +
+

Ruling

+

Certify the new knn row as the replacement bar / keep it withdrawn pending a Prodigy re-run on the corrected target / drop the knn cell from InferenceRefs_v1.

+
+ +
+

Your review

+
+ + + + +
+ +
+ +
+

Follow-ups

+
    +
  • [autolens_profiling]Re-run the Phase 8B Prodigy knn · log_reg arm on target sha256:ccafb8b191bc so the 494-nat deficit becomes scorable again.
    • +
  • [autolens_profiling]Decide whether a 1,459-nat Δmax-logL row can sit in a registry whose tolerance is 2 nats, or whether the truth bar for pixelized cells needs its own tolerance.
    • +
  • [autolens_profiling]Sync results/searches/…/knn/hst/hpc_hpc_a100_fp64_ref.json from RAL and update REFS_V1_HARVEST.md + results/baselines/InferenceRefs_v1/INDEX.json (which still lists only 2 of the 9 rows).
    • +
+
+ +
+

Where to look yourself

+
+
Project root
+
/home/jammy/Code/PyAutoLabs/autolens_profiling
+
Pulled artefacts (local)
+
…/scratchpad/packet/refs_v1/knn_hpc_a100_fp64_ref.{json,png}
+
Output folder
+
/mnt/c/Users/Jammy/Science/inference_programme/output/searches/nautilus/imaging/knn/hst/hpc_a100_fp64_ref/39b5ccc228ad4b4389b1129a7e5cf605/ (+ its .zip; the legacy 36d5967e… dir beside it is the withdrawn row)
+
Batch log
+
/mnt/c/Users/Jammy/Science/inference_programme/logs/output/output.342016_7.out
+
Scripts
+
submit …/hpc/batch_gpu/submit_search_nautilus_inference_refs_v1_array.sh (task 7)
+
entry point …/scripts/imaging/searches/nautilus/knn.py
+
target definition …/scripts/misc/searches/_setup.py
+
Ruling lands in
+
…/results/notes/inference/targets/REFS_V1_HARVEST.md (the “Rows certified” table + the FLAG section)
+
…/results/baselines/InferenceRefs_v1/INDEX.json
+
Open in order
+
    +
  1. The FLAG paragraph in REFS_V1_HARVEST.md.
    2. +
  2. packet/refs_v1/knn_hpc_a100_fp64_ref.json (results.max_log_likelihood = 30,062.67).
    3. +
  3. Compare against …/results/notes/inference/phase_08_regularization/RESULTS.md “8B — run history and harvest” (Prodigy 30,557.03) and note the target_id differs.
    4. +
  4. targets/TOLERANCES.md row knn_fp64.
    5. +
+
+
+ + Est. review-minutes 15 +
+
+ +
+
+
+

autolens_profiling · InferenceRefs_v1

+

Tasks 5 & 6 — slam_source_pix_nn pos-off / pos-on

+

342016_[5,6]

+
+
+ Healthy + +
+
+
+ +
+

Question this run was submitted to answer

+

Produce the two missing imaging/slam_source_pix_nn/hst reference rows on the corrected target. Task 5 replaces RAL 341908_5, which was ledgered as “0 Nautilus calls in 6 h” but had actually made 90,000 calls and died in a likelihood-overflow flood (finite log_l to 3e+303); task 6 was never submitted because task 5 was misread as “this cell is unaffordable” (submit-script header, RESUBMIT 2026-08-29; DECISIONS.md 2026-08-29).

+
+ +
+

Witness

+

The three provenance checks in REFS_V1_HARVEST.md; registry tolerance slam_source_pix_nn_fp64 / _pos_fp64 = Δmax logL ≤ 2 nats, mean shift ≤ 0.2σ, σ-ratio [0.8, 1.25] (targets/TOLERANCES.md). And the implicit witness of the whole overflow-flood wave: does the capped AdaptSplitPower target now terminate, with no finite garbage?

+
+ +
+

Health evidence

+

sacct COMPLETED both; walls 1:21:57 and 1:14:55 of a 6 h budget (23 % / 21 %) against a cell whose only prior run never terminated in 6 h; .err = one benign mask-padding warning (task 5 also two blurring_image warnings); 0 tracebacks, 0 resume markers in both .out; version stamp 2026.8.17.1 ✓; task 5 ran cold in the predicted-new identifier 7ced3989425354c2a53d4a79d7da2b11. checkpoint.hdf5 is not available — the run removes search_internal on completion and zips the rest, so n_like/explored cannot be read post-hoc; samples_info.json stands in (73,500–73,700 samples, acceptance ratio 1.0, no truncation). Finite-log_l scan over samples.csv: task 5 n=35,948, max 30,748.58, min 30,710.09; task 6 n=20,775, max 31,406.12, min 31,369.29 — 0 samples > 1e10 in either. The 3e+303 flood is gone. PyAutoFit version stamp MISMATCH — see the cross-cutting note.

+
+ +
+

Readout

+
+ + + + + + + + + +
Both reference rows
rowtaskconfigtarget_idwallsampler walllogZmaxLΔmax logL vs truthevalsms/evalKish ESSfree params
ref (positions off)342016_5hpc_a100_fp64_refsha256:ad291b57fc624,891.4 s4,833.3 s30,684.61530,748.575−773.1773,70065.587,215.114
ref_pos_tauto0.2_f1e8 (positions on, auto)342016_6hpc_a100_fp64_ref_pos_tauto0.2_f1e8sha256:8021b4b697ff4,483.2 s4,426.8 s31,324.89731,406.116−115.6373,50060.234,606.114
+
+

Context from the certified table: the sibling imaging/slam_source_pix/hst rows are logZ 30,718.02 / maxL 30,809.46 (pos off) and 31,305.06 / 31,411.38 (pos on). So slam_source_pix_nn sits 33 nats below its Rectangular sibling with positions off and 20 nats above it with positions on — comfortably inside the family, and both rows land in ~80 min on a cell that previously could not terminate at all. Both Δmax-logL values are far outside the 2-nat registry tolerance, as is the already-certified mge row (+265 nats).

+

Raw JSONs local: packet/refs_v1/hpc_hpc_a100_fp64_ref.json, packet/refs_v1/hpc_hpc_a100_fp64_ref_pos_tauto0.2_f1e8.json. Neither exists in the local repo (autolens_profiling/results/searches/nautilus/imaging/slam_source_pix_nn/ is absent locally); they are on RAL, uncommitted.

+
+
+ Bar chart, truth vs best-fit, task 5 positions off +
refs_v1/hpc_hpc_a100_fp64_ref.png — task 5, positions off
+
+
+ Bar chart, truth vs best-fit, task 6 positions on +
refs_v1/hpc_hpc_a100_fp64_ref_pos_tauto0.2_f1e8.png — task 6, positions on
+
+
+
+ +
+

Ruling

+

Certify rows 5 and 6 into REFS_V1_HARVEST.md + INDEX.json as-is / certify with a recorded Δmax-logL blemish like the mge row / re-run before certifying.

+
+ +
+

Your review

+
+ + + + +
+ +
+ +
+

Follow-ups

+
    +
  • [autolens_profiling]Sync both JSONs+PNGs from RAL into results/searches/nautilus/imaging/slam_source_pix_nn/hst/ and add the two rows to REFS_V1_HARVEST.md.
    • +
  • [autolens_profiling]Bring results/baselines/InferenceRefs_v1/INDEX.json up to the registry's real state (it still lists only the 2 retro-certified rows against 9–11 actual).
    • +
  • [autolens_profiling]Decide the registry-wide question the mge row already raised: a 2-nat Δmax-logL tolerance is violated by every pixelized reference row — retune it per cell family or drop it from the certification story.
    • +
  • [autolens_profiling]Stop deleting search_internal on reference rows, or archive checkpoint.hdf5 into the zip — n_like / explored are named provenance checks and are currently unrecoverable after the fact.
    • +
+
+ +
+

Where to look yourself

+
+
Project root
+
/home/jammy/Code/PyAutoLabs/autolens_profiling
+
Pulled artefacts (local)
+
…/scratchpad/packet/refs_v1/hpc_hpc_a100_fp64_ref.{json,png} and …_ref_pos_tauto0.2_f1e8.{json,png}
+
Output folders
+
/mnt/c/Users/Jammy/Science/inference_programme/output/searches/nautilus/imaging/slam_source_pix_nn/hst/hpc_a100_fp64_ref/7ced3989425354c2a53d4a79d7da2b11/
+
…/hst/pos_tauto0.2_f1e8/hpc_a100_fp64_ref_pos_tauto0.2_f1e8/35b83ae5f6eab98575809e7c5dc5f3af/ — both hold model.results, search.summary, image/, latent.results; search_internal already deleted
+
Batch logs
+
/mnt/c/Users/Jammy/Science/inference_programme/logs/output/output.342016_{5,6}.out
+
Scripts
+
submit …/hpc/batch_gpu/submit_search_nautilus_inference_refs_v1_array.sh (tasks 5, 6)
+
entry point …/scripts/imaging/searches/nautilus/slam_source_pix_nn.py
+
target/model …/scripts/misc/searches/_setup.py (_slam_source_pix_nn_model, _free_adapt_split)
+
Ruling lands in
+
…/results/notes/inference/targets/REFS_V1_HARVEST.md and …/results/baselines/InferenceRefs_v1/INDEX.json
+
Open in order
+
    +
  1. REFS_V1_HARVEST.md “Rows certified” + “What 'certified' does and does not mean”.
    2. +
  2. packet/refs_v1/hpc_hpc_a100_fp64_ref.json (results + performance blocks).
    3. +
  3. Compare its log_evidence 30,684.6 against the imaging/slam_source_pix/hst ref row (30,718.02) in that same table.
    4. +
  4. targets/TOLERANCES.md rows slam_source_pix_nn_fp64 / _pos_fp64.
    5. +
+
+
+ + Est. review-minutes 15 +
+
+ +
+
+
+

autolens_profiling · overflow-flood wave, rung 4 · issue #196

+

slogdet A/B on slam_source_pix_nn

+

342017_[0,1]

+
+
+ Healthy + +
+
+
+ +
+

Question this run was submitted to answer

+

“on the corrected target, does log_det_method='slogdet' change anything a nested sampler can see? … the honest answer here may well be 'no difference, because the cap already removed the failure' — which is a real finding, not a failed experiment. What it must NOT be is unmeasured while a slogdet default is being argued for elsewhere (W9, #166)” (submit-script header, rung 4 of the overflow-flood wave, autolens_profiling#196). Nothing had ever run a nested sampler with slogdet on a free-adapt-split cell.

+
+ +
+

Witness

+

A DECISIONS verdict on whether the two log-det tiers agree: logZ, wall, NaN counts. (The pre-registered CP-4 criteria from 2026-08-24 are about equality where the Cholesky succeeded; here the capped prior means the Cholesky never fails, so equality is the whole test.) Collision-safety witness from the header: both …_ld_cholesky and …_ld_slogdet output directories must exist — “if only one does, this submit measured nothing”.

+
+ +
+

Health evidence

+

sacct COMPLETED both; walls 1:23:58 and 1:14:37 of an 8 h budget (17 % / 16 %); .err files = mask-padding + two blurring_image UserWarnings each, nothing else; 0 tracebacks, 0 resume markers; version stamp 2026.8.17.1 ✓; collision safety confirmed — two distinct identifier dirs under two distinct name dirs (ld_cholesky/…/ae454f54e14847ca62929c70033c11a5, ld_slogdet/…/7eb80135de1db2112c5bb5383b32bc49), both carrying .completed. checkpoint.hdf5 unavailable (removed on completion); samples_info.json shows acceptance ratio 1.0 and no truncation on both. Finite-log_l scan: cholesky n=39,742 max 30,749.00 min 30,709.36; slogdet n=27,133 max 30,747.18 min 30,707.67; 0 samples > 1e10 in either arm, and no -inf/NaN rows recorded in either. PyAutoFit version stamp MISMATCH, and the two arms ran on different PyAutoFit trees — see the cross-cutting note; this is the one member where it is a genuine confound.

+
+ +
+

Readout

+
+ + + + + + + + + + + + + + + + + +
cholesky vs slogdet, same cell and target
cholesky (342017_0)slogdet (342017_1)Δ (slogdet − cholesky)
logZ30,684.313630,683.8129−0.50
maxL30,749.003930,747.1781−1.83
Δmax logL vs truth−772.740−774.566−1.83
total wall5,027.90 s4,467.09 s−560.8 s (−11.2 %)
likelihood evals77,90063,700−14,200
ms / eval64.5470.13+5.59 (+8.7 %)
Kish ESS8,726.54,903.5−3,823.0
samples > 1e10 (finite-garbage scan)00—
NaN / −inf samples recorded00—
vizdisableddisabled—
+
+

The two tiers agree. Δlog Z = 0.50 nats, Δmax logL = 1.83 nats. slogdet is 8.7 % slower per evaluation but finished 11 % sooner because it needed 18 % fewer evaluations — i.e. the wall difference is sampler-path noise, not a tier speed-up. Neither arm produced a single NaN, a single -inf, or a single finite-garbage sample: on the capped AdaptSplitPower target, the cholesky failure that motivated slogdet does not occur, so slogdet has nothing to rescue here.

+

Bonus — the programme's first same-configuration repeat. The cholesky arm and refs task 5 are the same cell, target (sha256:ad291b57fc62), n_live (300), seed (0) and positions (off): logZ 30,684.615 vs 30,684.314 (Δ 0.30 nats), maxL 30,748.575 vs 30,749.004 (Δ 0.43 nats), evals 73,700 vs 77,900 (5.7 % apart). That gives a ~0.3-nat empirical logZ reproducibility scale for this cell — directly usable wherever Gate A/C readings are quoted. Caveats: task 5 had visualization enabled (58 s) and the arms imported different PyAutoFit trees.

+

Raw JSONs local: packet/slogdet_ab/hpc_hpc_a100_fp64_slogdet_ab_{cholesky,slogdet}.json. Not present in the local repo; RAL-side and uncommitted.

+
+
+ Bar chart, truth vs best-fit, cholesky arm +
slogdet_ab/hpc_hpc_a100_fp64_slogdet_ab_cholesky.png — 342017_0
+
+
+ Bar chart, truth vs best-fit, slogdet arm +
slogdet_ab/hpc_hpc_a100_fp64_slogdet_ab_slogdet.png — 342017_1
+
+
+
+ +
+

Ruling

+

Write the DECISIONS verdict — “the two log-det tiers agree on the corrected target (Δlog Z 0.50 nats, zero NaN in either), so slogdet has nothing to rescue here” — or hold the verdict until the arms are re-run on one PyAutoFit tree.

+
+ +
+

Your review

+
+ + + + +
+ +
+ +
+

Follow-ups

+
    +
  • [autolens_profiling]Record the ~0.3-nat same-configuration logZ reproducibility figure in PROGRAMME.md §3 as the programme's first independent-run variance measurement.
    • +
  • [autolens_profiling]Feed this into the W9 / #166 “make slogdet standard” argument: on a capped free-adapt-split target the tiers are indistinguishable, so the case for slogdet rests entirely on uncapped cells.
    • +
  • [PyAuto stack]Freeze the RAL PyAuto checkout for the duration of a campaign (tag + git -C … status echo into every .out) so two arms of one A/B cannot import two different trees.
    • +
+
+ +
+

Where to look yourself

+
+
Project root
+
/home/jammy/Code/PyAutoLabs/autolens_profiling
+
Pulled artefacts (local)
+
…/scratchpad/packet/slogdet_ab/ (2 JSONs + 2 PNGs)
+
Output folders
+
/mnt/c/Users/Jammy/Science/inference_programme/output/searches/nautilus/imaging/slam_source_pix_nn/hst/ld_cholesky/hpc_a100_fp64_slogdet_ab_cholesky/ae454f54e14847ca62929c70033c11a5/
+
…/ld_slogdet/hpc_a100_fp64_slogdet_ab_slogdet/7eb80135de1db2112c5bb5383b32bc49/ — each with model.results, search.summary, .completed, image/
+
Batch logs
+
/mnt/c/Users/Jammy/Science/inference_programme/logs/output/output.342017_{0,1}.out
+
Scripts
+
submit …/hpc/batch_gpu/submit_search_nautilus_slogdet_ab_imaging_slam_source_pix_nn_a100_hst_fp64
+
entry point …/scripts/imaging/searches/nautilus/slam_source_pix_nn.py
+
arm tagging …/scripts/misc/searches/_samplers.py (log_det_arm_tag)
+
Ruling lands in
+
…/results/notes/inference/DECISIONS.md — a new dated entry beside the 2026-08-24 CP-4 / Phase 8A entry at line 291 and the 2026-08-24 W7 re-score at line 401
+
Open in order
+
    +
  1. packet/slogdet_ab/hpc_hpc_a100_fp64_slogdet_ab_cholesky.json and …_slogdet.json side by side — results.log_evidence and performance.
    2. +
  2. DECISIONS.md line 291 (“CP-4 / Phase 8A: slogdet FAILS the pre-registered A/B”) to see what the old verdict was scored on.
    3. +
  3. The submit script's “COLLISION SAFETY — READ THIS BEFORE ADDING AN ARM” block, then confirm both ld_* dirs exist on RAL.
    4. +
  4. packet/refs_v1/hpc_hpc_a100_fp64_ref.json for the same-configuration repeat comparison.
    5. +
+
+
+ + Est. review-minutes 12 +
+
+ +
+
+
+

subhalo_validation

+

pl_sersic_0 no-subhalo — the false-positive null

+

342027_0

+
+
+ Healthy + +
+
+
+ +
+

Question this run was submitted to answer

+

“run the full end-to-end SLaM chain (source_lp → source_pix → light → mass → subhalo grid search + refine) … on data that contains no subhalo … the validation quantity is the Δ log-evidence the subhalo stage reports — it should be small” (wiki/project/state.md §“Science goal”), with the hand-off instruction “read results/delaunay_adapt_split/*.json (Δlog-evidence no-subhalo — expect < 5; anything larger is a false positive to investigate)” (state.md §“Next session”, item 1).

+
+ +
+

Witness

+

evidence_increase < 5 for a no-subhalo lens. Larger ⇒ false positive to investigate.

+
+ +
+

Health evidence

+

sacct COMPLETED, wall 1-10:08:32 of a 2-day budget (70 %); error.342027_0.err is 0 bytes; 0 tracebacks in the 4.1 MB .out; the log ends on the real terminator (“Results written to … / Finished sample: pl_sersic_0”); all seven stage directories present with .zip archives and the final subhalo[2] zip written at 13:03:02. No huge finite log_l anywhere — every stage logZ is ~2.1e4. MaxRSS 27.0 GB against a 96 GB request (28 %). Stack: autolens 2026.8.17.1; PyAutoFit imported at 02:54 on the 30th, i.e. 56fb8b63b (the PR #1548 pool fix) — which was merged to main as e6ff1e654 at 03:52 the same night, so the run carries the intended fix. source_lp[1] reloaded from the shared output dir as designed (the “Fit Already Completed” reload is expected here, unlike the profiling reference rows).

+
+ +
+

Readout

+
+ + + + + + + + + + + + + +
The witness
quantityvalue
log_evidence_mass_total (no subhalo)21,377.0355
log_evidence_subhalo_refine21,376.7347
evidence_increase (the witness)−0.3009 → PASS (< 5, and negative)
log_likelihood_mass_total21,419.2107
log_likelihood_subhalo_refine21,420.1775
log_likelihood_increase+0.9668
spurious subhalo mass_at_2001.493e9 M⊙
spurious subhalo centre(2.1457, −0.2410) arcsec
+
+

This is the textbook null: the subhalo buys +0.97 nats of likelihood for 3 extra parameters and the evidence falls by 0.30 nats. The nuisance subhalo parks itself 2.16″ from the centre — outside the Einstein ring, where it does nothing.

+
+ + + + + + + + + + + +
Stage timeline, from the run-directory zip mtimes — the grid search is the whole cost
stageendedduration
source_lp[1] (reloaded)30 Aug 02:54:43—
source_pix[1]30 Aug 03:23:4329 min
source_pix[2]30 Aug 03:33:2910 min
light[1]30 Aug 04:47:0474 min
mass_total[1]30 Aug 07:35:57169 min
subhalo[1] grid + subhalo[2] refine31 Aug 13:03:0229.4 h (85 % of the run)
+
+

The subhalo grid is 2×2. scripts/imaging.py:94-95 sets subhalo_grid_dimension_arcsec: 3.0 and subhalo_number_of_steps: 2, so the “grid search” is four 3″-wide cells centred at (±3, ±3)-ish corners. All four agree (results.csv):

+
+ + + + + + + + + +
Grid cells vs the no-subhalo baseline
cellcentre_0centre_1log_evidencevs no-subhalo baseline 21,377.04
0−3.0−3.021,376.245−0.79
1−3.00.021,376.368−0.67
20.0−3.021,376.420−0.62
30.00.021,376.438−0.60
+
+

Every cell is below the baseline. The null is uniform across the grid — reassuring — but a 4-cell scan at 3″ resolution is a very coarse detection experiment for a lens with a ~1″ Einstein radius, and it costs ~29 h.

+

Figures: none pulled for this lens (hpc/sync pull excludes search_internal and this run's image/ directories are inside the per-stage .zip archives). See the two RUNNING lenses' dataset.png for what the data look like.

+

Raw result JSON: packet/subhalo_validation/pl_sersic_0_no_subhalo.json; grid CSV packet/subhalo_validation/pl_sersic_0_grid_results.csv. hpc/sync pull did not bring the witness JSON down — pull() iterates PULL_DIRS (output/, logs), not results/, so /mnt/c/Users/Jammy/Science/subhalo_validation/results/delaunay_adapt_split/ still does not exist locally. Both files above were scp'd into the packet directly.

+
+ +
+

Ruling

+

Accept pl_sersic_0 as a passed no-false-positive witness and record it in results_summary.md, and rule on whether a 2×2 / 3″-step subhalo grid is the intended detection experiment or a smoke-test leftover.

+
+ +
+

Your review

+
+ + + + +
+ +
+ +
+

Follow-ups

+
    +
  • [subhalo_validation]Fix hpc/sync pull to include results/ — the pre-registered witness lives there and the documented pull does not fetch it.
    • +
  • [subhalo_validation]Raise subhalo_number_of_steps (2 → 5) for the science runs, or record explicitly that the 2×2 grid is deliberate; at ~7 h per cell, cost the change first.
    • +
  • [subhalo_validation]Fill wiki/project/results_summary.md with evidence_increase for pl_sersic_0 once the other two lenses land, so all three go in together.
    • +
  • [autolens]Silence or fix the latent.yaml lists 'total_galaxy_0_flux' but no such latent is registered warning — it is emitted once per sample and is ~95 % of the 4.1 MB batch log.
    • +
+
+ +
+

Where to look yourself

+
+
Project root
+
/mnt/c/Users/Jammy/Science/subhalo_validation
+
Pulled artefacts (local)
+
…/scratchpad/packet/subhalo_validation/pl_sersic_0_no_subhalo.json and pl_sersic_0_grid_results.csv
+
Output folder (pulled locally, stage dirs hold .zip archives)
+
/mnt/c/Users/Jammy/Science/subhalo_validation/output/subhalo/detect/delaunay_adapt_split/pl_sersic_0_no_subhalo/ — in particular subhalo[2]_[single_plane_refine]/f5149ca75c58d241c808938aff23a21e.zip (holds model.results, image/) and subhalo[1]_[search_lens_plane]/ae37aada789b983fcd5dc54f9d6ef440/results.csv
+
Batch log (4.1 MB)
+
/mnt/c/Users/Jammy/Science/subhalo_validation/hpc/batch_cpu/output/output.342027_0.out
+
Witness JSON (now pulled — hpc/sync pull fixed)
+
/mnt/c/Users/Jammy/Science/subhalo_validation/results/delaunay_adapt_split/pl_sersic_0_no_subhalo.json
+
Scripts / recipe
+
submit /mnt/c/Users/Jammy/Science/subhalo_validation/hpc/batch_cpu/submit_source_pix (array index 0 → pl_sersic_0)
+
entry point /mnt/c/Users/Jammy/Science/subhalo_validation/scripts/imaging.py (grid settings at lines 94–95, subhalo_grid_search at line 472)
+
pipeline …/scripts/pipelines/ (delaunay_adapt_split)
+
Ruling lands in
+
/mnt/c/Users/Jammy/Science/subhalo_validation/wiki/project/results_summary.md (and the “Open, carried forward” list in wiki/project/state.md)
+
Open in order
+
    +
  1. packet/subhalo_validation/pl_sersic_0_no_subhalo.json — evidence_increase is the whole witness.
    2. +
  2. packet/subhalo_validation/pl_sersic_0_grid_results.csv — four rows is the surprise.
    3. +
  3. scripts/imaging.py lines 94–95 to see where four comes from.
    4. +
  4. wiki/project/state.md §“Next session” item 1, which is the instruction this member answers.
    5. +
+
+
+ + Est. review-minutes 10 +
+
+ +
+
+
+

subhalo_validation

+

pl_eff_0 and pl_eff_1_outer — still in the refine

+

342027_[1,2] · euclid-ral-compute-21 · wall 2026-09-01 02:54

+
+
+ Running + +
+
+
+ +
+

Question this run was submitted to answer

+

The same false-positive witness as above, on the two clumpy ElsonFreeFall sources: pl_eff_0 (six components) and pl_eff_1_outer (a seventh compact component outside the tangential caustic, giving an extra doubly-imaged pair) — wiki/project/state.md §“Data on hand”.

+
+ +
+

Witness

+

evidence_increase < 5 per lens, from results/delaunay_adapt_split/{pl_eff_0,pl_eff_1_outer}_no_subhalo.json. Not yet written — both are in the final stage.

+
+ +
+

Health evidence

+

squeue: both RUNNING on euclid-ral-compute-21, elapsed 1-12:37:32 of a 2-00:00:00 budget (75 %), wall at 2026-09-01 02:54. error.342027_{1,2}.err are 0 bytes; 0 tracebacks in either .out. Both are in the last stage — subhalo[1] grid search is complete for both (results.csv written, all four cells) and subhalo[2]_[single_plane_refine] started 31 Aug 13:49:37 (pl_eff_0) and 14:31:09 (pl_eff_1_outer). Live checkpoint.hdf5 (mtime 15:33, minutes old):

+
+ + + + + + + +
Live checkpoint read
tasklensstageboundsn_likeexploredn_live / n_dimrefine elapsed
342027_1pl_eff_0subhalo[2]_[single_plane_refine]1616,060False600 / 111 h 48 m
342027_2pl_eff_1_outersubhalo[2]_[single_plane_refine]1210,480False600 / 111 h 06 m
+
+

Both are advancing (~150 likelihood calls/min); late bounds show 99.0–99.5 % proposal rejection, which is normal nautilus shell-narrowing, not a stall.

+

Memory is flat, and the “6–9 GB healthy” baseline was a stage-1 reading. sstat: 342027.batch MaxRSS 32.82 GB / AveRSS 32.06 GB; 342029.batch (the second array task's underlying job id) MaxRSS 29.91 GB / AveRSS 28.91 GB. Ave within 3 % of Max ⇒ the footprint stepped up early and has been flat since, not creeping. Both are 31–34 % of the 96 GB request, and the completed task 0 finished at MaxRSS 27.0 GB — so ~27–33 GB is this chain's real plateau. The hand-off's 5.9–6.4 GB was measured 13 minutes in, during source_pix[1]. This is not the PR #1548 leak returning.

+

CPU: AveCPU 9-19:39 and 9-18:44 over 1-12:37 elapsed ⇒ 6.4 of 8 cores busy on both, matching the hand-off's ~6 cores.

+
+ +
+

Readout

+

No science numbers yet. ETA: task 0's subhalo[1]+subhalo[2] legs took 29.4 h combined and its grid alone ran ~28 h; these two grids took 28.9 h and 29.9 h, and task 0's refine leg was therefore ~1.5 h. pl_eff_0 is 1.8 h into its refine, pl_eff_1_outer 1.1 h, with ~11 h of budget left. Both should finish well inside the wall unless the clumpy source makes the refine several times slower than the smooth one — worth one more check this evening rather than a pre-emptive requeue.

+

Raw JSONs after the pull: not yet written — they will appear on RAL at /mnt/ral/jnightin/subhalo_validation/results/delaunay_adapt_split/{pl_eff_0,pl_eff_1_outer}_no_subhalo.json and land at /mnt/c/Users/Jammy/Science/subhalo_validation/results/delaunay_adapt_split/ on the next hpc/sync pull. The pull() gap noted above is fixed: that project’s PULL_DIRS now includes results, and the zip counter no longer aborts the pull part-way.

+
+
+ Masked imaging data going into the pl_eff_0 refine +
subhalo_validation/pl_eff_0_refine_dataset.jpg — masked data going into the refine
+
+
+ Masked imaging data going into the pl_eff_1_outer refine +
subhalo_validation/pl_eff_1_outer_refine_dataset.jpg — masked data going into the refine
+
+
+ light[1] fit for pl_eff_0 +
subhalo_validation/pl_eff_0_light1_fit.jpg — the light[1] fit, the most recent completed-stage fit image available locally
+
+
+

Figures pulled from the live image/ dirs, which hpc/sync pull did fetch.

+
+ +
+

Ruling

+

Leave both to finish on the current wall (recommended) / requeue with a longer --time now, before the 2026-09-01 02:54 deadline forces the choice.

+
+ +
+

Your review

+
+ + + + +
+ +
+ +
+

Follow-ups

+
    +
  • [subhalo_validation]Re-poll checkpoint.hdf5 explored on both tasks around 20:00 tonight; if either is still explored: False past ~22:00, decide requeue vs let-it-wall while there is still time.
    • +
  • [subhalo_validation]Once both land, read all three evidence_increase values into wiki/project/results_summary.md in one edit (state.md “Open, carried forward” item 1).
    • +
  • [subhalo_validation]Record the measured stage costs (grid search ≈ 29 h of a ≈ 34 h chain) in the project so the next --time and the rectangular_adapt comparison run are sized from data, not from the 48 h guess.
    • +
  • [subhalo_validation]hpc/batch_cpu/template and subhalo_simulations/hpc/batch_cpu/submit_imaging still say --partition=cpu and lack PyAutoNerves on PYTHONPATH (state.md §“Next session” 3d) — fix before the rectangular_adapt run reuses them.
    • +
+
+ +
+

Where to look yourself

+
+
Project root
+
/mnt/c/Users/Jammy/Science/subhalo_validation
+
Output folders (pulled locally, live)
+
/mnt/c/Users/Jammy/Science/subhalo_validation/output/subhalo/detect/delaunay_adapt_split/pl_eff_0_no_subhalo/subhalo[2]_[single_plane_refine]/2429cc0206b60e10da18b7c950f9804f/
+
…/pl_eff_1_outer_no_subhalo/subhalo[2]_[single_plane_refine]/6f9d0ee150e8eba94a0797ed1061e63c/ — each has image/dataset.png, image/image_with_positions.png, files/; search_internal/ is excluded by the pull
+
Live checkpoints (excluded from the pull by design)
+
RAL only…/2429cc0206b60e10da18b7c950f9804f/files/search_internal/checkpoint.hdf5 and …/6f9d0ee150e8eba94a0797ed1061e63c/files/search_internal/checkpoint.hdf5 under /mnt/ral/jnightin/subhalo_validation/output/subhalo/detect/delaunay_adapt_split/
+
Completed grid searches (pulled)
+
…/pl_eff_0_no_subhalo/subhalo[1]_[search_lens_plane]/fe8dd2650a5531db17a336821aa52b98/results.csv
+
…/pl_eff_1_outer_no_subhalo/subhalo[1]_[search_lens_plane]/814f813355599c8ce590014ec8de6c7f/results.csv
+
Batch logs (4.0–4.3 MB; the Nautilus progress line is block-buffered and only flushes on exit, so read the checkpoint, not the log)
+
/mnt/c/Users/Jammy/Science/subhalo_validation/hpc/batch_cpu/output/output.342027_{1,2}.out
+
Scripts / recipe
+
/mnt/c/Users/Jammy/Science/subhalo_validation/hpc/batch_cpu/submit_source_pix (array indices 1, 2)
+
/mnt/c/Users/Jammy/Science/subhalo_validation/scripts/imaging.py
+
/mnt/c/Users/Jammy/Science/subhalo_validation/hpc/sync (jobs, pull)
+
Ruling lands in
+
/mnt/c/Users/Jammy/Science/subhalo_validation/wiki/project/state.md §“Next session” (rewrite at hand-off) and, once the numbers land, …/wiki/project/results_summary.md
+
Open in order
+
    +
  1. hpc/sync jobs to confirm both are still R.
    2. +
  2. packet/subhalo_validation/pl_eff_0_refine_dataset.jpg to see the clumpy-source data the refine is fitting.
    3. +
  3. The two subhalo[1] results.csv files above — the grid legs are already done and their four cells are the pre-refine read of the same witness.
    4. +
  4. wiki/project/state.md §“Next session” item 1 for the instruction, and its “Confirmed 2026-08-30 03:07 BST” note for the 6–9 GB figure this packet revises.
    5. +
+
+
+ + Est. review-minutes 8 +
+
+ +
+
+
+

euclid_strong_lens_modeling_pipeline · retrospective

+

euclid-dr1-prep — retrospective of phase 2

+

PR #46 · issue #45 · main @ 361e83b

+
+
+ Merged + +
+
+
+ +
+

Question — what it set out to do

+

Give euclid_strong_lens_modeling_pipeline its first CI: TEST-mode smoke over every example script running on committed simulated data (deliberately not the auto-simulate convention every other PyAuto workspace uses, because this repo's users fit real Euclid data), plus known-answer unit tests on every Analysis latent and a run-level check that a fit actually writes them.

+
+ +
+

Witness

+

The epic ledger's phase-2 decision checklist — each decision resolved to a path and a pattern in the live repo (the table under Readout) — together with the phase's own CI: 9/9 checks green on the PR head and green again on main after the merge.

+
+ +
+

Health evidence — what shipped

+

PR #46 test: TEST-mode CI on committed simulated data + latent known-answer tests (epic phase 2), merged 2026-08-29 23:24 as 361e83b, 55 files, +3670 / −472 text lines (gh reports −7741 because it counts the deleted binary datasets). Four commits, staged A–D: c78f789 simulator + dataset, a8ce0c3 tests, ceba36b CI, 2633e00 docs. CI: 9/9 checks green on the PR head (Smoke Tests run 33278353869, Tests run 33278353935), and both re-ran green on main after the merge (33280913921, 33280913907). Nothing has pushed to main since, so those are still the live green runs. Issue #45 closed on merge.

+

Phase 1 for context (the structural review spans it): PR #44, merged 4a0d9c5, 63 files, +4484 / −531; issue #43 closed. It ran zero checks — the repo had no .github/workflows/ at all — so it merged on local evidence only. That gap is exactly what #46 closed.

+
+ +
+

Readout — glance at the PR, the hunks worth your eyes

+
    +
  • tests/test_compute_latent_variable.py:254 (_assert_replays) — the shape of the whole latent suite. Each of the 12 latents gets two assertions: a replay against truth.json, and an independent check against the simulator's separately-recorded truth block. The replay leg compares the pipeline's computation to a number the same pipeline wrote, so it would pass a consistently-wrong formula; the independent leg is the one with teeth. For the 4 aperture latents that independent leg is monotonic scaling with FWHM (:352, parametrized k=1..4), not an absolute flux — the weakest rung in an otherwise strong suite. Worth knowing before you trust “12/12 known-answer”.
    • +
  • tests/test_latent_run_level.py:34 and :262 — the NaN check is a key-count assertion, because a NaN latent is dropped from latent_summary.json entirely rather than written as NaN. Clever and correct, but it means the test hard-codes == 12; enabling a latent in config/latent.yaml breaks CI by design. Science-adjacent: this is the only check that the pipeline still emits the DR1 catalogue columns at all.
    • +
  • scripts/simulator.py:558 (apply_sersic_index_prior_edge_rule) — a behaviour change with science consequences downstream. When a --from-result resimulation finds the lens-light sersic_index pinned at the prior edge of 5, it silently lowers it to a chosen replacement and records both sersic_index_inferred and sersic_index_simulated in truth.json. That is the epic's phase-5 instruction implemented early; phase 6a's whole question (“do we recover Sersic indices?”) reads those two fields. Check the replacement default matches what you meant.
    • +
  • scripts/simulator.py:70 (docstring) — --from-result applies the same fitted intensities to every band, i.e. a flat SED, recorded as sed: "flat" in truth.json. Documented and deliberate, but phase 5 inherits it, and a flat SED is not a Euclid colour. This is the single largest known simplification in the phase-2 deliverable.
    • +
  • config/build/no_run.yaml (+41 lines, 17 new reasoned exclusions) — this is what “CI over every example script” actually means. The smoke gate runs 9 entries; the repo has 49 tracked .py files. Everything excluded (7 catalogue producers, 6 workflow/aggregator examples, 4 GUIs, 3 preprocess/one-off tools, 2 CI runners, 2 shared modules) carries a written reason, and tests/test_repo_invariants.py enforces that every script is in one list or the other. The exclusions are honest, but the headline “every script” is really “every script on the fit chain”.
    • +
+

All five paths are under /home/jammy/Code/PyAutoLabs/euclid_strong_lens_modeling_pipeline/.

+
+ +
+

Readout — repo structure (main @ 361e83b, 221 tracked files, 5.0 MB, 3.9 MB of it data)

+ 
(root)                    start_here.py (63-line shim) · util.py (989 L) · __init__.py ·
+                          smoke_tests.txt · pytest.ini · activate.sh · AGENTS/README/CLAUDE/AI_POLICY ·
+                          test_report.md (generated, stale) · sie_fit.png (stray)
+.github/    5 files       2 workflows (Smoke Tests, Tests) + 3 runner scripts — the whole of phase 2's CI
+scripts/    13 files      the fit chain: initial → sersic → waveband → sersic_waveband, plus full_model,
+                          mge_lens_only, simulator, 2 latent diagnostics, build_inspect + bundle .sh
+catalogue/  10 files      7 producers of the DR1 reference-tile CSVs + catalogue_util.py + README
+                          (13-file → producer table lives here)
+config/     92 files      77 of them priors/; build/ (no_run.yaml, profile_smoke.yaml), latent.yaml,
+                          visualize/, non_linear/
+dataset/    20 files      q1_walsmley/ (1 real lens, 6 files) + simulated/euclid_dr1_like/ (13 files,
+                          incl. truth.json + segmentation/) + README
+tests/      9 files       5 test modules + 4 checked-in DR1 header fixtures
+workflow/   33 files      aggregator examples (csv/png/fits) + their reference outputs — all no_run
+hpc/        12 files      batch_cpu (1 submit + template), batch_gpu (3 submits), sync + example conf
+preprocess/ 5 files       segmentation, mask GUIs, one-off migration
+tools/      4 files       3 GUI/PSF utilities
+docs/       1 file        drift_report.md — the phase-1 port record, 11 sections
+
+ + + + + + + + + + + + + + + +
Decision checklist
Decision (from the ledger)LandedPath / evidence
full_model.py Delaunay, matching autolens_workspace .../pixelization/delaunay.pyYscripts/full_model.py:499 (stage 1 Delaunay + AdaptSplit, Overlay(26,26)+30 edge pts) and :567 (stage 2). Matches the workspace file's SLaM stages (delaunay.py:685, :770), not its standalone example at :376 which uses ConstantSplit
Hilbert pixels=500Yscripts/full_model.py:524-527; deviation from the workspace's 1000 documented in docs/drift_report.md §4 (VIS cut-outs are small; VRAM)
Critical-path scripts only (initial → sersic → waveband → sersic_waveband)Yall four in scripts/; _pix variants, galaxy_sersic_model.py, audit_*, reorganize_normies.py absent and listed in docs/drift_report.md §10
util.py latents + WORST_PSF_* machineryYutil.py:290 (psf_fwhm_arcsec_from_primary_header, WORST_PSF_MER/_HDR/_PSF with the −99 sentinel), :831 (WORST_BAND degradation path), :459 LatentEuclid (8 library + 4 aperture = 12). 989 L vs the science tree's 978
13-file reference-tile bundle + build_inspect.py / bundle .shY (with a deliberate override)scripts/build_inspect.py, scripts/build_inspection_bundle.sh, producer table in catalogue/README.md:26. Note the ledger's phase-1 user decision listed build_inspect.py as not to port; it was ported anyway to reach 13/13, and that reversal is documented in docs/drift_report.md §9 rather than hidden
start_here.py as a shimY63 lines, forwards to scripts.initial_lens_model.fit; docs/drift_report.md §2
mge_lens_only.py keptYscripts/mge_lens_only.py, in smoke_tests.txt
dataset/simulated/euclid_dr1_like/ with truth.jsonY13 files, 883 KB, incl. segmentation/, positions.json, truth.json (250 L)
CI: Smoke Tests + Tests via Heart's reusable workflowY.github/workflows/smoke_tests.yml, tests.yml — both uses: PyAutoLabs/PyAutoHeart/.github/workflows/smoke-tests.yml@main; Tests splits unit / slow
config/priors/mesh/voronoi.yaml naming an absent classN/A — already fixedThe file was deleted in phase 1's config drift sweep (90955ac), with priors/hyper_data.yaml and image_mesh/kmeans.yaml; reasons in docs/drift_report.md §8. config/priors/mesh/ now holds only classes that exist
+
+
+ +
+

Readout — things that look off

+
    +
  • The new CI gates nothing. PyAutoHeart/config/repos.yaml:97 required_workflows has keys for libraries, workspaces, workspaces_test, howto — but no pipelines, and euclid is in the pipelines group (:70). So the two workflows this phase built are polled but cannot turn Heart red. Why it matters: the phase's own value proposition (don't debug science and software at once in phase 4) is only half-delivered until a red euclid CI blocks something. Follow-up already filed at PyAutoMind/draft/test/pyautoheart/euclid_pipeline_release_blocking_gate.md — unstarted.
    • +
  • test_report.md is tracked, generated, stale, and carries absolute local paths — /home/jammy/Code/PyAutoLabs/... on lines 47–51, and its table covers 5 scripts when smoke_tests.txt lists 9. Its own header admits both. Why it matters: it is the only file in the repo with a machine-specific path, it is the first “did this pass?” artifact a new reader opens, and it currently reports a full_model.py PASS recorded against a version of the script that could not run.
    • +
  • sie_fit.png is tracked at the repo root while .gitignore has a rule (# Loose image at project root) written specifically to exclude it. Committed 2025-11-29 in a commit called “fixes”; ignore rules do not apply to already-tracked files, so the guard is inert. Why it matters: a contradiction between the ignore file and reality, and an undocumented artifact at the top level of a first-class public repo.
    • +
  • workflow/fits/hi_0.fits.fits — doubled extension, tracked past the *.fits ignore. Why it matters: cosmetic, but it is a reference output that someone will eventually try to match by name.
    • +
  • AGENTS.md's structure block omits four tracked root entries — test_report.md, sie_fit.png, AI_POLICY.md, CLAUDE.md, plus root __init__.py and .claude/. /home/jammy/Code/PyAutoLabs/euclid_strong_lens_modeling_pipeline/AGENTS.md:27-73. Why it matters: the block is the repo's map; the two undocumented entries are exactly the two above that shouldn't be there.
    • +
  • hpc/batch_cpu/ holds one submit script and a template, against 18 in the source-of-truth tree, and has none of the vis_lp_cpu_jax / vis_pix_cpu_nojax pair. Why it matters: not a defect — that pair is literally phase 3's deliverable — but if you read the repo today expecting the DR1 CPU route to be reproducible, it is not yet.
    • +
  • Duplicated utilities: nothing found. util.py (pipeline: dataset loading, analysis, latents) and catalogue/scripts/catalogue_util.py (arg parsing, path resolution, per-tile CSV splitting) share no function names.
    • +
  • Scripts outside the documented layout: nothing found.
    • +
  • Leftover _pix variants meant to be skipped: nothing found. The only _pix file is scripts/diagnose_latent_vis_pix.py, which is a latent diagnostic the decision explicitly asked for, correctly excluded from smoke with a reason.
    • +
  • Dead configs: nothing found beyond the voronoi/hyper_data/kmeans sweep already done in phase 1.
    • +
  • Missing __init__.py: nothing found that matters — scripts/, catalogue/, catalogue/scripts/, preprocess/, tools/, workflow/ and the root all have one; tests/ deliberately does not (rootdir collection, and the suite passes).
    • +
  • Test dirs vs CI mismatch: nothing found. pytest.ini testpaths = tests; the slow marker is applied module-wide at tests/test_latent_run_level.py:69 and CI's slow job passes --slow to .github/scripts/run_tests.py.
    • +
  • CRLF: nothing found — zero CRLF files, and .gitattributes pins eol=lf with *.fits/*.png opted out after the * rule (order is correct and commented).
    • +
  • Both completion records say a worktree was “deliberately left in place”; both are already gone (git worktree list shows only the canonical checkout). That follow-up is discharged — the records just haven't been updated.
    • +
+
+ +
+

Ruling

+

Structure is sound → phase 3 is dispatchable as-is. Nothing found blocks it; the only judgement call is whether to fold the three tidy-ups (delete sie_fit.png, regenerate-or-untrack test_report.md, sync AGENTS.md's structure block) into phase 3's PR or file them separately — and whether the euclid CI should become release-blocking before the phase-4 science run rather than after.

+
+ +
+

Your review

+
+ + + + +
+ +
+ +
+

Follow-ups

+
    +
  • [PyAutoHeart]Ship PyAutoMind/draft/test/pyautoheart/euclid_pipeline_release_blocking_gate.md — add a pipelines: ["Smoke Tests", "Tests"] key to Heart's required_workflows and put euclid in the weekly sweep. Small, and it is what makes phase 2 real.
    • +
  • [euclid_pipeline]Hygiene one-off: untrack sie_fit.png, regenerate or untrack test_report.md (it is the only source of hard-coded local paths), fix workflow/fits/hi_0.fits.fits, add the four missing root entries to AGENTS.md:27.
    • +
  • [euclid_pipeline]Strengthen the 4 aperture latents' independent leg from monotonic-scaling to an absolute µJy value against truth.json's aperture block (the data is already there).
    • +
  • [PyAutoMind]Note in the phase-5 prompt that simulator.py --from-result writes a flat SED; decide whether a colour-correct SED is in scope before the resimulations run.
    • +
  • [PyAutoMind]Housekeeping: both completion records claim a worktree was left in place; both are gone — correct the records or leave them, but don't act on them.
    • +
+
+ +
+

Where to look yourself, in order

+
+
Open in order
+
    +
  1. /home/jammy/Code/PyAutoLabs/euclid_strong_lens_modeling_pipeline/ — repo root, main @ 361e83b, clean.
    2. +
  2. /home/jammy/Code/PyAutoLabs/euclid_strong_lens_modeling_pipeline/AGENTS.md — the structure block at :27 and the CI section at :229; this is the map.
    3. +
  3. /home/jammy/Code/PyAutoLabs/euclid_strong_lens_modeling_pipeline/docs/drift_report.md — 11 sections; §4 (Delaunay), §9 (13/13 coverage), §10 (what was deliberately not ported). This is the phase-1 audit trail and it is genuinely good.
    4. +
  4. /home/jammy/Code/PyAutoLabs/euclid_strong_lens_modeling_pipeline/scripts/full_model.py and /home/jammy/Code/PyAutoLabs/euclid_strong_lens_modeling_pipeline/scripts/simulator.py — the two files that define what the pipeline fits and what it simulates.
    5. +
  5. /home/jammy/Code/PyAutoLabs/euclid_strong_lens_modeling_pipeline/.github/workflows/tests.yml — the CI file with the reasoning in it (why unit and slow are separate jobs).
    6. +
  6. /home/jammy/Code/PyAutoLabs/euclid_strong_lens_modeling_pipeline/dataset/simulated/euclid_dr1_like/truth.json — the number every latent test is measured against.
    7. +
  7. /home/jammy/Code/PyAutoLabs/PyAutoMind/draft/feature/euclid/euclid_dr1_prep_epic.md — the ledger. Phase 3 starts from line 197: draft/feature/euclid/cpu_vis_lp_jax_vis_pix_numba_submission.md — preserve and document the two-stage vis_lp (JAX) → vis_pix (numba + multiprocessing) CPU route and the full-JAX-GPU route. Gate 1 (satisfied); can overlap 2 (shipped).
    8. +
+
+
+ +
+

Est. review-minutes

+

~10 to skim PR #46's five flagged hunks, ~15 in the repo tree with AGENTS.md and drift_report.md open side by side, ~10 to rule on the Heart gate and the tidy-up bundling. Add 20 if you also want to read phase 1's PR #44 diff rather than trusting its drift report.

+ Est. review-minutes 35 +
+
+
+ +
+
+
+

Applies to every profiling member

+

Cross-cutting: the RAL PyAutoFit stack moved during the campaign

+

342016_[5,6,7] · 342017_[0,1] · 342018_[0,1,2]

+
+
+ Provenance +
+
+
+ +
+

This applies to every profiling member and is the reason none of them can be stamped “version match Y” without qualification.

+

Expected stamps at submission: Fit b70cf7fc3 / Array 302d5df32 / Galaxy 2ee44d507 / Lens af514d179. PyAutoArray, PyAutoGalaxy and PyAutoLens match exactly and did not move after 2026-08-29 23:40. PyAutoFit moved four times overnight (git -C /mnt/ral/jnightin/PyAuto/PyAutoFit reflog --date=iso):

+
+ + + + + + + + + +
PyAutoFit on disk, overnight
time (BST)PyAutoFit on disk
29 Aug 23:38:11b70cf7fc3 (main) — the expected stamp
29 Aug 23:50:381f7808e69 (feature/nautilus-serial-bound-training)
30 Aug 02:53:0756fb8b63b (same branch, pool fix)
30 Aug 03:51:53 → 03:52:45b70cf7fc3 → e6ff1e654 (main, PR #1548 merged)
+
+

Mapping onto import time (each job imports in its first ~30 s):

+
+ + + + + + + + + + + +
Which tree each task actually imported
taskwindowPyAutoFit importedmatches expected?
342016_523:44–01:06b70cf7fc3Y
342016_601:06–02:211f7808e69N
342016_702:21–03:201f7808e69N
342017_0 (cholesky)03:20–04:4456fb8b63bN
342017_1 (slogdet)04:44–05:58e6ff1e654N
342018_0/1/2 (SMC)05:58–06:52e6ff1e654N
+
+
+ +
+

Two separate consequences

+
    +
  1. b70cf7fc3 → 1f7808e69 → 56fb8b63b touch only nautilus/search.py (serial bound training, then the _LikelihoodWorkerPool fix). Every profiling run here is number_of_cores: 1 / force_x1_cpu: true, so no pool is built and the change is plausibly inert — but that is a reading of the diff, not a verified equivalence.
    2. +
  2. b70cf7fc3 → e6ff1e654 additionally carries dc53a2597 / f661b75c9 “ship log_likelihood_ceiling disabled by default”, which flips autofit/config/general.yaml from log_likelihood_ceiling: 1.0e+20 to blank. So 342016_* and 342017_0 ran with the overflow backstop and 342017_1 + 342018_* ran without it. Worse: the repo-level opt-in was not there either — the local repo added log_likelihood_ceiling: 1.0e+20 to autolens_profiling/config/general.yaml in dfeafbf (merged 6fe8f72), but the RAL checkout is at f284f24, one merge behind, and grep log_likelihood_ceiling /mnt/ral/jnightin/autolens_profiling/config/ returns nothing. No task in this campaign had the repo-level ceiling the submit headers assume. +

    Mitigation, and it is a strong one: the samples.csv scan over all five nautilus runs found zero samples above 1e10 (largest finite log_l anywhere: 31,406.12). The ceiling never had anything to clip, so the difference is a provenance defect rather than a numerical one. The one place it still bites is the slogdet A/B, whose two arms differ in log_det_method and in whether the guard was armed — the confound is inert on this evidence, but it is not clean.

    +
    3. +
+
+
+
+ +
+ Packet-building notes +
+ +
+

What took the time — science-run packet

+
    +
  • Reconstructing the PyAutoFit timeline. sacct says COMPLETED and every JSON says "version": "2026.8.17.1", so the recorded stamp is identical across all six tasks and looks fine. The drift only surfaces from git reflog --date=iso on the RAL checkout cross-referenced against per-task start/end times. That was ~20 minutes and it changed the reading of two members.
    • +
  • Chasing the ceiling. Finding that general.yaml's ceiling had been switched off upstream mid-campaign, and that the repo-level opt-in had not reached RAL, took three separate checks (PyAutoFit diff → PyAutoFit config diff → RAL autolens_profiling/config/). Neither fact is visible from any artefact the runs produced.
    • +
  • The SMC diagnostics. The top-line JSON fields (converged: true, valid: true, ESS 210/256) all look healthy; the failure is only legible in acceptance_rate_list, lambda_list and log_likelihood_increment_list read together, plus the comparison against a number that lives in a different file (REFS_V1_HARVEST.md's 31,690.42) and a sentence in PROGRAMME.md line 29.
    • +
  • RAL NFS. Batching remote commands kept this to ~15 ssh round-trips; find over output/ was avoided except at -maxdepth 2/3.
    • +
+
+ +
+

What could not be verified, and why

+
    +
  • checkpoint.hdf5 n_like / explored for all five profiling nautilus runs. The runs log “Removing search internal folder” then “Removing all files except for .zip file” on completion, and search_internal/ is not in the resulting zip. The named provenance check is therefore unrecoverable after the fact; files/samples_info.json (total samples, acceptance ratio, no truncation) and the samples.csv finite-log_l scan were substituted. This is exactly the check the harvest rules ask for, and it is being deleted.
    • +
  • NaN / -inf counts for the slogdet arms. samples.csv only records accepted samples, so “0 NaN” here means “no NaN reached the posterior”, not “the likelihood never returned NaN”. A true resample/NaN count needs the sampler-internal state that was deleted with search_internal.
    • +
  • Whether the nautilus/search.py changes between b70cf7fc3 and 56fb8b63b are truly inert at number_of_cores: 1. The diff was read; no equivalence test was run.
    • +
  • Task 0's per-stage sampler timings in subhalo_validation — unzip -p <zip> files/samples_info.json did not resolve inside the archives, so the stage table is built from zip mtimes (correct for durations, but it gives no total_samples or per-stage logZ).
    • +
  • Why the SMC warm arms start 124,000 nats below their MAP source. Diagnosed from the artefacts as a broken warm start; the actual cause is inside af.SMC's InitializerParamStartPoints / prior_scaled path and needs a local reproduction, which is development work, not harvest.
    • +
+
+ +
+

What a batch collect for science runs would need to automate

+
    +
  1. Never trust State. For every task: .err tail, .out traceback/resume-marker grep, wall-vs-Timelimit ratio, ExitCode (and specifically flag 0:53, which means SLURM could not open the log path and no .err exists to read).
    2. +
  2. Stamp the stack per task, at import time, into the artefact. "version": "2026.8.17.1" is not a provenance stamp — a package version cannot see a git checkout. Every submit should echo git -C $PyAutoX rev-parse HEAD for all five repos into the .out and the results JSON, and the collector should diff them against the campaign's declared stamps and against each other across tasks of the same array. This one check would have caught both cross-cutting findings automatically.
    3. +
  3. Preserve, don't delete, the sampler state on reference-grade runs — or at minimum copy checkpoint.hdf5's sampler attributes (n_like, explored, shell_log_l, shell_n_eff) into the results JSON before the folder is removed. Half the harvest rules currently cannot be applied to a completed run.
    4. +
  4. Run the finite-log_l scan automatically over samples.csv (max, min, count > ceiling, count non-finite) and put the four numbers in the results JSON. It is cheap and it is the failure signature the whole overflow-flood wave exists for.
    5. +
  5. Score against the pre-registered witness, not the sampler's own flag. The collector should read the witness out of the notes (REFS_V1_HARVEST bar, TOLERANCES.md row, state.md's “< 5”) and print witness / observed / PASS-FAIL. And it should treat a sampler's self-reported converged / valid as input, never as verdict — Phase 7 is the proof.
    6. +
  6. Pull the witness file. subhalo_validation's documented hpc/sync pull fetches output/ and logs but not results/, which is where the pre-registered number is written. Any collector must fetch the results directory explicitly.
    7. +
  7. Diff sibling arms of an A/B before reporting them. Same target_id, same seed, same n_live, same imported stack — and say so, or say which of those does not hold. The slogdet pair failed the last of those four and nothing in the artefacts said so.
    8. +
+
+ +
+

What a future automated retrospective for a merged phase would need — Euclid member

+
    +
  1. gh pr diff N --stat does not exist on gh 2.x — the reliable stat is git diff --stat <merge-parent-1> <merge-parent-2> from a local clone, and it disagrees with gh pr view --json additions,deletions whenever binaries move (here 472 vs 7741 deletions). Report the git number and say why.
    2. +
  2. The retrospective's spine is the completion record in PyAutoMind/complete/YYYY/MM/, not the PR — it already carries commits, verification, scope-vs-filed, traps, follow-ups and the heart-ack. A generator should diff the record's claims against the live repo and report only the deltas; that is where all six findings above came from.
    3. +
  3. Ledger decisions need a machine-checkable form. Every check here was a bespoke grep (“does full_model.py contain Hilbert(pixels=500)”). A Verify: block in the phase prompt listing path:pattern pairs would turn the checklist table into a script, and would have caught the build_inspect.py decision-reversal automatically instead of by eye.
    4. +
  4. Follow-ups filed as Mind drafts (draft/test/pyautoheart/...) are invisible from the shipped repo. A retrospective must resolve each record's follow-ups: line to a file and report its state (filed / issued / shipped) — an unstarted follow-up is the most common reason a “merged” phase is not actually finished.
    5. +
  5. Record-vs-reality drift is normal and cheap to detect: both records here claim worktrees still exist that were removed. Check the small factual claims (worktrees, CI run ids, file counts) mechanically before quoting the record to a human.
    6. +
+
+ +
+
+ +
+ +
+ Sources: /tmp/claude-1000/-home-jammy-Code-PyAutoLabs/7130ec3d-8b5a-4b7d-884f-26c6c3f0da1d/scratchpad/packet/REVIEW_PACKET.md
+ /tmp/claude-1000/-home-jammy-Code-PyAutoLabs/7130ec3d-8b5a-4b7d-884f-26c6c3f0da1d/scratchpad/packet/EUCLID_MEMBER.md
+ Figures: .../scratchpad/packet/{refs_v1,slogdet_ab,smc_probe,subhalo_validation}/ +
+ +
+

Ruled 0 of 8 · decisions 0 of 8

+ +
+ + + + +
+
+
+ + + + diff --git a/batches/packets/2026-08-31-pm.html b/batches/packets/2026-08-31-pm.html new file mode 100644 index 0000000..8f61c41 --- /dev/null +++ b/batches/packets/2026-08-31-pm.html @@ -0,0 +1,1980 @@ + + + + + +Slot Review 31 Aug 2026 · PM + + + + + +
+ +
+

31 Aug 2026 · pm · both lanes · first human-composed batch under the review workflow

+

Slot Review — PM

+

Eleven members, human-pasted from the dashboard and dispatched 18:40Z — nine cloud dev members and two laptop/RAL science members. This review covers the nine ready members: six delivered (five PR sets green + one research verdict) and three parked at ship. The two laptop members’ overnight results are now in — the jax-refs 342091 array is complete (10/10) and both subhalo pl_eff witnesses landed clean; they still carry into the next batch’s packet for their formal ruling (they have no review chip here), but the readouts below are final except for the one rectangular chain still running. The jax-refs leg also routed three legacy-reuse rulings into this slot, included below. Choose a decision and leave a note on each member, then Submit review at the bottom — the review lands in PyAutoMind/batches/reviews/ for the orchestrator chat to enact.

+

Science member outputs mirror to the laptop via each project’s hpc/sync pull at collect; dev members are reviewed on their PRs.

+

Generated 2026-08-31 18:40Z · refreshed 2026-09-01 13:01Z (overnight harvest: the whole 342091 array landed — 10 of 10 tasks ended clean on the A100, per-task table refilled with maxLL/logZ and the knn row finally replaced by a real run; subhalo — both pl_eff delaunay chains finished inside their wall and wrote clean witnesses, 342095 rectangular chain now in subhalo[1] — earlier refreshes: 22:57Z subhalo AdaptSplit figures, 22:53Z jax-refs table, 22:00Z CI green on all three opened PRs)

+

Permanent home: pyautolabs.github.io/PyAutoMind/packets/2026-08-31-pm.html

+
+ +
+
14Members
+
6Delivered
+
6Rulings (3 new PRs, all green)
+
2Carried to next batch
+
+ 130 + Est. review-minutes + 20 + 3 + 20 + 0 + 0 + 3 + 3 + 3 + 30 + 5 + 4 + 4 (+ carried: 15 + 20) + budget uncapped this slot (human at dispatch) +
+
+ +
+
+ + +
+
+ + +
+

First timed batch: the minutes figure is the first real calibration point the review-cost estimate gets.

+
+ +
+

Most important finding

+

The prodigy-49 bug as reported does not exist: a MultiStart search cannot stop at a quick-update boundary (the cadence is inert for that family). The “49” was Total Samples = 49 — a construction-time count — misread as a step count because the summary never reported the run’s real length. PR #1555 makes the summary state steps and stop reason, and pins the non-interaction with a tripwire the sibling wiring branch must flip. Merge order for the PyAutoFit trio: #1554 → #1555 → the parked wiring branch.

+
+ +
+

Rulings needed

+
    +
  1. release-smoke-env — now PR #274 — CI green (3/3) — merge / amend / reject.
    2. +
  2. autofit-multistart-iterations — now PR #1556 — merges last of the trio; workspace phase-2 queues either way.
    3. +
  3. euclid-3a-prose — now PR #48 — merge / amend / reject, noting the adversary leg was NOT run and judgment call 1 (AGENTS.md scope).
    4. +
  4. numba-vs-jax-sparse — rule on the verdict (“keep both, port nothing”); optionally /intake its two scholar-mode candidates.
    5. +
  5. autofit-prodigy-49 — accept the reframed root cause (reported bug does not exist; summary misread) before merging.
    6. +
  6. memory-queue-filing-gate — after merging: re-drive #71/#72 (label off/on) and #69 — the witness completes post-merge.
    7. +
  7. mge-pos-ref-reuse — adopt the quarantined-but-reusable 340210_9 run as the mge_pos_fp64 InferenceRefs_v1 reference, or reject it and queue array task 9 fresh.
    8. +
  8. mge-fp64-retro-baseline — does the retro-adopted mge_fp64 baseline stand under the redo standard, or does it need a fresh _ref run (for which no array task exists)?
    9. +
  9. delaunay-fp64-retro-baseline — the delaunay_fp64 baseline rests on legacy_wrong (never-cite) evidence and has no array vehicle: add array task 11 now, or defer until the 342091 wave lands?
    10. +
+
+ +
+ + + +
+ + + + + + + + + + + + + +
+
+
+

PyAutoHands · bug · supervised · parked

+

Release smoke ignores ENV declarations

+

PyAutoHands#274 (from the parked branch) · issue #272

+
+
+ PR green · 3/3 checks + +
+
+
+
+

Question this member was dispatched to answer

+

The run_smoke_tests job in release.yml ignores per-script ENV: declarations (witt_wynne.py’s full_datasets), so PointSolver cases run under SMALL_DATASETS and return the fixed 2-image pair.

+
+
+

Witness

+

None declared — review is the diff plus a release-smoke run honouring the declaration.

+
+
+

Health evidence

+

Branch feature/release-smoke-env-declarations (head b7df526) pushed to PyAutoHands; no PR opened — the run parked at the ship checkpoint per its supervised autonomy, with the batched question on PyAutoHands#272. Verified against GitHub at collect: branch present, open-PR list does not contain it.

+
+
+

Ruling

+

PR opened at your request: PyAutoHands#274 — CI green (3/3) — merge / amend / reject.

+
+
+

Where to look

+

PyAutoHands#272 (plan + parked checkpoint) · the branch

+
+
+

Your review

+
+ + + + +
+ +
+

Est. 20 review-minutes · judge

+
+
+ +
+
+
+

PyAutoFit · bug · supervised · parked · phase split

+

MultiStartGradient searches ignore iterations_per_quick_update

+

PyAutoFit#1556 (from the parked branch) · issue #1552

+
+
+ PR green · 4/4 checks + +
+
+
+
+

Question this member was dispatched to answer

+

Wire real quick updates into the multi-start step loop (the cadence is currently inert — announced in the log, never fired), honest startup message, EXEMPT-map removal, workspace AST guard.

+
+
+

Witness

+

One fit.png per cadence boundary (perform_quick_update call-count test); EXEMPT entry removed; workspace AST guard. The workspace half was declared phase 2 by the member — library first.

+
+
+

Health evidence

+

Branch feature/multistart-quick-update-wiring (head 94a658b) pushed; no PR — parked at ship (supervised) on PyAutoFit#1552 with the plan and unit decision (count per gradient step, not per batched evaluation). Session reports 2334 tests passing. Verified at collect: branch present, no PR from it.

+
+
+

Readout

+

Interacts with #1555: that PR adds a quick_update_count == 0 inertness tripwire this wiring change must flip to a “fired” assertion. Merge order: #1554 → #1555 → this branch’s PR (rebased over both).

+
+
+

Ruling

+

Approve opening the PR (merged last of the PyAutoFit trio, rebased) / request changes on #1552 / reject. The workspace phase-2 prompt goes to the queue either way.

+
+
+

Where to look

+

PyAutoFit#1552 (plan + park) · the branch

+
+
+

Your review

+
+ + + + +
+ +
+

Est. 3 review-minutes · glance

+
+
+ +
+
+
+

euclid_strong_lens_modeling_pipeline · docs · supervised · parked

+

Euclid phase 3a — narrative prose restored, start_here.py end-to-end guide

+

euclid#48 (from the parked branch) · issue #47

+
+
+ PR green · 9/9 checks + +
+
+
+
+

Question this member was dispatched to answer

+

Restore the narrative lost at 355b309: start_here.py 63-line shim → documented end-to-end guide (keeping the delegation to initial_lens_model.fit), scripts/ as chapters, drift-verified prose, plus the batch-review addendum (README structural edits, scripts/tools/ move, AGENTS.md shortening).

+
+
+

Health evidence

+

Branch feature/restore-narrative-prose pushed, no PR — parked on euclid#47. Member-reported: pytest 61 passed (fast+slow), smoke 9/9 PASS incl. the moved tools path, prose-only AST proof (string-stripped ASTs identical to main except the forced move/path fixes), all drift greps zero, no empty section headers. Caveat: the independent adversary leg was NOT run (self-review only) — by doctrine a self-run leg is an absent leg; this park IS the review, so judge accordingly.

+
+
+

Readout

+

start_here.py 0 → 24 sections (~630 prose lines); initial_lens_model __Source Pix__ filled (9 sections); full_model 10 → 26, lens_model_waveband 0 → 22, mge_lens_only 0 → 27, sersic 2 → 9, simulator +3 filled headers; 6 catalogue producers 5–9 sections each; README addendum edits done; AGENTS.md 471 → 251 lines. Three judgment calls flagged in the park comment: (1) AGENTS.md shortening treated as in scope; (2) a batch_size=50 kwarg silently swallowed by af.Nautilus — prose corrected, bug filed as draft/bug/euclid/vis_lp_batch_size_kwarg_silently_ignored.md; (3) two small doc follow-ups surfaced, not acted on.

+
+
+

Ruling

+

PR opened at your request: euclid#48 — merge / amend / reject; rule also on judgment call 1 (AGENTS.md scope). Adversary leg was not run — weight your read accordingly.

+
+
+

Where to look

+

euclid#47 (plan + park with full evidence) · the branch

+
+
+

Your review

+
+ + + + +
+ +
+

Est. 30 review-minutes · judge

+
+
+ +
+
+
+

autolens_profiling · InferenceRefs_v1 · from jax-inference-phase1-refs

+

mge_pos reference — reuse the legacy row, or run it again?

+

340210_9 (2026-08-25) · array task 9, deliberately excluded from 342091

+
+
+ Ruling required + +
+
+
+ +
+

Question

+

Adopt the 340210_9 legacy row as the mge_pos_fp64 InferenceRefs_v1 reference? The 2026-08-31 rewind's MGE reuse rule put this run on the reusable side of the quarantine (output/legacy/, not output/legacy_wrong/), so task 9 was held back from the 342091 array rather than resubmitted. Nothing has been adopted; the row is presented for your call.

+
+ +
+

Witness — refs spec vs the row

+
+ + + + + + + + + + + + +
InferenceRefs_v1 reference standard against 340210_9
Spec itemRequired340210_9Match
n_live≥ 2× fiducial (200) = 400400yes
seed00yes
config namehpc_a100_fp64_refhpc_a100_fp64_refyes
positionsononyes
thresholdautoauto (collapses to the 0.2 floor)yes
factor1e81e8yes
priors_refcurrent _targets.py_targets.py@7f2f14765de7 (= current)yes
+
+

Every line of the standard is met — this is an exact match, not a near miss. It is the only one of the three rulings on this page where that is true.

+
+ +
+

Health evidence

+

.completed marker present; PyAuto version stamp 2026.8.17.1; sampler_wall 939.09 s, consistent with this cell's recorded cost (17:07 elapsed job time) and well inside budget; priors_ref resolves to the current _targets.py hash, so the row is not behind a target-stack boundary. RAL copy and the laptop mirror are identical.

+
+ +
+

Readout

+
+ + + + + + + + + + + +
340210_9 — mge, positions on, tauto 0.2, factor 1e8
MetricValue
maxLL31,786.7965
logZ31,690.4174
evals96,704
Kish ESS7,586.1
sampler_wall939.09 s
target_idsha256:6b93f0e52ecd
+
+
+ +
+

Ruling

+

Accept → cite the row, mv its run dir back out of output/legacy/ into the active tree, and adopt it into results/baselines/InferenceRefs_v1/mge_pos_fp64/. Reject → queue array task 9 fresh (one more A100 task on top of the running ten).

+
+ +
+

Your review

+
+ + + + +
+ +
+ +
+

Where to look yourself

+
+
Local mirror (run dir)
+
/mnt/c/Users/Jammy/Science/inference_programme/output/legacy/searches/nautilus/imaging/mge/hst/pos_tauto0.2_f1e8/hpc_a100_fp64_ref_pos_tauto0.2_f1e8/dc42087fb0524e78fd43eced7706b365/
+
RAL (identical)
+
/mnt/ral/jnightin/autolens_profiling/output/legacy/searches/nautilus/imaging/mge/hst/pos_tauto0.2_f1e8/…/dc42087fb0524e78fd43eced7706b365/
+
Repo artifact
+
autolens_profiling/results/searches/nautilus/imaging/mge/hst/hpc_hpc_a100_fp64_ref_pos_tauto0.2_f1e8.json
+
Ruling lands in
+
…/results/baselines/InferenceRefs_v1/mge_pos_fp64/ (+ INDEX.json / INDEX.md, and the SUBMIT_LIST.md row)
+
Dispatch record
+
autolens_profiling#201 · PR #202 · SUBMIT_LIST.md § “2026-08-31 — Phase 1 redo dispatch”
+
+
+ +

Est. 5 review-minutes · judge

+
+
+ +
+
+
+

autolens_profiling · InferenceRefs_v1 · from jax-inference-phase1-refs

+

mge_fp64 retro baseline — off-spec under the redo standard

+

results/baselines/InferenceRefs_v1/mge_fp64/

+
+
+ Ruling required + +
+
+
+ +
+

Question

+

Does the retro-adopted mge_fp64 baseline still stand once the redo standard is applied to it? It was adopted retrospectively from an existing run, not produced by the refs array, and it does not meet the standard the ten fresh rows are being held to.

+
+ +
+

Witness — where it is off-spec

+
+ + + + + + + + + +
Refs standard vs the adopted mge_fp64 source run
Spec itemRequiredAdopted runMatch
n_live400200no
seed0nullno
config namehpc_a100_fp64_refhpc_a100_fp64no
priors_refcurrent _targets.py_targets.py@bf2c8742c334 (STALE)no
+
+

The stale priors_ref is the load-bearing one: the baseline's target.json pins a _targets.py that is no longer the current definition, so anything scored against it is scored across a target-stack boundary.

+
+ +
+

Health evidence

+

The run itself is healthy and on the reusable side of the quarantine — source run mge/hst/hpc_a100_fp64/181b13114ba3c2298191185ff74f90d8, now under output/legacy/. .completed present, version 2026.8.17.1. This is a spec question, not a soundness question.

+
+ +
+

Readout

+
+ + + + + + + + + + +
mge_fp64 adopted baseline — metadata
MetricValue
maxLL31,786.6288
logZ31,690.4965
evals62,208
sampler_wall706.50 s
version2026.8.17.1
+
+

For scale: the positions-on row above (340210_9) reaches maxLL 31,786.7965 on the same cell — the two agree to ~0.17 nats despite the n_live and seed differences.

+
+ +
+

Ruling

+

Keep as-is — accept the spec deviation and re-cite the baseline from legacy/, recording the deviation in the registry. Or refresh — queue a fresh mge positions-off hpc_a100_fp64_ref run, which requires adding a new array task: no vehicle for this row exists in the submit script today.

+
+ +
+

Your review

+
+ + + + +
+ +
+ +
+

Where to look yourself

+
+
Baseline
+
autolens_profiling/results/baselines/InferenceRefs_v1/mge_fp64/ (reference.json + target.json + README.md)
+
Local mirror (source run)
+
/mnt/c/Users/Jammy/Science/inference_programme/output/legacy/searches/nautilus/imaging/mge/hst/hpc_a100_fp64/181b13114ba3c2298191185ff74f90d8/
+
Vehicle question
+
…/hpc/batch_gpu/submit_search_nautilus_inference_refs_v1_array.sh — the 11 tasks are 0–10; an mge positions-off ref row is not one of them
+
Dispatch record
+
autolens_profiling#201 · PR #202
+
+
+ +

Est. 4 review-minutes · judge

+
+
+ +
+
+
+

autolens_profiling · InferenceRefs_v1 · from jax-inference-phase1-refs

+

delaunay_fp64 retro baseline — rests on rejected evidence

+

results/baselines/InferenceRefs_v1/delaunay_fp64/

+
+
+ Ruling required + +
+
+
+ +
+

Question

+

The delaunay_fp64 baseline is off-spec and its source run was quarantined to output/legacy_wrong/ by the rewind — the never-cite side. It therefore rests on evidence the rewind has already rejected. There is no array task for a delaunay positions-off reference run, so “just re-run it” is not available without changing the submit script.

+
+ +
+

Witness — two independent problems

+
+ + + + + + + + + +
Refs standard vs the adopted delaunay_fp64 source run
Spec itemRequiredAdopted runMatch
n_live300 (2× this cell's fiducial)150no
seed0nullno
priors_refcurrent _targets.py_targets.py@bf2c8742c334 (STALE)no
quarantine sidelegacy/ (reusable)legacy_wrong/ (NEVER CITE)no
+
+

The last row is the one that cannot be waived by accepting a deviation: the rewind's whole point is that legacy_wrong/ evidence is not cited again.

+
+ +
+

Readout

+
+ + + + + + + + +
delaunay_fp64 adopted baseline — metadata
MetricValue
maxLL30,623.1668
logZ30,562.2402
sampler_wall1,891.32 s
+
+
+ +
+

Task-10 footnote — what the doctrine call costs

+

The sibling row 340210_10 (delaunay_pos_fp64) COMPLETED and is spec-conformant — n_live 300, seed 0, priors_ref current, maxLL 31,338.9667, logZ 31,264.8427, wall 3,206.5 s. It is nevertheless doctrine-blocked, because it too sits in legacy_wrong/. A fresh run was therefore dispatched as 342091_10, spending ~53 minutes of A100 time to reproduce a row that already exists in a form that would pass every check. Recorded here so the cost of the never-cite rule is visible and can be revisited deliberately rather than by accident.

+
+ +
+

Ruling

+

Queue a fresh run — add array task 11 (delaunay, positions off, hpc_a100_fp64_ref, n_live 300, seed 0) and submit it alongside the wave. Or defer — leave the baseline flagged and revisit once the 342091 wave lands, when the machine is free and the delaunay cells' true cost is measured.

+
+ +
+

Your review

+
+ + + + +
+ +
+ +
+

Where to look yourself

+
+
Baseline
+
autolens_profiling/results/baselines/InferenceRefs_v1/delaunay_fp64/
+
Local mirror (source run, never-cite side)
+
/mnt/c/Users/Jammy/Science/inference_programme/output/legacy_wrong/searches/nautilus/imaging/delaunay/hst/hpc_a100_fp64/
+
Task-10 row (also legacy_wrong)
+
…/output/legacy_wrong/searches/nautilus/imaging/delaunay/hst/ — 340210_10, superseded in flight by 342091_10
+
Vehicle question
+
…/hpc/batch_gpu/submit_search_nautilus_inference_refs_v1_array.sh — adding task 11 is the change a “queue it” ruling implies
+
Dispatch record
+
autolens_profiling#201 · PR #202
+
+
+ +

Est. 4 review-minutes · judge

+
+
+ +
+
+
+

PyAutoArray · research · verdict awaiting ruling

+

numba solve vs JAX sparse operator

+

PyAutoArray#513 · Mind ledger 7385b138

+
+
+ Verdict delivered + +
+
+
+
+

Question this member was dispatched to answer

+

Is the numba CPU positive-only solve the same linear algebra as the JAX sparse-operator mode — and if not, is it GPU-portable or intrinsically CPU-suited? (Also stands in for the numba-cpu-likelihood epic’s next step.)

+
+
+

Readout — the verdict

+

Same linear algebra, deliberately different machines — keep both, port nothing. Identical formulation (regularised normal equations, W = HᵀN⁻¹H); every difference is hardware-deliberate: explicit 177 MB sparse preload + branchy active-set fnnls + cross-eval warm-start memo (CPU cache/state levers) vs matrix-free FFT operator + fixed-shape PDIP with differentiable relaxed-KKT VJP (GPU/vmap/grad levers). Each transplant was already measured and failed (warm-start hurts PDIP 17→38 iters; BPP/ADMM slower/fail). Two scholar-mode intake candidates offered (a design note recording the two-modes decision; landing/retiring the ledger’s pending A100 rows).

+
+
+

Health evidence

+

Full verdict posted as a comment on PyAutoArray#513 (verified at collect); no PR by design (no library changes); Mind ledger merged. Judge tier — parks for this ruling, no decide-and-flag.

+
+
+

Ruling

+

Accept the verdict (issue closes, prompt retires) / reject with reasoning; separately: /intake either scholar-mode candidate?

+
+
+

Where to look

+

PyAutoArray#513 — the verdict comment is the deliverable.

+
+
+

Your review

+
+ + + + +
+ +
+

Est. 20 review-minutes · judge

+
+
+ +
+
+
+

PyAutoNerves · feature · delivered

+

Silence the non-Colab setup_colab message on CLI runs

+

PyAutoNerves#157 · 3/3 checks green · mergeable clean

+
+
+ PR green + +
+
+
+
+

Question this member was dispatched to answer

+

Plain CLI runs should not print the “You are not running in a Google Colab environment” block; notebooks keep it.

+
+
+

Witness

+

CLI run prints nothing; notebook path still surfaces it; asserted in test_setup_colab.py. Witness test present and green in CI (context-detection via live IPython shell; silent path demoted to logger.debug).

+
+
+

Health evidence

+

PyAutoNerves#157: 3/3 checks success (unittest 3.12/3.13/nojax), mergeable clean, 169/169 tests, adversary leg reported CLEAN (independent model, report in PR conversation). Verified against GitHub at collect.

+
+
+

Ruling

+

Merge (/prm 157) / tweak / reject. notify tier — witness-covered.

+
+
+

Where to look

+

PyAutoNerves#157

+
+
+

Your review

+
+ + + + +
+ +
+

Est. 0 review-minutes · notify

+
+
+ +
+
+
+

PyAutoMemory · bug · delivered

+

Repair the queue-automation filing gate

+

PyAutoMemory#76 · validate green · mergeable clean

+
+
+ PR green · decision-taken + +
+
+
+
+

Question this member was dispatched to answer

+

queue_filing.yml red on every run since 2026-08-24; queue_actions.yml lost commits on push conflict and failed silently.

+
+
+

Witness

+

Post-merge only: re-apply queue-intake to #71 → green run + filing PR. Cannot execute pre-merge (label-triggered workflows run from main) — basis-cited instead (local red/green repro of the 30 theme-import failures; live-log confirmation of the diagnosed death step).

+
+
+

Health evidence

+

PyAutoMemory#76: validate green, mergeable clean. Three fixes (missing PyAutoBrain checkout in the gate; conflict-proof rebuild-retry replacing the discarding rebase; failure reporting). Adversary found 2 real issues, both fixed in b5bc248; finding 3 (same retry shape in two sibling workflows) recorded as follow-up. decide-and-flag: opened the PR rather than park (supervised cap) — the one flagged decision.

+
+
+

Ruling

+

Merge, then re-drive #71 and #72 (label off/on) and #69 — the witness completes post-merge (checklist in the PR). Queue the finding-3 follow-up?

+
+
+

Where to look

+

PyAutoMemory#76 (post-merge checklist at the bottom)

+
+
+

Your review

+
+ + + + +
+ +
+

Est. 0 review-minutes · notify

+
+
+ +
+
+
+

PyAutoFit · bug · delivered · merge 1st of trio

+

search.summary: Resampling Info section at the bottom

+

PyAutoFit#1554 · 4/4 checks green · mergeable clean

+
+
+ PR green · decision-taken + +
+
+
+
+

Question this member was dispatched to answer

+

Move the resampling diagnostics from mid-file to a closing “Resampling Info” section of search.summary.

+
+
+

Witness

+

File ends: blank line after Visualization time → subheader → six lines. Observed on a real smoke run and asserted by a new ordering test; searches without counters write byte-identical files (differential check vs main).

+
+
+

Health evidence

+

PyAutoFit#1554: 4/4 checks success, mergeable clean; 2328 passed/44 skipped; adversary CLEAN. decide-and-flag: opened the PR rather than park (bug-cap supervised).

+
+
+

Ruling

+

Merge first of the three PyAutoFit PRs (/prm 1554).

+
+
+

Where to look

+

PyAutoFit#1554

+
+
+

Your review

+
+ + + + +
+ +
+

Est. 3 review-minutes · glance

+
+
+ +
+
+
+

PyAutoFit · bug · delivered · merge 2nd of trio

+

MultiStartProdigy “stops at 49 steps” — root cause reframed

+

PyAutoFit#1555 · 4/4 checks green · mergeable clean

+
+
+ PR green · reframed + +
+
+
+
+

Question this member was dispatched to answer

+

MultiStartProdigy with iterations_per_quick_update=50 reportedly terminates at 49 steps.

+
+
+

Readout — the bug as reported does not exist

+

The search cannot stop at a quick-update boundary — the cadence is inert for this family (the sibling member’s EXEMPT entry). The “49” was Total Samples = 49 (1 best + 48 per-start final points, fixed at construction) misread as a step count, while the run’s real length appeared nowhere in the summary. Fix: the summary now labels the composition and reports Total Steps and Stop Reason; a new e2e regression runs a cadence mid-budget to completion and carries a quick_update_count == 0 tripwire the sibling wiring change must flip.

+
+
+

Health evidence

+

PyAutoFit#1555: 4/4 checks success, mergeable clean; witness reproduced at the reported configuration; adversary first pass found 2 issues (vacuously-green e2e; composition label false on reload paths), both fixed, re-review CLEAN.

+
+
+

Ruling

+

Accept the reframed root cause and merge second (/prm 1555).

+
+
+

Where to look

+

PyAutoFit#1555 — the investigation is on #1553.

+
+
+

Your review

+
+ + + + +
+ +
+

Est. 3 review-minutes · glance

+
+
+ +
+
+
+

six organ repos · feature · delivered

+

GitHub Page link in every organ board

+

5 PRs: brain#326 · heart#194 · hands#273 · memory#77 · scientist#25

+
+
+ All PRs green + +
+
+
+
+

Question this member was dispatched to answer

+

Add a GitHub Page link to each of the six generated organ board HTML pages (witness: link asserted in each generator’s board test).

+
+
+

Status

+

Finished 20:51Z, 5 PRs green: PyAutoBrain#326, PyAutoHeart#194, PyAutoHands#273, PyAutoMemory#77, PyAutoScientist#25. Merge/tweak/reject as one ruling — six repos, one witness.

+
+
+

Your review

+
+ + + + +
+ +
+

Est. 3 review-minutes · glance

+
+
+ +
+
+
+

subhalo_validation · research · LOCAL-DEV

+

Subhalo follow-up wave — adapt_split fix + RectangularBilinear

+

draft/research/subhalo_validation/follow_up_wave_adapt_split_and_rectangular.md

+
+
+ REFRESHED 14:00 BST — both pl_eff witnesses landed CLEAN · rectangular chain still running · carried +
+
+
+ +
+

Question this member was dispatched to answer

+

Three, from the AM review’s accepted follow-up:

+
    +
  1. The one that matters. Does AdaptSplit at source_pix[1] keep the compact outer source component that ConstantSplit appeared to smooth away, on pl_eff_1_outer? Nothing downstream moves until this is judged — it decides whether pl_eff_1_outer gets a full rerun and whether the default recipe’s source_pix[1] regularization changes.
    2. +
  2. Does the rectangular_adapt recipe run end-to-end on pl_sersic_0 (the mesh/regularization comparison column)?
    3. +
  3. Do the two pl_eff delaunay chains produce clean witnesses — which would unblock the parked second rectangular_adapt submission?
    4. +
+
+ +
+

Witness

+

Validation witness — evidence_increase in results/<pipeline>/<lens>_no_subhalo.json. The data contains no subhalo, so a clean null is < 5; anything at or above 5 would conventionally read as a detection and would be a false positive.

+

AdaptSplit witness — the rerun’s source_pix[1]/model.info must actually read AdaptSplit (N=3) (not the baseline’s ConstantSplit (N=1)), and its Bayesian evidence is then compared like-for-like against the baseline stage: same lens, same 600-pixel Delaunay mesh, same areas_factor 0.5, same 10 free parameters, same dataset. Only the regularization differs.

+
+ +
+

Health evidence — read 2026-09-01 13:55–13:58 BST

+

Batch .out logs on this project are block-buffered and flush on exit, so none of the states below come from a log tail: they come from sacct, from each job’s own closing lines after exit, from the witness JSONs, and from a live checkpoint.hdf5 mtime.

+
    +
  • 342027_1 (pl_eff_0) — COMPLETED 0:0 in 1-21:10:24, ended 2026-09-01 00:04:55 BST, about 2 h 50 m inside the 2-day wall. Its log closes Search complete, returning result → Evidence Increase: -0.24247446289882646 → Results written to: …/pl_eff_0_no_subhalo.json → Finished sample: pl_eff_0. error.342027_1.err is 0 bytes.
    • +
  • 342027_2 (pl_eff_1_outer) — COMPLETED 0:0 in 1-23:05:05, ended 01:59:36 BST — 55 minutes short of the 2-day wall. Same closing sequence, Evidence Increase: 2.7360672678223636, witness written, error.342027_2.err 0 bytes.
    • +
  • The wall-clock flag raised at 22:57Z did not materialise. Both chains wrote their witness JSONs; no chain was cut off, nothing needs a resume-from-checkpoint resubmit.
    • +
  • 342095_0 — still RUNNING (14:03:13 elapsed of 2-00:00:00 at the read) on euclid-ral-compute-10-2. It is past all five pixelized stages and now inside subhalo[1]_[search_lens_plane]: two of the grid cells are finished and zipped, a third started at 10:52 BST, and its checkpoint.hdf5 (22.5 MB) carries a 13:57 BST mtime — current at the moment of reading. Sampling, not hung. error.342095_0.err is 0 bytes.
    • +
  • 342093_0 / 342094_0 — unchanged since the 22:57Z refresh; both COMPLETED 0:0 with their evidence as recorded below.
    • +
+

One log-level flag, benign but repeated: 342095’s log emits latent.yaml lists 'total_galaxy_0_flux' but no such latent is registered; dropping on every update. It costs nothing in this run — the other eight latents register — but the recipe’s latent.yaml is asking for a latent that does not exist.

+
+ +
+

Readout — the three actions, RAL states read 2026-09-01 13:55 BST

+
+ + + + + + + + + + +
Jobs dispatched by this member (sacct state is not health evidence; the right-hand column is)
JobWhat it runssacct stateElapsed / wallEvidence
342093_0pl_eff_1_outer source_pix[1] under AdaptSplit — recipe delaunay_adapt_split_fix, --stop_after source_pix_1COMPLETED 0:001:16:49 / 08:00:00.completed · 33,180 samples · acceptance 1.0 · model.info reads AdaptSplit (N=3) · all figures written
342094_0pl_sersic_0 / rectangular_adapt job A — source_lp[1], JAX, 4 coresCOMPLETED 0:001:44:40 / 12:00:00ended 2026-08-31 23:52:17 BST; log closes Finished sample: pl_sersic_0
342095_0pl_sersic_0 / rectangular_adapt job B — the rest of the chain, 8 cores 96 gb, afterok:342094RUNNING14:03:13 / 2-00:00:00five stages done and zipped; inside subhalo[1] grid (2 of 4 cells zipped, 3rd started 10:52 BST); checkpoint 22.5 MB mtime 13:57 BST
342027_1pl_eff_0 — original delaunay chain, through subhalo[2]_[single_plane_refine]COMPLETED 0:01-21:10:24 / 2-00:00:00ended 00:04:55 BST; witness pl_eff_0_no_subhalo.json written; .err empty
342027_2pl_eff_1_outer — original delaunay chain, through subhalo[2]COMPLETED 0:01-23:05:05 / 2-00:00:00ended 01:59:36 BST, 55 min short of wall; witness pl_eff_1_outer_no_subhalo.json written; .err empty
+
+
+ +
+

Readout — witness JSONs on the mirror after this hpc/sync pull

+
+ + + + + + + + + +
results/ after the pull — three witnesses now, two of them new overnight; a clean null is evidence_increase < 5
Witness fileevidence_increaselog_likelihood_increasesubhalo M200 / centreVerdict
delaunay_adapt_split/pl_sersic_0_no_subhalo.json−0.3009+0.96681.49e9 · (2.146, −0.241)CLEAN — already accepted 2026-08-31, not re-litigated here
delaunay_adapt_split/pl_eff_0_no_subhalo.json−0.2425+0.60931.21e8 · (−1.302, −2.513)CLEAN — new, written 00:04:55 BST by 342027_1
delaunay_adapt_split/pl_eff_1_outer_no_subhalo.json+2.7361+15.37168.20e8 · (1.412, 1.419)CLEAN (< 5) — new, written 01:59:36 BST by 342027_2; the largest of the three and the one to read carefully
rectangular_adapt/pl_sersic_0_no_subhalo.json———not written yet — 342095_0 is inside subhalo[1]
+
+

All three witnesses are nulls, as the data demands — no false positive anywhere in the delaunay column. Read pl_eff_1_outer with its two numbers together: the subhalo model buys +15.37 in maximum log likelihood but only +2.74 in evidence, i.e. the extra parameters very nearly pay for themselves at an 8.2e8 M⊙ perturber sitting 2.0″ from centre. It is comfortably under the conventional 5, but it is the one row where the null is not emphatic.

+

Note, as before, that none of the three JSONs carries the subhalo_stage key — all three chains predate it, so every number above is subhalo[2]/refine-derived and is not directly comparable with a post-key run. The 342093_0 rerun writes no witness by design (--stop_after source_pix_1).

+
+ +
+

Readout — does rectangular_adapt run end-to-end? (question 2)

+

On the evidence so far, yes through the pixelized stages: 342094+342095 have carried pl_sersic_0 through light[1], mass_total[1], source_lp[1], source_pix[1] and source_pix[2] — all five present and zipped on the mirror — and the run is now inside the subhalo[1] lens-plane grid. source_pix[2]/model.info reads RectangularBilinearAdaptImage (N=2) with Adapt (N=3) regularization, which is the recipe under test.

+
+ + + + + + + + + + +
source_pix[2] on pl_sersic_0, the mesh comparison column — indicative only, not like-for-like (different mesh size and free-parameter count)
Quantitydelaunay_adapt_splitrectangular_adaptDifference
Mesh / regularizationDelaunay, 1,310 px · AdaptSplit (N=3)RectangularBilinearAdaptImage 32×32 = 1,024 px · Adapt (N=3)different mesh and pixel count
Free parameters35+2
Bayesian evidence21,386.82321,418.501+31.678
Maximum log likelihood21,399.31121,435.853+36.542
Samples / runtime2,060 · 0:07:473,140 · 0:10:07+52% samples, +30% wall
+
+

The rectangular column is ahead at this stage, but the two stages differ in more than the mesh, so this is a progress readout rather than a mesh verdict — the comparable number is the witness, which lands when 342095 finishes its grid and refine (about a day of wall left).

+
+ +
+

Readout — 342093, AdaptSplit at source_pix[1] (the thing to look at)

+
+ + + + + + + + + + + + +
Like-for-like: same lens, same 600-pixel Delaunay mesh, same 10 free parameters, same dataset — only the regularization differs
QuantityBEFORE — baseline source_pix[1]AFTER — 342093 rerunChange
RegularizationConstantSplit (N=1)AdaptSplit (N=3)the fix, confirmed in model.info
Bayesian evidence18,312.47319,390.673+1,078.200
Maximum log likelihood18,377.14219,462.741+1,085.599
Regularization coefficientscoefficient 1.321 (flat everywhere)inner 1.0e−4 · outer 79.76 (3σ 44.3–113.6) · signal_scale 0.0164flat → spatially varying, 0.02 at the core to 23.19 in the outskirts
Einstein radius1.9261.9245 (3σ 1.9242–1.9247)unmoved
Chi-squared map max54.6963.20+8.5 — see the residual caveat
Samples / runtime22,580 · 0:43:0733,180 · 1:14:43+47% samples, +73% wall
+
+ +
+ Source-plane regularization map, ConstantSplit versus AdaptSplit +
The mechanism, in one panel. BEFORE: a single flat coefficient 1.32 over the whole source plane. AFTER: AdaptSplit drops the coefficient to 0.02 on the bright source core (dark blue) and raises it to 23.19 in the empty outskirts (red) — it stops smoothing where there is signal and smooths hard where there is none. This is what the +1,078 nats of evidence buys.
+
+ +
+ Source-plane reconstruction, maximum zoom, ConstantSplit versus AdaptSplit +
Source Plane (Zoom), same colour scale (0 – ~6.05 e⁻ s⁻¹). The compact source is not smoothed away by AdaptSplit — the bright core survives and the secondary knot up and to the left of it is rendered as a distinct, higher-contrast feature rather than the softer shoulder ConstantSplit gives it.
+
+ +
+ Source-plane reconstruction, mid zoom with tangential caustic, ConstantSplit versus AdaptSplit +
Source Plane (Mid Zoom) with the tangential caustic drawn in white — the panel that answers the question. Under AdaptSplit the emission separates into a bright peak inside the caustic plus a resolved knot straddling and extending past its upper-left edge; under ConstantSplit that same region is one smoother blob with a low-contrast extension. Whether that knot is the seventh (outer) component of the ElsonFreeFall source or internal structure of the main clump is the judgement being asked for — the truth positions are in dataset/pl_eff_1_outer/no_subhalo/positions_outer.json.
+
+ +
+ Normalized residual map, ConstantSplit versus AdaptSplit +
The honest caveat. Coherent arc-shaped residuals along the ring persist in both fits at comparable amplitude (colour range ±7.40 before, ±7.95 after; chi-squared max 54.69 → 63.20). The +1,078 nats therefore come from the adaptive smoothing prior fitting the source better, not from the ring residual structure being resolved away. If those arcs are the reason for suspecting the outer component was lost, AdaptSplit has not removed them.
+
+
+ +
+

Decision waiting on you — “submit rectangular for which pl_eff?”

+

The gate has opened overnight, and nothing was done about it. The parked second rectangular_adapt submission was gated on a clean evidence_increase < 5 witness from 342027_1 (pl_eff_0) or 342027_2 (pl_eff_1_outer). Both witnesses now exist and both are clean (−0.24 and +2.74). The choice of which pl_eff to submit is therefore live — and it is yours.

+

The command recorded in the project’s wiki/project/state.md is PIPELINE=rectangular_adapt ARRAY=<1|2> bash hpc/run_chain.sh (ARRAY: 0 pl_sersic_0, 1 pl_eff_0, 2 pl_eff_1_outer). It has not been run. This refresh is read-and-record only: no RAL job was submitted, cancelled or modified, and the parked submission stays PARKED until you rule. Worth weighing when you do: the one CPU chain already running (342095) still has about a day of wall to go on the same node.

+

Carried automatically, as before: this member stays in flight (one job running) and enters the next batch’s packet without re-specification. The 342093 AdaptSplit readout and both pl_eff witnesses are, however, complete and can be judged now.

+
+ +
+

Follow-ups this refresh proposes

+
    +
  • [subhalo_validation] The wall-clock risk closed itself — both 342027 chains finished inside the 2-day wall (the tighter one by 55 minutes) and wrote their witnesses, so no resume-from-checkpoint resubmit is needed. The margin was thin enough to be worth remembering when sizing the next delaunay chain.
    • +
  • [subhalo_validation] Both pl_eff witnesses predate the subhalo_stage JSON key, so their numbers are refine-derived and not directly comparable with post-key runs. Decide whether the three delaunay witnesses get re-stamped or are simply annotated in the ledger.
    • +
  • [subhalo_validation] pl_eff_1_outer’s +15.37 log-likelihood increase against a +2.74 evidence increase is the least emphatic null in the set — worth a look at where its 8.2e8 M⊙ candidate sits relative to the arc before the recipe is called settled.
    • +
  • [subhalo_validation] latent.yaml lists total_galaxy_0_flux, which is not a registered latent; every update logs a drop warning. Harmless, but it should either be registered or removed.
    • +
  • [subhalo_validation] If the AdaptSplit reconstruction is accepted, the open question is whether source_pix[1]’s regularization changes in the default delaunay_adapt_split recipe, or stays a _fix variant — and whether pl_eff_1_outer earns a full rerun on it.
    • +
  • [subhalo_validation] The persisting coherent ring residuals in both fits are unexplained by the regularization change and may deserve their own look (mass model? PSF? over-sampling at pix[1]?).
    • +
+
+ +
+

Where to look yourself

+
+
Project root (laptop mirror)
+
/mnt/c/Users/Jammy/Science/subhalo_validation/ — source activate.sh first; hpc/sync jobs / hpc/sync pull drive RAL.
+
The two new witnesses
+
results/delaunay_adapt_split/pl_eff_0_no_subhalo.json and results/delaunay_adapt_split/pl_eff_1_outer_no_subhalo.json — both pulled to the mirror by this refresh, alongside the pl_sersic_0 witness accepted yesterday.
+
Their chains, end to end
+
hpc/batch_cpu/output/output.342027_1.out (5.2 MB) and output.342027_2.out (5.7 MB) on the mirror — the last dozen lines carry the evidence-increase block and Finished sample:. Run directories under output/subhalo/detect/delaunay_adapt_split/pl_eff_0_no_subhalo/ and …/pl_eff_1_outer_no_subhalo/.
+
The AdaptSplit rerun, in full
+
output/subhalo/detect/delaunay_adapt_split/pl_eff_1_outer_no_subhalo_adapt_split_fix/source_pix[1]/085f85ba6970436a1b8b115b6c9b3fbf/ — model.info (reads AdaptSplit), model.results (evidence 19,390.673), search.summary, and image/fit.png + image/inversion_0_0.png at full resolution.
+
The ConstantSplit baseline it is compared against
+
output/subhalo/detect/delaunay_adapt_split/pl_eff_1_outer_no_subhalo/source_pix[1]/27660ebc6b9f9ef5e4c48d393a628efd.zip — on RAL the stage is zipped; the mirror holds only its files/galaxy_images_snr.fits, so unzip the archive (or re-pull) to open the baseline’s own fit.png.
+
The rectangular chain in flight
+
output/subhalo/detect/rectangular_adapt/pl_sersic_0_no_subhalo/ — five zipped stages plus subhalo[1]_[search_lens_plane]/…/results.csv, which fills in one row per finished grid cell. source_pix[2]/…/model.info is where the RectangularBilinearAdaptImage mesh is confirmed.
+
The four figures above, at higher resolution
+
results/figures/adapt_split_fix_regularization_before_after.jpg, ..._source_plane_before_after.jpg, ..._source_plane_midzoom_before_after.jpg, ..._residuals_before_after.jpg (1000 px wide, committed to the project).
+
Live job state
+
hpc/sync jobs, or ssh euclid_jump then sacct -j 342027,342093,342094,342095. Do not read the batch .out logs for progress — they are block-buffered and flush only on exit.
+
Project ledgers
+
wiki/project/state.md (head pointer) and wiki/project/2026-08-31-adapt-split-fix-and-rectangular.md (yesterday’s journal entry).
+
+
+ +

Est. 15 review-minutes · judge

+
+
+ +
+
+
+

autolens_profiling · epic-slice · LOCAL-DEV

+

JAX inference Phase 1 redo — InferenceRefs_v1 baselines

+

PROGRAMME.md rewind restart · queue anchor autolens_profiling#200

+
+
+ WAVE COMPLETE — 10/10 tasks delivered clean · carried for ruling +
+
+
+
+

Question this member was dispatched to answer

+

Certify the InferenceRefs_v1 rows: legacy-MGE reuse rulings first (surfaced as packet members), fresh RAL submissions for missing/rejected rows (incl. re-queued refs 5/6/7). Phase 2 stays queued behind these baselines.

+
+
+

Status — RAL job 342091, read 2026-09-01 12:55Z

+

Dispatched from the laptop 21:08Z: RAL job 342091 (sbatch --array=0-8,10 --requeue), config hpc_a100_fp64_ref, A100 80 GB, n_live 300 (2× fiducial), 6:00:00 wall per task. The array ran one task at a time on the single GPU and all ten finished overnight: first task started 2026-08-31 22:00:39 BST, last ended 2026-09-01 08:08:54 BST — 10 h 08 m of continuous GPU, every task COMPLETED 0:0 on euclid-ral-gpu-2, no requeue, no preemption, the queue now empty for this project. Task 9 (mge_pos) was held back under the MGE reuse rule — its ruling and the two retro-baseline rulings are the three members above. Recorded on autolens_profiling#201 with docs-only PR #202. Refreshed here by hpc/sync pull to /mnt/c/Users/Jammy/Science/inference_programme/.

+
+
+

Health evidence — classified from artefacts, not from sacct

+

Every one of the ten tasks clears all five checks, so the classification below is evidence-based rather than an ExitCode 0:0 reading:

+
    +
  • Wall plausible — 00:36:21 to 01:24:16, i.e. 10–23% of the 6 h budget. Nothing near the wall, nothing suspiciously instant.
    • +
  • Result row written — ten result JSONs under results/searches/nautilus/imaging/…/hst/, each stamping version 2026.8.17.1, each with a target_id matching its intended cell.
    • +
  • .err clean — all ten error files are 222 bytes and contain only the mask_2d_util.py:564 UserWarning: Mask padded from (141, 141) to (159, 159). No Traceback in any error or output log.
    • +
  • Ended properly — every .out closes with the timing block, the results-JSON path, Finished. and a timestamp; ten .completed markers, one per run directory.
    • +
  • No resume, no overflow — Fit Already Completed appears zero times across all ten logs (every search started cold), and no log_l anywhere resembles the pre-fix 3e+303 signature. The only hit from an inf|nan|e+30 sweep is the literal string model.info.
    • +
+
+
+

Witness — per-task readout, all ten rows final

+
+ + + + + + + + + + + + + + + + +
342091 array tasks — truth log likelihood for this dataset is 31,521.744; Δ is maxLL minus truth
TaskTargetsacct stateElapsedDelivered checksmaxLL / logZΔ vs truthFlags
0pixelization, pos offCOMPLETED 0:000:36:21.err clean (mask-padding UserWarning only) · Finished. printed · .completed present · version 2026.8.17.1 · no Fit Already Completed · sampler wall 2,093.03 s (10% of 6 h)29,835.4509
29,778.0569		−1,686.29none — finite, ~3e4
1delaunay_nn, pos offCOMPLETED 0:000:52:55.err clean (mask-padding UserWarning only) · Finished. printed · .completed present · version 2026.8.17.1 · no Fit Already Completed · sampler wall 3,110.97 s (14%)30,650.8283
30,591.1280		−870.92none
2delaunay_nn, pos on (fixed 0.3, f 1e8)COMPLETED 0:000:53:03.err clean (mask-padding UserWarning only) · Finished. printed · .completed present · version 2026.8.17.1 · no Fit Already Completed · sampler wall 3,113.59 s (14%)31,351.3909
31,275.2300		−170.35positions arm confirmed: 4 truth-derived points, threshold fixed 0.300000, factor 1e+08
3slam_source_pix, pos offCOMPLETED 0:001:13:34.err clean (mask-padding UserWarning only) · Finished. printed · .completed present · version 2026.8.17.1 · no Fit Already Completed · sampler wall 4,336.11 s (20%)31,026.6354
30,951.7885		−495.11none
4slam_source_pix, pos on (auto)COMPLETED 0:001:07:56.err clean (mask-padding UserWarning only) · Finished. printed · .completed present · version 2026.8.17.1 · no Fit Already Completed · sampler wall 3,982.84 s (18%)31,547.2396
31,452.0997		+25.50the only row above the truth likelihood
5slam_source_pix_nn, pos offCOMPLETED 0:001:24:16.err clean (mask-padding UserWarning only) · Finished. printed · .completed present · version 2026.8.17.1 · no Fit Already Completed · sampler wall 4,986.87 s (23%) — longest task30,750.9893
30,684.1146		−770.75target sha256:ad291b57fc62 — the corrected AdaptSplitPower target, as intended
6slam_source_pix_nn, pos on (auto)COMPLETED 0:001:13:18.err clean (mask-padding UserWarning only) · Finished. printed · .completed present · version 2026.8.17.1 · no Fit Already Completed · sampler wall 4,330.27 s (20%)31,405.6282
31,324.8382		−116.12target sha256:8021b4b697ff — the cell that had never previously been submitted
7knn, pos offCOMPLETED 0:000:58:02.err clean (mask-padding UserWarning only) · Finished. printed · .completed present · version 2026.8.17.1 · no Fit Already Completed · sampler wall 3,415.75 s (16%)30,062.6117
29,995.6676		−1,459.13the stale row is gone — see below
8delaunay_matern, pos offCOMPLETED 0:000:55:13.err clean (mask-padding UserWarning only) · Finished. printed · .completed present · version 2026.8.17.1 · no Fit Already Completed · sampler wall 3,246.46 s (15%)30,675.9711
30,615.1603		−845.77none
10delaunay, pos on (auto)COMPLETED 0:000:53:29.err clean (mask-padding UserWarning only) · Finished. printed · .completed present · version 2026.8.17.1 · no Fit Already Completed · sampler wall 3,141.26 s (15%)31,338.4344
31,264.8345		−183.31none
9mge, pos on (auto)not submitted—deliberately excluded from the array under the MGE legacy-reuse ruling — see mge-pos-ref-reuse above
+
+

The knn row is now a real run. At the 22:53Z refresh the mirror’s knn/hst/hpc_hpc_a100_fp64_ref.json still read the pre-fix 30,077.028 — a git stash artefact, not a result. Task 7 has now replaced it with a genuine 342091 row at 30,062.612 (logZ 29,995.668), which is 14.4 nats lower still and sits ~494 nats below the Prodigy arm bar 30,557.03. The stale-row explanation is therefore closed, and the knn cell’s underperformance is real rather than an artefact — a result for the programme to absorb, not a defect in this wave.

+

Shape of the wave, offered as observation only. The four positions-on arms (tasks 2, 4, 6, 10) all land within 25–183 nats of truth, with task 4 (slam_source_pix, auto threshold) the only row above it at +25.50. The positions-off pixelized rows sit 495–1,686 nats below truth, pixelization lowest. Each pos-on log carries the arm banner — POSITIONS ARM (target_class 3), Idealised: truth-derived positions, not re-solved from a completed search, PositionsLH ACTIVE — so the arms are idealised by construction and the gap to the pos-off rows should be read with that in mind.

+

Relevant to a ruling already on this page, stated as fact and nothing more: the delaunay-fp64 retro-baseline question was framed as “add array task 11 now, or defer until the 342091 wave lands?”. The wave has landed and the GPU queue for this project is empty.

+
+
+

Where to look yourself

+
+
Local checkout
+
/home/jammy/Code/PyAutoLabs/autolens_profiling — hpc/sync jobs / hpc/sync pull drive RAL; hpc/sync.conf points pulls at the mirror below rather than at the checkout.
+
Laptop mirror (everything pulled by this refresh)
+
/mnt/c/Users/Jammy/Science/inference_programme/ — results/searches/nautilus/imaging/<cell>/hst/hpc_hpc_a100_fp64_ref*.json for the ten rows, logs/output/output.342091_<task>.out and logs/error/error.342091_<task>.err for the logs, output/searches/… for the run directories and their .completed markers.
+
The two rows worth opening first
+
knn/hst/hpc_hpc_a100_fp64_ref.json (the replaced row) and slam_source_pix/hst/hpc_hpc_a100_fp64_ref_pos_tauto0.2_f1e8.json (the one above truth).
+
Live job state
+
ssh euclid_jump then sacct -j 342091 --format=JobID,State,ExitCode,Elapsed,Start,End; squeue -u jnightin now shows nothing for this project.
+
Programme record
+
autolens_profiling#201 (queue anchor #200) and docs-only PR #202.
+
+
+
+

No ruling chip on this page

+

This member carried into the slot without a review control and keeps it that way — under carry-forward it is formally ruled in the next batch. What has changed is that there is nothing left to wait for: the wave is complete, all ten rows are certified-clean by the checks above, and the readout can be judged whenever you want to. The three legacy-reuse rulings it routed into this slot (mge-pos-ref-reuse, mge-fp64-retro-baseline, delaunay-fp64-retro-baseline) do carry their own chips and are yours to rule now.

+
+
+

Follow-ups this refresh proposes

+
    +
  • [autolens_profiling] Fold the ten certified rows into the InferenceRefs_v1 table and close out #201; the knn row now needs a note that its low value is a measured result, not the old stash artefact.
    • +
  • [autolens_profiling] Phase 2 was queued behind these baselines (ordered phases). With Phase 1 complete, it is dispatchable at the next slot.
    • +
  • [autolens_profiling] The wave used 10 h of a single A100 for 10 tasks at 10–23% of a 6 h wall each — the per-task wall can be cut substantially, which matters if Phase 2 queues a wider array.
    • +
+
+

Est. 20 review-minutes · judge

+
+
+ +
+ Packet-building notes +
+

Carry-forward adopted mid-slot at the human's direction: unfinished long-running members are reviewed in the batch where their results land, entering the next packet automatically; this packet's review therefore counts 9 members, not 11.

+

Written at dispatch (18:40Z), all members PENDING per the evening-dispatch flow; the collect regenerates member sections in place and bumps the refreshed stamp. Deviation from TEMPLATE.md: pending members carry their review controls already, so early notes persist through the refresh. Composition was human-pasted (the planner proposal was not used); constraint report and expected-effects live in batches/2026-08-31-pm.md. Heart YELLOW acknowledged in-chat at dispatch (manifest drift + web-container CI blindness). Epic resolutions at plan time: numba-cpu → deduped into the numba-vs-jax-sparse member; jax-inference → Phase 1 only, Phase 2 queued (ordered phases); euclid → phase 3a with the start_here.py ruling already resolved by the human (Mind 32c639e5).

+
+
+ +
+

Ruled 0 of 12 · decisions 0 of 12

+ +
+ + + +
+
+
+ + + + + diff --git a/batches/packets/TEMPLATE.md b/batches/packets/TEMPLATE.md index 850ee0d..66fa54e 100644 --- a/batches/packets/TEMPLATE.md +++ b/batches/packets/TEMPLATE.md @@ -71,6 +71,10 @@ The submit button assembles exactly: ``` +Any non-member section the submit assembles (an optional `### Follow-ups accepted` block +between the header and the first member) is an `###`: `check` requires every `##` in a +review file to name a member of the slot's batch record. + One `## — ` section per member, in page order; the slug is the phase file's stem (embed them in the page's `MEMBERS` JS array); `` ∈ `HEALTHY | SUSPECT | FAILED | RUNNING`. The orchestrator parses on those exact heading forms and the two `- key:` diff --git a/batches/reviews/2026-08-31-am.md b/batches/reviews/2026-08-31-am.md new file mode 100644 index 0000000..edff2f6 --- /dev/null +++ b/batches/reviews/2026-08-31-am.md @@ -0,0 +1,71 @@ +# Batch review 2026-08-31-am + +- packet: batches/packets/2026-08-31-am.html +- reviewed-at: 2026-08-31 13:34 (local) +- review-minutes-actual: (not given) + +### Follow-ups accepted +- subhalo_validation follow-up wave: source_pix[1] AdaptSplit rerun of pl_eff_1_outer ("_adapt_split_fix", for human inspection); disable subhalo[2] for future runs (subhalo[1] detection sufficient); RectangularBilinear runs for the two successful lenses. Next batch. +- Intake: is the numba positive-only-solve linear algebra the same as the JAX sparse-operator mode; if not, GPU-JAX amenable or CPU-sparsity-bound? +- euclid pipeline docs/structure tidy (README reorder + trim, WORST_* contract out of README, scripts/tools/ move, drop start_here.py, shorten AGENTS.md) — a **dev** follow-up, carried on the Mind's own review of this slot. +- Profiling members 342016/17/18: REJECTED wholesale — the restarted JAX gradient epic takes precedence; no ledger writes, no certifications. Filed here as `drop` (R-20260831-02..05). + +Transcription note (2026-09-01, birth epic phase 4): the Mind original recorded every +member as `ruled: no`, because there was no Cortex to file a ruling into — the verbs +were the human's decision and nothing more. Filing them is what phase 4 does, so the +seven members that carry a verb read `ruled: yes` here and name R-20260831-02..08. The +Mind's dev vocabulary maps one way only: `reject` → `drop`, `UNREVIEWED` → `(none)`. +The eighth Mind member, `euclid-phase2-pr46`, was a dev retrospective and is not a +science member of this slot. + +## failed_submissions_342008_10 — FAILED +- decision: (none) +- ruled: no + +(no note) + +## smc_probe_342018 — FAILED +- decision: drop +- ruled: yes + +I have updated the JAX gradient epic to do alot of phases back from the start, so wee will basically discard this and start again. Dont change anything just let that recent intake that precendent. + +## knn_rerun_342016_7 — SUSPECT +- decision: drop +- ruled: yes + +I have updated the JAX gradient epic to do alot of phases back from the start, so wee will basically discard this and start again. Dont change anything just let that recent intake that precendent. + +## refs_5_6_342016_56 — HEALTHY +- decision: drop +- ruled: yes + +I have updated the JAX gradient epic to do alot of phases back from the start, so wee will basically discard this and start again. Dont change anything just let that recent intake that precendent. + +## slogdet_ab_342017 — HEALTHY +- decision: drop +- ruled: yes + +I have updated the JAX gradient epic to do alot of phases back from the start, so wee will basically discard this and start again. Dont change anything just let that recent intake that precendent. + +## delaunay_adapt_split_pl_sersic_0 — HEALTHY +- decision: accept +- ruled: yes + +Review on numba likelihood function stuff: 1) End to end runs are good in general, with numba speed up run times roughly x8 faster than old code, incredible. Do an intake to investigate if the linear algebra approach taken for this is the same as the sparse operator mode alreasdy implemented for JAX, and if it is not ask whether it is amenable to GPU JAX speed up or if there is some sparsity exploitation that means this is suited to CPU. 2) The pl_eff_1_outer_no_subhalo task did not get a good result, the outer source galaxy messed up the solution. I think the issue is that source_pix[1] used ConstantSplit, not AdaptSplit, so can you rerun source_pix[1] on this lens with a suffix "_adapt_split_fix" and I will inspect the result. + +We dont need subhalo[2] for these currently, knowing if theres a detection in subhalo[1] is sufficent so dont run subhalo[2] for future runs. + +Lets also do a RectangularBilinear run for the two successful lenses in the next batch / task. + +## delaunay_adapt_split_pl_eff_0 — RUNNING +- decision: leave-to-finish +- ruled: yes + +(same note as delaunay_adapt_split_pl_sersic_0 — numba intake, pl_eff_1_outer AdaptSplit rerun, no subhalo[2] in future runs, RectangularBilinear runs next batch) + +## delaunay_adapt_split_pl_eff_1_outer — RUNNING +- decision: leave-to-finish +- ruled: yes + +(same note as delaunay_adapt_split_pl_sersic_0 — numba intake, pl_eff_1_outer AdaptSplit rerun, no subhalo[2] in future runs, RectangularBilinear runs next batch) diff --git a/batches/reviews/2026-08-31-pm.md b/batches/reviews/2026-08-31-pm.md new file mode 100644 index 0000000..1d6f6a7 --- /dev/null +++ b/batches/reviews/2026-08-31-pm.md @@ -0,0 +1,66 @@ +# Batch review 2026-08-31-pm + +- packet: batches/packets/2026-08-31-pm.html +- reviewed-at: 2026-09-01 15:13 (transcribed from the human's review submission) +- review-minutes-actual: (not given) + +### Follow-ups accepted +- Every mesh / pixelization run of the redo — the refs array's mesh rows included — must carry a `PositionsLH` positions penalty (positions + threshold per the workspace convention, e.g. 0.3"). MGE-source rows are unaffected and stay valid. Mesh rows produced without the penalty are not citable as references, and the "Inference Baselines v1" summary carries the same unphysical-solution issue. Filed as autolens_profiling#203. +- Clear the RAL active `output/` of all mesh results, the `image_plane`/`source_plane` point-source fits, the `Cluster` folder and the `point_source` folder — **moved, not deleted**, to a new `output/legacy_point/`, so the active tree is clean and updates with each task. Laptop action, filed as autolens_profiling#205. +- `340210_9` is adopted as the `mge_pos_fp64` reference: cite it and `mv` it from `output/legacy/searches/…` back into the active tree on RAL and on the mirror. Array task 9 stays excluded from the refs array. + +Transcription note (2026-09-01, birth epic phase 4): the science half of this slot's +review was recorded in three places and never in one — `autolens_profiling`'s +DECISIONS.md for the three ruling members, the Mind's batch record for the carried +members, and the subhalo project's wiki. This file is the single transcription; the +verbatim human words below come from the DECISIONS.md entry "2026-09-01 — Batch +2026-08-31-pm review rulings (InferenceRefs_v1 legacy-reuse members)". The nine dev +members of the Mind slot are not science members and are not sectioned here. + +## mge_pos_ref_reuse — HEALTHY +- decision: accept +- ruled: yes + +**mge-pos-ref-reuse — ACCEPTED.** The quarantined-but-reusable `340210_9` run (2026-08-25) is adopted as the `mge_pos_fp64` InferenceRefs_v1 reference. Per the MGE reuse rule: cite it and `mv` it from `output/legacy/searches/…` back into the active tree (RAL **and** mirror) — laptop action, queued for the next local-dev slot. Array task 9 stays excluded from the refs array. + +## mge_fp64_retro_baseline — HEALTHY +- decision: accept +- ruled: yes + +**mge-fp64-retro-baseline — ACCEPTED.** The retro-adopted `mge_fp64` baseline stands under the redo standard; no fresh `_ref` run needed. + +## delaunay_fp64_retro_baseline — FAILED +- decision: drop +- ruled: yes + +**delaunay-fp64-retro-baseline — REJECTED, with a programme-wide finding.** The human inspected `output/legacy_wrong/searches/nautilus/imaging/delaunay/hst/hpc_a100_fp64/` and identified the solution as a **demagnified-source unphysical solution** — the classic Inversion bias `PositionsLH` exists to remove (https://pyautolens.readthedocs.io/en/latest/general/demagnified_solutions.html). Human, verbatim: *"This is why we basically need to rerun loads of JAX gradient runs (MGE sources are not effect by this so safe) wit the position penalty set up and used. The problem is it is not used in all these JAX Gradient runs!"* + +## inference_refs_v1_redo — HEALTHY +- decision: (none) +- ruled: no + +Carried. The wave landed complete (10/10 tasks delivered clean) and the delaunay-fp64 ruling's "defer until the wave lands" branch is now decidable, but no ruling was written this slot: R-20260901-03 has just made every mesh row produced without a positions penalty spent evidence, so this wave's mesh rows have to be checked against that before any certification. The human rules it next slot. + +## delaunay_adapt_split_pl_sersic_0 — HEALTHY +- decision: (none) +- ruled: no + +Carried, already accepted this morning under R-20260831-06. Nothing new to rule; it rides in this packet because its project's other three phases do. + +## delaunay_adapt_split_pl_eff_0 — HEALTHY +- decision: (none) +- ruled: no + +Carried. Left to finish under R-20260831-07 and now finished: `evidence_increase = −0.2425` (logL +0.6093), landed 2026-09-01 00:04 BST inside the 2-day wall. Under the < 5 null. Awaiting the human's ruling. + +## delaunay_adapt_split_pl_eff_1_outer — SUSPECT +- decision: (none) +- ruled: no + +Carried. Left to finish under R-20260831-08 and now finished: `evidence_increase = +2.7361` (logL +15.3716), landed 2026-09-01 01:59 BST with 55 min of wall to spare. Under the < 5 null but the least emphatic of the three, and the largest false positive on the board so far. The `342093_0` AdaptSplit-at-`pix[1]` readout for the same lens is separately awaiting judgement — it decides the default recipe's `pix[1]` regularization, and nothing downstream moves until it lands. Awaiting the human's ruling. + +## rectangular_adapt_pl_sersic_0 — HEALTHY +- decision: (none) +- ruled: no + +Carried. Witness landed 2026-09-01 17:54 BST: `evidence_increase = +0.2538`, `subhalo_stage: subhalo[1]_[search_lens_plane]` — the project's first grid-derived witness, so not like-for-like with the three delaunay rows. Its results JSON is deliberately untracked in git by the human's ruling and lives only on the laptop. Awaiting the human's ruling. diff --git a/batches/reviews/AGENTS.md b/batches/reviews/AGENTS.md index dc5b1fc..1ac87ee 100644 --- a/batches/reviews/AGENTS.md +++ b/batches/reviews/AGENTS.md @@ -21,6 +21,9 @@ follow-ups. - reviewed-at: 2026-09-01T17:10Z - review-minutes-actual: 18 +### Follow-ups accepted +- + ## — - decision: accept | rerun | drop | leave-to-finish | (none) - ruled: yes | no @@ -29,6 +32,10 @@ The human's words, verbatim, or (no note). For rerun this paragraph becomes the rerun phase's seed; for drop it is the ruling body. ``` +`Follow-ups accepted` is optional and is an `###`, not a `##`: `check` requires +every `##` in a review file to name a member of the slot's batch record, so a +non-member section has to sit a level down. + `` ∈ `HEALTHY | SUSPECT | FAILED | RUNNING`. The four verbs are the **whole** vocabulary — the Mind's `merge | tweak | reject | defer` belongs to the dev surface and never appears here. Members the human left untouched are diff --git a/dashboard.html b/dashboard.html index 7eef8f9..494087c 100644 --- a/dashboard.html +++ b/dashboard.html @@ -218,21 +218,41 @@

PyAutoCortexDashboard

Question. Run. Rule.

Every science phase the Cortex is holding. Tap a phase's 📋 and its next command is on your clipboard. The verdict is always yours — this page hands you the command, never the ruling.

-
  • 0Awaiting ruling
  • 0Running / submitted
  • 0Ready
  • 0Gated
  • 0Recent rulings
+
  • 4Awaiting ruling
  • 0Running / submitted
  • 0Ready
  • 1Gated
  • 12Recent rulings

Last updated 2026-09-01. This page is generated from phases/, rulings/, batches/, epics.md and projects.yaml, so it is only as current as they are. dashboard_refresh.yml re-renders it on every push to main, and the daily gate grading flips cleared gates before it runs.

markdown version · GitHub Page

Awaiting ruling markdown version

Results are in and nothing is running — the human's verdict is the only thing outstanding. Ordered failures first, then the phases a ruling is required for, then the clean ones.

-

(nothing awaiting a ruling)

+

Inference_programme — phase 10: InferenceRefs_v1 redo — the Phase 1 restart wave (array 342091)awaiting-ruling — inference_programme phase 10 · budget 6:00 · 20 review-min · runs 342091_[0-8,10]

+

Subhalo_validation — phase 2: delaunay_adapt_split on pl_eff_0 — the clumpy-source lensawaiting-ruling — subhalo_validation phase 2 · budget 48:00 · 8 review-min · runs 342027_1

+

Subhalo_validation — phase 3: delaunay_adapt_split on pl_eff_1_outer — the outer-component lens, plus the AdaptSplit fix rerunawaiting-ruling — subhalo_validation phase 3 · budget 48:00 · 8 review-min · runs 342027_2, 342093_0

+

Subhalo_validation — phase 4: rectangular_adapt on pl_sersic_0 — the RectangularBilinear comparisonawaiting-ruling — subhalo_validation phase 4 · budget 48:00 · 15 review-min · runs 342094_0, 342095_0

Running / submitted markdown version

On the queue or on the machine. Wall is what the run lines record at the last refresh, against the phase's own budget.

(nothing on the queue)

Ready markdown version

Gate cleared, witness registered — these are what cortex plan admits into a laptop slot.

(nothing ready to launch)

Gated markdown version

Waiting on development work. The daily grading job flips a phase to ready when every reference it names has closed.

-

(nothing gated)

+

Euclid — phase 4: DR1 prelim science run — 10 real Euclid lenses in euclid_dr1_prelim on RALeuclid_strong_lens_modeling_pipeline#48euclid_strong_lens_modeling_pipeline#49 — euclid phase 4 · budget 48:00 · 25 review-min

Recent rulings markdown version

The ledger of record, newest first. A verdict recorded only outside the Cortex does not exist.

-

(no rulings yet)

+ + + + + + + + + + + + + + +
RulingVerbPhaseBatch
R-20260901-04dropphases/inference_programme/failed_submissions_342008_10.md-
R-20260901-03dropphases/inference_programme/delaunay_fp64_retro_baseline.md2026-08-31-pm
R-20260901-02acceptphases/inference_programme/mge_fp64_retro_baseline.md2026-08-31-pm
R-20260901-01acceptphases/inference_programme/mge_pos_ref_reuse.md2026-08-31-pm
R-20260831-08leave-to-finishphases/subhalo_validation/delaunay_adapt_split_pl_eff_1_outer.md2026-08-31-am
R-20260831-07leave-to-finishphases/subhalo_validation/delaunay_adapt_split_pl_eff_0.md2026-08-31-am
R-20260831-06acceptphases/subhalo_validation/delaunay_adapt_split_pl_sersic_0.md2026-08-31-am
R-20260831-05dropphases/inference_programme/slogdet_ab_342017.md2026-08-31-am
R-20260831-04dropphases/inference_programme/refs_5_6_342016_56.md2026-08-31-am
R-20260831-03dropphases/inference_programme/knn_rerun_342016_7.md2026-08-31-am
R-20260831-02dropphases/inference_programme/smc_probe_342018.md2026-08-31-am
R-20260831-01dropphases/inference_programme/rewind_2026_08_31.md-

Epics markdown version

Long-running programmes whose science half lives here; each card links its development half in the Mind.

-

(no epics yet)

+

prepare the Euclid DR1 modelling pipeline, then prove it on real DR1 lenses — Mind half: PyAutoMind/epics.md#euclid-dr1-prep — ledger: phases/euclid/ — science half opens at phase 4, gated on the 3a PR and the 3b issue; 5–7 planned behind it

+

profile and certify the JAX inference stack — samplers, meshes, reference baselines — Mind half: PyAutoMind/epics.md#jax-inference-profiling — ledger: autolens_profiling/results/notes/inference/PROGRAMME.md — REWOUND to Phase 1 (2026-08-31); the InferenceRefs_v1 refs redo is awaiting ruling

+

expectation propagation as the scalable alternative to graphical joint fits — Mind half: PyAutoMind/epics.md#graphical-ep — ledger: PyAutoMind/research/graphical_ep/ep_campaign.md — both science phases planned; the projects are dormant and the campaign's

+

the Source & Cluster arc — cluster mass modelling and pixelized source inference — Mind half: PyAutoMind/epics.md#cluster-strong-lensing — ledger: PyAutoMind/draft/feature/autolens/source_cluster_arc.md — one science phase (arc phase 11), planned behind the arc's phase 10

Projects markdown version

The science body map — where each project lives, what syncs it and where its witness lands.

diff --git a/dashboard.md b/dashboard.md index 5a56f93..283c28a 100644 --- a/dashboard.md +++ b/dashboard.md @@ -10,17 +10,47 @@ Every science phase the Cortex is holding, on one page: what is waiting on your | Where | Count | |-------|------:| -| [Awaiting ruling](#awaiting-ruling) | 0 | +| [Awaiting ruling](#awaiting-ruling) | 4 | | [Running / submitted](#running--submitted) | 0 | | [Ready](#ready) | 0 | -| [Gated](#gated) | 0 | -| [Recent rulings](#recent-rulings) | 0 | +| [Gated](#gated) | 1 | +| [Recent rulings](#recent-rulings) | 12 | ## Awaiting ruling [markdown version](https://github.com/PyAutoLabs/PyAutoCortex/tree/main/phases/) — Results are in and nothing is running — the human's verdict is the only thing outstanding. Ordered failures first, then the phases a ruling is required for, then the clean ones. -- _(nothing awaiting a ruling)_ +
📋 Inference_programme — phase 10: InferenceRefs_v1 redo — the Phase 1 restart wave (array 342091) — inference_programme phase 10 · budget 6:00 · 20 review-min · runs 342091_[0-8,10] + +``` +Review the PyAutoCortex phase phases/inference_programme/inference_refs_v1_redo.md and help me rule on it: read its `## Witness` and the pulled evidence under its `## Where to look`, score the witness, then draft the ruling body for my approval and run `python3 scripts/cortex.py rule phases/inference_programme/inference_refs_v1_redo.md --body `. +``` + +
+ +
📋 Subhalo_validation — phase 2: delaunay_adapt_split on pl_eff_0 — the clumpy-source lens — subhalo_validation phase 2 · budget 48:00 · 8 review-min · runs 342027_1 + +``` +Review the PyAutoCortex phase phases/subhalo_validation/delaunay_adapt_split_pl_eff_0.md and help me rule on it: read its `## Witness` and the pulled evidence under its `## Where to look`, score the witness, then draft the ruling body for my approval and run `python3 scripts/cortex.py rule phases/subhalo_validation/delaunay_adapt_split_pl_eff_0.md --body `. +``` + +
+ +
📋 Subhalo_validation — phase 3: delaunay_adapt_split on pl_eff_1_outer — the outer-component lens, plus the AdaptSplit fix rerun — subhalo_validation phase 3 · budget 48:00 · 8 review-min · runs 342027_2, 342093_0 + +``` +Review the PyAutoCortex phase phases/subhalo_validation/delaunay_adapt_split_pl_eff_1_outer.md and help me rule on it: read its `## Witness` and the pulled evidence under its `## Where to look`, score the witness, then draft the ruling body for my approval and run `python3 scripts/cortex.py rule phases/subhalo_validation/delaunay_adapt_split_pl_eff_1_outer.md --body `. +``` + +
+ +
📋 Subhalo_validation — phase 4: rectangular_adapt on pl_sersic_0 — the RectangularBilinear comparison — subhalo_validation phase 4 · budget 48:00 · 15 review-min · runs 342094_0, 342095_0 + +``` +Review the PyAutoCortex phase phases/subhalo_validation/rectangular_adapt_pl_sersic_0.md and help me rule on it: read its `## Witness` and the pulled evidence under its `## Where to look`, score the witness, then draft the ruling body for my approval and run `python3 scripts/cortex.py rule phases/subhalo_validation/rectangular_adapt_pl_sersic_0.md --body `. +``` + +
## Running / submitted @@ -38,19 +68,68 @@ Every science phase the Cortex is holding, on one page: what is waiting on your [markdown version](https://github.com/PyAutoLabs/PyAutoCortex/tree/main/phases/) — Waiting on development work. The daily grading job flips a phase to ready when every reference it names has closed. -- _(nothing gated)_ +
📋 Euclid — phase 4: DR1 prelim science run — 10 real Euclid lenses in euclid_dr1_prelim on RAL — euclid phase 4 · budget 48:00 · 25 review-min — euclid_strong_lens_modeling_pipeline#48, euclid_strong_lens_modeling_pipeline#49 + +``` +python3 scripts/cortex.py gates --grade # euclid_strong_lens_modeling_pipeline#48, euclid_strong_lens_modeling_pipeline#49 +``` + +
## Recent rulings [markdown version](https://github.com/PyAutoLabs/PyAutoCortex/tree/main/rulings/) — The ledger of record, newest first. A verdict recorded only outside the Cortex does not exist. -- _(no rulings yet)_ +| Ruling | Verb | Phase | Batch | +|---|---|---|---| +| [R-20260901-04](rulings/2026/09/R-20260901-04.md) | drop | phases/inference_programme/failed_submissions_342008_10.md | - | +| [R-20260901-03](rulings/2026/09/R-20260901-03.md) | drop | phases/inference_programme/delaunay_fp64_retro_baseline.md | 2026-08-31-pm | +| [R-20260901-02](rulings/2026/09/R-20260901-02.md) | accept | phases/inference_programme/mge_fp64_retro_baseline.md | 2026-08-31-pm | +| [R-20260901-01](rulings/2026/09/R-20260901-01.md) | accept | phases/inference_programme/mge_pos_ref_reuse.md | 2026-08-31-pm | +| [R-20260831-08](rulings/2026/08/R-20260831-08.md) | leave-to-finish | phases/subhalo_validation/delaunay_adapt_split_pl_eff_1_outer.md | 2026-08-31-am | +| [R-20260831-07](rulings/2026/08/R-20260831-07.md) | leave-to-finish | phases/subhalo_validation/delaunay_adapt_split_pl_eff_0.md | 2026-08-31-am | +| [R-20260831-06](rulings/2026/08/R-20260831-06.md) | accept | phases/subhalo_validation/delaunay_adapt_split_pl_sersic_0.md | 2026-08-31-am | +| [R-20260831-05](rulings/2026/08/R-20260831-05.md) | drop | phases/inference_programme/slogdet_ab_342017.md | 2026-08-31-am | +| [R-20260831-04](rulings/2026/08/R-20260831-04.md) | drop | phases/inference_programme/refs_5_6_342016_56.md | 2026-08-31-am | +| [R-20260831-03](rulings/2026/08/R-20260831-03.md) | drop | phases/inference_programme/knn_rerun_342016_7.md | 2026-08-31-am | +| [R-20260831-02](rulings/2026/08/R-20260831-02.md) | drop | phases/inference_programme/smc_probe_342018.md | 2026-08-31-am | +| [R-20260831-01](rulings/2026/08/R-20260831-01.md) | drop | phases/inference_programme/rewind_2026_08_31.md | - | ## Epics [markdown version](https://github.com/PyAutoLabs/PyAutoCortex/blob/main/epics.md) — Long-running programmes whose science half lives here; each card links its development half in the Mind. -- _(no epics yet)_ +
📋 prepare the Euclid DR1 modelling pipeline, then prove it on real DR1 lenses — Mind half: PyAutoMind/epics.md#euclid-dr1-prep — ledger: phases/euclid/ — science half opens at phase 4, gated on the 3a PR and the 3b issue; 5–7 planned behind it + +``` +Work out where the Cortex half of the epic euclid-dr1-prep stands from phases/euclid/ and tell me what its next phase is. +``` + +
+ +
📋 profile and certify the JAX inference stack — samplers, meshes, reference baselines — Mind half: PyAutoMind/epics.md#jax-inference-profiling — ledger: autolens_profiling/results/notes/inference/PROGRAMME.md — REWOUND to Phase 1 (2026-08-31); the InferenceRefs_v1 refs redo is awaiting ruling + +``` +Work out where the Cortex half of the epic jax-inference-profiling stands from autolens_profiling/results/notes/inference/PROGRAMME.md and tell me what its next phase is. +``` + +
+ +
📋 expectation propagation as the scalable alternative to graphical joint fits — Mind half: PyAutoMind/epics.md#graphical-ep — ledger: PyAutoMind/research/graphical_ep/ep_campaign.md — both science phases planned; the projects are dormant and the campaign's + +``` +Work out where the Cortex half of the epic graphical-ep stands from PyAutoMind/research/graphical_ep/ep_campaign.md and tell me what its next phase is. +``` + +
+ +
📋 the Source & Cluster arc — cluster mass modelling and pixelized source inference — Mind half: PyAutoMind/epics.md#cluster-strong-lensing — ledger: PyAutoMind/draft/feature/autolens/source_cluster_arc.md — one science phase (arc phase 11), planned behind the arc's phase 10 + +``` +Work out where the Cortex half of the epic cluster-strong-lensing stands from PyAutoMind/draft/feature/autolens/source_cluster_arc.md and tell me what its next phase is. +``` + +
## Projects diff --git a/docs/schema_decisions.md b/docs/schema_decisions.md index a8e1ebe..f68f870 100644 --- a/docs/schema_decisions.md +++ b/docs/schema_decisions.md @@ -306,3 +306,63 @@ Choices made in phase 2 — the conductor, the board and the workflows `Jammy2211/…` and the row says so in `remote:` with the reason in `note:`, so nobody re-derives the answer to "should this move into the org?" once a quarter. + +53. **Provenance header: `Migrated-from:` joins `PHASE_KEYS` and `RULING_KEYS`** + (one tuple edit each, plus the REFERENCE key tables and a test); its value + is the Mind path or ledger anchor the file was transcribed from. Batch + records may carry `- migrated-from:` (there is no key check on a record). + Why: phase 4's whole job is moving records that were written somewhere + else, and a migrated file that cannot name its source is a claim without a + provenance. The prompt already required the line; without the key the + current schema would have flagged every migrated file as unknown-key + drift, so the requirement and the checker had to be reconciled — in favour + of the requirement. + +54. **Intra-Cortex sequencing is a body line, not a gate.** `Gates:` stays + GitHub-refs-only. A phase that waits on another *Cortex* phase's ruling is + `State: planned` with a line under `## Question` reading `Ready when: + accepted (ruling R-…)`; the human moves it to `ready` once that + ruling lands. A phase whose development gate has no issue yet is likewise + `planned`, with `Ready when: is issued — add its ref to + Gates: and move to gated`. + Why: `gates --grade` polls GitHub, and the daily grader is the one + scheduled job that mutates the ledger. A gate it cannot poll would either + have to fail closed for ever or teach the grader a second, non-GitHub + vocabulary. A science phase waiting on a sibling phase is waiting on a + *human ruling*, which is exactly the thing the Cortex refuses to automate, + so it belongs in prose the human reads and not in a field a robot grades. + +55. **A Cortex-spawned development follow-up gets its issue at filing, without + leaving `draft/`.** The Mind prompt carries a body line `Issue: + (opened as a Cortex gate ref; reuse in start_dev — never open a + second)`, and `create_issue` reuses that ref instead of opening another; + Mind `REFERENCE.md` gains the rule. The same mechanism is what gives the + euclid 3b gate its ref. + Why: `Gates:` is GitHub-refs-only (decision 54), so a Cortex phase gated on + work the Mind has not started yet has nothing to point at. The alternatives + were to promote the prompt to `active/` before anyone works it — which + lies about the workflow state — or to leave the phase ungated and hope. An + issue is cheap, is the organism's real join key, and is the one artefact + both halves can name; the `Issue:` line stops the second, duplicate issue + that `start_dev` would otherwise open. + +56. **Legacy-born encodings: how pre-Cortex history enters the ledger.** A run + that happened before the Cortex existed is a `legacy` (reusable) or + `legacy_wrong` (quarantined) run line carrying `where:`; a phase whose runs + are *all* `legacy_wrong` is ruled `drop` straight from `ready`, because it + cannot be pulled — there is nothing reviewable to pull. A programme-wide + directive such as the 2026-08-31 REWIND is one ruling on a synthetic phase + (`inference_programme/rewind_2026_08_31`) whose runs are the ids the + directive itself names, with the directive verbatim as the ruling body. A + multi-lens project gets one phase per lens. Transcribed batch records are + rewritten in the Cortex grammar (member slug = phase slug) and the Mind + originals stay intact with a `- moved-to:` line rather than being reduced + to stubs; dev-only members of a mixed slot are noted in the record's + `notes:`, never transcribed as science members. + Why: the alternative — inventing `done` run lines for runs nobody reviewed + — would let quarantined evidence re-enter the programme through the back + door, which is the precise failure the REWIND was called to stop. Keeping + the Mind originals verbatim is a deviation from the prompt's "stub": a + history that only survives in `git log` is a history the next reader will + not find, and two records that disagree are worse than one record and a + pointer. diff --git a/epics.md b/epics.md index a48771c..2d5cae0 100644 --- a/epics.md +++ b/epics.md @@ -18,10 +18,65 @@ Schema per entry — the Mind's, plus one field: - notes: ``` -`- mind-half:` names the Mind entry by slug; the Mind's entries gain the -reciprocal `- cortex-half:` in phase 4 of the birth epic, when the split epics -(`euclid-dr1-prep` phases 4, 5, 6a, 6b; `jax-inference-profiling` whole; -`graphical-ep` phases 3/4 science halves; `cluster-strong-lensing` phase 11) -move here. Until then this file holds no entries. +`- mind-half:` names the Mind entry by slug; the Mind's entries carry the +reciprocal `- cortex-half:`. The four split epics below moved their science +halves here in phase 4 of the birth epic (2026-09-01): `euclid-dr1-prep` +phases 4, 5, 6a and 6b (renumbered here as Cortex phases 4–7); +`jax-inference-profiling` whole; the science halves of `graphical-ep` phases +3 and 4; and `cluster-strong-lensing` phase 11. This file is ledger: it auto-merges (`scripts/ledger_merge.py`). + +--- + +## euclid-dr1-prep +- title: prepare the Euclid DR1 modelling pipeline, then prove it on real DR1 lenses +- ledger: phases/euclid/ +- mind-half: PyAutoMind/epics.md#euclid-dr1-prep +- status: science half opens at phase 4, gated on the 3a PR and the 3b issue; 5–7 planned behind it +- notes: The Mind keeps the development phases (0–3, 8, 9) and renumbered them in + phase 4 of the birth epic so the old 3a/3b/6a/6b/6c collisions are gone. Here, + phase 4 is the 10-lens DR1 prelim science run, 5 the resimulator, 6 Sersic-index + recovery (was 6a) and 7 magnification robustness (was 6b); 6c stays in the Mind as + its `Phase: 8` because it is a source-code audit, not a run. Phase 4 gates 5, and 5 + gates 6 and 7 — as intra-Cortex sequencing (decision 54) those waits are `Ready + when:` lines in each phase's `## Question`, not `Gates:` refs. + +## jax-inference-profiling +- title: profile and certify the JAX inference stack — samplers, meshes, reference baselines +- ledger: autolens_profiling/results/notes/inference/PROGRAMME.md +- mind-half: PyAutoMind/epics.md#jax-inference-profiling +- status: REWOUND to Phase 1 (2026-08-31); the InferenceRefs_v1 refs redo is awaiting ruling +- notes: The whole epic's science half lives here. The 2026-08-31 REWIND quarantined + every mesh / pixelization result to `output/legacy_wrong/` and every MGE result to + `output/legacy/` (reusable pending batch review), made gates A, B pt 1 and B pt 2 + provisional, and restarted the programme at Phase 1. Ten of the eleven phases here + are legacy-born (decision 56); the eleventh, `cluster_extended_source_inference`, + belongs to `cluster-strong-lensing` and sits in this project directory because that + is where its runs would land. R-20260901-03 binds the redo: a mesh row produced + without a `PositionsLH` positions penalty is not a citable reference. + +## graphical-ep +- title: expectation propagation as the scalable alternative to graphical joint fits +- ledger: PyAutoMind/research/graphical_ep/ep_campaign.md +- mind-half: PyAutoMind/epics.md#graphical-ep +- status: both science phases planned; the projects are dormant and the campaign's + development phases run first +- notes: Two science phases, one per project — `slope_hierarchy/n25_scale_up` (the + optional N=25–50 scale-up of a project that answered all four of its goals at N=5) + and `ic50_workspace/ep_scale_up` (EP end to end on the real IC50 model, with the + derived-variable handling used exactly as it exists). Both wait on the campaign's + Mind phase 2 being issued — `Ready when:` lines, not `Gates:` refs, until that + prompt has an issue to point at. A planned phase on a dormant project is legal. + +## cluster-strong-lensing +- title: the Source & Cluster arc — cluster mass modelling and pixelized source inference +- ledger: PyAutoMind/draft/feature/autolens/source_cluster_arc.md +- mind-half: PyAutoMind/epics.md#cluster-strong-lensing +- status: one science phase (arc phase 11), planned behind the arc's phase 10 +- notes: Only the arc's line 11 — the feasibility study for joint gradient-based + inference of cluster mass plus pixelized sources — is a run-and-ruling phase; the + rest of the twelve are development. It is filed under `phases/inference_programme/` + because that is the project whose RAL tree and profiling harness its runs would use. + The arc's `Phase: 10` collision with `cluster_regime_narrative.md` is noted in the + Mind's arc ledger, not fixed here. diff --git a/phases/euclid/dr1_prelim_10_lens_science_run.md b/phases/euclid/dr1_prelim_10_lens_science_run.md new file mode 100644 index 0000000..e51d857 --- /dev/null +++ b/phases/euclid/dr1_prelim_10_lens_science_run.md @@ -0,0 +1,66 @@ +# Euclid — phase 4: DR1 prelim science run — 10 real Euclid lenses in euclid_dr1_prelim on RAL + +Project: euclid +Phase: 4 +State: gated +Gates: euclid_strong_lens_modeling_pipeline#48, euclid_strong_lens_modeling_pipeline#49 +Witness: 10 lenses fitted end to end on RAL from the pipeline repo alone, with a complete catalogue folder (latents present) whose numerics match the 20260623 reference within the tolerance stated before the run +Budget: 48:00 +Runs: +Ruling: +Lane: local-dev +Review-minutes: 25 +Epic: euclid-dr1-prep +Filed: 2026-08-28 +Migrated-from: PyAutoMind/draft/research/euclid/dr1_prelim_10_lens_science_run.md + +## Question + +Can everything delivered for the DR1 runs out of `Science/euclid` now be delivered from +`euclid_strong_lens_modeling_pipeline` alone? A new science project at +`/mnt/c/Users/Jammy/Science/euclid_dr1_prelim` takes the first 10 lenses alphanumerically from +`/mnt/c/Users/Jammy/Science/euclid`, fits them with the two-stage CPU route on RAL (`vis_lp` +JAX → reset → `vis_pix` numba + multiprocessing, submitted from the pipeline repo's own +`hpc/batch_cpu` scripts), and produces a catalogue folder with the full latent-variable output. + +This is a science run on RAL, not a software task: it is human-driven, runs on wall-clock +timescales of days, and its deliverable is a result and a written verdict. It must never be +handed to an autonomous ship gate. + +The run goes through the `autolens_assistant` euclid skill family — `euclid_prepare_data`, +`euclid_setup_pipeline`, `euclid_model_lens`, `euclid_hpc_runs`. (There is no literal +`euclid_mode` in the assistant; the skills are what the request meant.) + +State the sort key explicitly and list the 10 chosen datasets by name in the issue before +running, so the selection is reproducible and auditable. + +## Witness + +Acceptance, verbatim from the Mind prompt: + +- 10 lenses fitted end to end on RAL from the pipeline repo alone. +- Catalogue folder complete, latents present, numerics approximately matching the + 20260623 reference within a stated tolerance. +- A written verdict: **can everything delivered for the DR1 runs out of + `Science/euclid` now be delivered from `euclid_strong_lens_modeling_pipeline`?** + Any "no" is itself a finding and should feed back into phase 1. +- Gates phase 5 (the resimulations need these results as their truth inputs). + +The catalogue product set is the reference tile's: `lens_mass.csv`, `lens_sersic.csv`, +`source_sersic.csv`, `magnitudes.csv`, `model.fits`, `pre_psf.fits`, and the PNG set +(`fit_sersic.png`, `fit_multi_wavelength.png`, `rgb.png`, `segmentation.png`, +`vis_lp_fit.png`, `vis_lp_image_with_positions.png`, `vis_pix_fit.png`). + +## Where to look + +- `euclid_dr1_prelim` (project row `euclid_dr1_prelim`): the new project tree, once created +- `euclid` (project row `euclid`): `catalogue/catalogue/dr1_prelim_grade_ab_catalogue_csvs_20260623/` + — the numerical comparison reference for the same 10 lenses +- `autolens_assistant/skills/euclid_{prepare_data,setup_pipeline,model_lens,hpc_runs}.md` +- `euclid_strong_lens_modeling_pipeline`: `catalogue/scripts/`, `hpc/batch_cpu/` + +## Runs + +## Ruling + +(none) diff --git a/phases/euclid/magnification_robustness.md b/phases/euclid/magnification_robustness.md new file mode 100644 index 0000000..2c82897 --- /dev/null +++ b/phases/euclid/magnification_robustness.md @@ -0,0 +1,73 @@ +# Euclid — phase 7: how robust are magnification estimates? model-match vs mismatch across the 10 Euclid lenses + +Project: euclid +Phase: 7 +State: planned +Gates: +Witness: all five model rungs run on all 10 simulated lenses (or a documented, justified reduction), source-mismatch and lens-light-leakage effects quantified separately, and a written Delaunay magnification verdict +Budget: 24:00 +Runs: +Ruling: +Lane: local-dev +Review-minutes: 20 +Epic: euclid-dr1-prep +Filed: 2026-08-28 +Migrated-from: PyAutoMind/draft/research/euclid/magnification_robustness.md + +## Question + +Ready when: euclid 5 accepted. Runs in parallel with euclid 6. + +There are known systematics in magnification estimation: when the lens light and source model +match the simulated data (Sersics fitted to Sersics) it is fine; when there is mismatch (an MGE +is used) it breaks — for two distinct reasons. An MGE *source* is source-model mismatch; an MGE +*lens light* lets lens light "leak" into the source model and corrupt the magnification. The +Delaunay source magnification has never been validated and could carry a bug (pixel areas). + +The comparison matrix, fitted against the recorded true magnification of the 10 phase-5 +simulations (Sersic lens light + Sersic source): + +1. **Matched** — Sersic lens light, Sersic source. The control. If it is not fine, nothing + below is interpretable. +2. **MGE source, Sersic lens light** — isolates *source*-model mismatch. +3. **MGE lens light, Sersic source** — isolates the **lens-light leakage** channel. A + different failure mode from (2); the two must be separated, not lumped as "MGE breaks it". +4. **MGE both** — to see whether the effects add. +5. **Delaunay source** (with Sersic and MGE lens light) — never validated. The novel leg. + +Definitions matter: there is more than one magnification in play (the point/Hessian value and +the area-based ratio `A_img / A_src`). State which is compared at each rung and make sure the +phase-5 truth is the same quantity; a disagreement between the two is itself a finding. + +Prior art to consult before designing the runs — do not duplicate it: the cluster epic's +`mesh_magnification_correctness` prompt already covers simulate-and-recover magnification +across every mesh variant, and its audit found `areas_for_magnification` exists for only two +mesh geometries and has no direct test. This phase is the Euclid-data instance of that +question. The user's own earlier magnification research is the origin of the "matched is fine, +mismatched breaks" intuition — surface it rather than rediscovering it. + +## Witness + +Acceptance, verbatim from the Mind prompt: + +- All five rungs run on all 10 lenses (or a documented, justified reduction). +- Source-mismatch and lens-light-leakage effects quantified separately. +- A written robustness verdict, including a usable statement of when Euclid DR1 + magnifications can be trusted and when they cannot. +- Feeds phase 7's magnification products. + +("Phase 7" there is the Mind's numbering — the catalogue phase, now Mind `Phase: 9`.) + +## Where to look + +- `euclid_dr1_prelim` (project row): the phase-5 simulations and their recorded true magnifications +- Mind `draft/bug/autoarray/delaunay_area_magnification_audit.md` — the source-code half (6c), + which this phase hands its Delaunay leg to whatever it shows +- Mind `draft/test/workspaces/mesh_magnification_correctness.md` — the cluster epic's + simulate-and-recover work; read it, do not duplicate it + +## Runs + +## Ruling + +(none) diff --git a/phases/euclid/resimulate_fitted_lens_simulator.md b/phases/euclid/resimulate_fitted_lens_simulator.md new file mode 100644 index 0000000..fa664b6 --- /dev/null +++ b/phases/euclid/resimulate_fitted_lens_simulator.md @@ -0,0 +1,65 @@ +# Euclid — phase 5: resimulate a fitted Euclid lens, and resimulate the 10 prelim lenses with true magnifications recorded + +Project: euclid +Phase: 5 +State: planned +Gates: +Witness: a user with a fit result resimulates it in one command; 10 resimulations exist with recorded truths including magnifications, and the prior-edge rule's inferred and simulated Sersic index is recorded per lens +Budget: 24:00 +Runs: +Ruling: +Lane: local-dev +Review-minutes: 20 +Epic: euclid-dr1-prep +Filed: 2026-08-28 +Migrated-from: PyAutoMind/draft/feature/euclid/resimulate_fitted_lens_simulator.md + +## Question + +Ready when: euclid 4 accepted (it needs the 10 real fits as truth inputs). + +Can a Euclid lens we have a result for be resimulated, so the simulated data can be fitted +and PyAutoLens tested for recovery of the correct values? Two artefacts, two homes: + +- **`scripts/simulator.py` in `euclid_strong_lens_modeling_pipeline`** — a general, + documented example: *"I have fitted a lens; resimulate it."* The script already exists + (shipped 2026-08-29, PR #46) with a `--from-result` mode in place, though its SED is flat + for now. **Extend that script; do not write a second one.** That real PR is filed as a Mind + development prompt when this phase opens. +- **The 10 resimulations in `euclid_dr1_prelim`** — science outputs built from phase 4's + results; they live in the science project, not the public repo. + +Recipe: Sersic lens light, Sersic source (deliberately over-simple for some of the 10, and +that is explicitly fine — phase 6 tests recovery under a *matched* model first). Record the +true magnification of every simulated lens as a first-class output stored alongside the +dataset — phase 7's entire question depends on that ground truth, so it must not be +reconstructible-only-by-rerunning, and it must record enough to distinguish the point and +area magnification definitions. Prior-edge correction: many real fits pin `sersic_index = 5` +at the prior edge; when a lens's inferred lens-light Sersic index is at that edge, lower it to +a value in [2, 4], record both values, and document the rule for choosing the replacement +(fixed 3.0? drawn? population median?) — phase 6's interpretation depends on it. Match the +real data's PSF, noise and exposure characteristics, and be explicit about which aspect is +idealised and which is faithful. + +## Witness + +Acceptance, verbatim from the Mind prompt: + +- A user with a fit result can resimulate it in one command. +- All 10 resimulations exist with recorded truths, including magnifications. +- The prior-edge rule is applied and both values recorded per lens. +- Gates 6a and 6b. + +(6a and 6b are Cortex phases 6 and 7.) + +## Where to look + +- `euclid_strong_lens_modeling_pipeline`: `scripts/simulator.py`, `smoke_tests.txt` +- `euclid_dr1_prelim` (project row): the 10 simulated datasets and their truth table +- `phases/euclid/dr1_prelim_10_lens_science_run.md` — the fits these resimulations start from + +## Runs + +## Ruling + +(none) diff --git a/phases/euclid/sersic_index_recovery.md b/phases/euclid/sersic_index_recovery.md new file mode 100644 index 0000000..36dd415 --- /dev/null +++ b/phases/euclid/sersic_index_recovery.md @@ -0,0 +1,62 @@ +# Euclid — phase 6: do we recover Sersic indices? 10 simulated Euclid lenses vs the real prior-edge pile-up + +Project: euclid +Phase: 6 +State: planned +Gates: +Witness: 10 simulated fits complete with a recovered-vs-input Sersic index table and plot, the same 10 real lenses' prior-edge status established from phase 4, and a written verdict naming the likely cause +Budget: 24:00 +Runs: +Ruling: +Lane: local-dev +Review-minutes: 20 +Epic: euclid-dr1-prep +Filed: 2026-08-28 +Migrated-from: PyAutoMind/draft/research/euclid/sersic_index_recovery.md + +## Question + +Ready when: euclid 5 accepted. + +Real Euclid fits pile up at `sersic_index = 5` — the prior's upper edge. Two competing +explanations: + +- **Real signal.** These lenses genuinely have high-index light profiles and the prior is + simply too narrow. +- **Artefact.** A defective PSF model (or another data-side systematic) drives the fit to + absorb the mismatch into a high Sersic index. + +The phase-5 resimulations are built with indices in [2, 4] and without whatever data defect is +suspected. If the fits to the *simulated* data recover the input index, the pile-up is an +artefact of the real data; if they *also* pile up at 5, the problem is in the model or the +fitting, not the data. + +Do **not** implement the fix here — the request is explicit that we get to the point where we +have results for 10 lenses and then work out what changes we want to make. + +Interpretation hazards to respect: N = 10 is small, and a pile-up in 2 of 10 does not +distinguish the hypotheses — say up front what fraction would count as a result. Note which +phase-5 replacement-index rule was used and what it costs the inference. Confirm the simulated +fits use the **same** prior as the real fits, or the comparison is not like-for-like. + +## Witness + +Acceptance, verbatim from the Mind prompt: + +- 10 simulated fits complete, recovered-vs-input table and plot produced. +- The real-lens prior-edge status of the same 10 lenses established from phase 4 output. +- A written verdict naming the likely cause and the recommended change, filed as a + follow-up prompt if a change is warranted. No fix implemented here. + +## Where to look + +- `euclid` (project row): `catalogue/scripts/plot_lens_sersic_index.py` — already plots this + population; reuse it rather than hand-rolling a new plot +- `euclid_dr1_prelim` (project row): the phase-5 simulated datasets and their truth table +- `euclid_strong_lens_modeling_pipeline`: the `sersic_lens_model.py` family used on the real data + +## Runs + +## Ruling + +(none) diff --git a/phases/ic50_workspace/ep_scale_up.md b/phases/ic50_workspace/ep_scale_up.md new file mode 100644 index 0000000..6cd16c2 --- /dev/null +++ b/phases/ic50_workspace/ep_scale_up.md @@ -0,0 +1,64 @@ +# Ic50_workspace — phase 1: IC50 use case — EP end to end with the existing derived-variable handling + +Project: ic50_workspace +Phase: 1 +State: planned +Gates: +Witness: one committed EP-vs-graphical parity table (means ± errors) at small N, EP running at the largest N reached with per-rung timings committed, and any derived-variable or scaling blocker recorded as its own filed prompt +Budget: 48:00 +Runs: +Ruling: +Lane: local-dev +Review-minutes: 25 +Epic: graphical-ep +Filed: 2026-08-19 +Migrated-from: PyAutoMind/draft/research/graphical_ep/ic50_ep_scale_up.md + +## Question + +Ready when: graphical-ep phase 2 (Mind) is issued — add its ref to `Gates:` and move to `gated`. + +The IC50 cancer use case is the scale target: the end goal is graphical + EP fits at 10 000+ +datasets, with a clear demonstration that EP matches the graphical joint fit at small N before +anyone trusts it at large N. + +The final model has a **derived variable**: the per-dataset factor results inform the priors on +the global model, currently through the declarative framework. **Scope decision, made at intake +on 2026-08-19: use the derived-variable handling exactly as it exists.** Formalising derived +variables as a first-class EP/API concept is explicitly out of scope — do not let it derail the +runs. If the existing handling blocks an EP fit outright, record the blocker and route it as its +own prompt; do not redesign inline. + +The work, in order: + +1. **Get an EP fit running end to end** on the real IC50 model, derived variable and all, at + small N. The development half is the Mind prompt `feature/autofit/ep_lbfgs_jax.md` (swap + DynestyStatic → LBFGS/JAX for the simple 3-parameter factor fits) — the speed lever for the + per-factor fits; it can land before or alongside this and stays its own PR. +2. **EP-vs-graphical parity at small N** — same model both ways, compare parameter means and + errors. This is the trust-building deliverable. +3. **Scale ladder** — grow N stepwise toward 10k, recording wall time, memory and disk at each + rung. Expect the per-factor fits to be fast and the autofit wrapper overhead to dominate; + that evidence feeds the EP-profiling epic rather than being fixed inline here. + +## Witness + +Acceptance, verbatim from the Mind prompt: + +- One committed parity table (EP vs graphical, means ± errors) at small N. +- EP runs at the largest N reached, with per-rung timings committed. +- Any derived-variable or scaling blocker recorded as its own filed prompt, not patched ad hoc. + +## Where to look + +- `ic50_workspace` (project row) — external checkout, non-standard layout +- — scientific context and run help +- Mind `draft/feature/autofit/ep_lbfgs_jax.md` — the development half (the speed lever) +- Mind `research/graphical_ep/ep_campaign.md` phase 4 — the campaign row this feeds +- Mind `research/autofit/autofit_profiling_bootstrap.md` — where the wrapper-overhead evidence goes + +## Runs + +## Ruling + +(none) diff --git a/phases/inference_programme/cluster_extended_source_inference.md b/phases/inference_programme/cluster_extended_source_inference.md new file mode 100644 index 0000000..0507d0a --- /dev/null +++ b/phases/inference_programme/cluster_extended_source_inference.md @@ -0,0 +1,53 @@ +# Inference_programme — phase 11: cluster extended-source inference — gradient-based fitting building on JAX knowledge + +Project: inference_programme +Phase: 11 +State: planned +Gates: +Witness: a written feasibility verdict with profiling numbers and a go/no-go for a follow-up implementation prompt — not shipped inference machinery +Budget: 24:00 +Runs: +Ruling: +Lane: local-dev +Review-minutes: 25 +Epic: cluster-strong-lensing +Filed: 2026-08-19 +Migrated-from: PyAutoMind/draft/research/autolens/cluster_extended_source_inference.md + +## Question + +Ready when: cluster phase 10 (Mind draft `draft/feature/workspaces/cluster_pixelized_analysisfactor.md`) +is issued — add its ref to `Gates:` and move to `gated`. + +Part of the Source & Cluster arc (phase 11 of 12). User request, verbatim: "Once this is +robust begin to extend inference with extended sources, build on JAX gradient knowledge." + +Research-first. Phase 10 delivers post-inference refinement (mass model mostly fixed); this +phase asks whether full joint inference of cluster mass + pixelized sources is tractable with +the JAX gradient stack (MultiStart / gradient samplers, the SMC warm-start work, implicit-diff +PointSolver gradients from #657). + +Known constraints from the JAX campaign memory: pixelized-source gradient sampling was +previously found infeasible (reg/logdet NaN localisation, Delaunay `sqrt(dual_area)` NaN +gradients, Delaunay needs `custom_jvp`); factor-graph fits multiply the cost by `n_sources`. +The research question is what changed and what is needed — which mesh (rectangular uniform is +the gradient-safest), which sampler tier, what the per-iteration cost is at cluster scale, and +whether positions-likelihood + imaging-likelihood factor graphs are jointly jit-able. + +## Witness + +A written feasibility verdict with profiling numbers and a go/no-go for a follow-up +implementation prompt — not shipped inference machinery. + +## Where to look + +- Mind `draft/feature/autolens/source_cluster_arc.md` — the arc ledger (this is its line 11) +- Mind `draft/feature/workspaces/cluster_pixelized_analysisfactor.md` — cluster phase 10, the gate +- `inference_programme` (project row): `output/searches/*/cluster/` — the cluster trees the + REWIND left in the active tree + +## Runs + +## Ruling + +(none) diff --git a/phases/inference_programme/delaunay_fp64_retro_baseline.md b/phases/inference_programme/delaunay_fp64_retro_baseline.md new file mode 100644 index 0000000..1b692e8 --- /dev/null +++ b/phases/inference_programme/delaunay_fp64_retro_baseline.md @@ -0,0 +1,50 @@ +# Inference_programme — phase 9: the retro-adopted delaunay_fp64 InferenceRefs_v1 baseline + +Project: inference_programme +Phase: 9 +State: dropped +Gates: +Witness: the retro-adopted delaunay_fp64 baseline is a physical solution — checkable by inspecting the run's source-plane reconstruction for the demagnified-source signature +Budget: 6:00 +Runs: 339071 +Ruling: R-20260901-03 +Lane: local-dev +Review-minutes: 4 +Epic: jax-inference-profiling +Filed: 2026-08-24 +Migrated-from: autolens_profiling/results/notes/inference/DECISIONS.md § 2026-09-01 — Batch 2026-08-31-pm review rulings (InferenceRefs_v1 legacy-reuse members) + +## Question + +The twin of phase 8: `results/baselines/InferenceRefs_v1/delaunay_fp64/` was adopted +retroactively on the same day from an existing Nautilus run. Does *that* adoption stand? + +The run behind the row is RAL job **339071** (24 Aug 2026) — same identification route as +phase 8: the baseline's `source_artifact` is +`results/searches/nautilus/imaging/delaunay/hst/hpc_hpc_a100_fp64.json`, and the only mirror +log that writes it is `logs/output/output.339071.out`, whose `Total wall: 1938.52 s` matches +the baseline's `total_wall_s: 1938.52`. + +Its run tree is in `output/legacy_wrong/` — it is a mesh cell, quarantined by the REWIND. + +## Witness + +The reconstruction is a physical source, not the demagnified-source solution the Inversion +bias produces when no positions likelihood constrains the mass model. + +See . + +## Where to look + +- `autolens_profiling/results/baselines/InferenceRefs_v1/delaunay_fp64/{README.md,reference.json,target.json}` +- `inference_programme` (project row): `output/legacy_wrong/searches/nautilus/imaging/delaunay/hst/hpc_a100_fp64/` — the tree the human inspected +- `logs/output/output.339071.out` on the mirror — the log that wrote the source artifact + +## Runs + +- 339071: legacy_wrong — gpu — submitted 2026-08-24 — wall 0:32 — pre-Cortex run, migrated + where: output/legacy_wrong/searches/nautilus/imaging/delaunay/hst/hpc_a100_fp64/b29ffe390c18e070b3eaba60270cb502 + +## Ruling + +R-20260901-03 — drop diff --git a/phases/inference_programme/failed_submissions_342008_10.md b/phases/inference_programme/failed_submissions_342008_10.md new file mode 100644 index 0000000..a18cdf8 --- /dev/null +++ b/phases/inference_programme/failed_submissions_342008_10.md @@ -0,0 +1,50 @@ +# Inference_programme — phase 6: the three submissions that died in a second (CWD trap) + +Project: inference_programme +Phase: 6 +State: dropped +Gates: +Witness: the three submissions produce result rows; they did not — each died in about one second before writing any output +Budget: 6:00 +Runs: 342008, 342009, 342010 +Ruling: R-20260901-04 +Lane: local-dev +Review-minutes: 1 +Epic: jax-inference-profiling +Filed: 2026-08-29 +Migrated-from: PyAutoMind/batches/2026-08-31-am.md § member failed-submissions-342008-10 + +## Question + +Three hand submissions from the 2026-08-29 night wave died in about a second each. Why, and is +anything recoverable from them? + +The answer is the CWD trap: the jobs were submitted from a directory the batch scripts could +not resolve their paths against. Carried into the 2026-08-31-am packet as `UNREVIEWED`, the one +member the human left untouched, and carried again past the pm slot. + +## Witness + +Result rows under the active `output/searches/` tree for the three job ids. + +None exist. Each job died in ~1 s before writing a run directory, so there is no output to +pull, nothing to quarantine but the job ids themselves, and no `.out` log reached the mirror. + +## Where to look + +- `inference_programme` (project row): nothing under `output/` — the runs wrote no directory +- `PyAutoMind/batches/2026-08-31-am.md` — member `failed-submissions-342008-10`, `UNREVIEWED, carried to next packet` +- `autolens_profiling/results/notes/inference/DECISIONS.md` — the 2026-08-31 REWIND entry that superseded the question + +## Runs + +- 342008: legacy_wrong — gpu — submitted 2026-08-29 — wall 0:00 — pre-Cortex run, migrated; died in ~1 s under the CWD trap — no run directory, no output, no .out log on the mirror + where: output/legacy_wrong/ +- 342009: legacy_wrong — gpu — submitted 2026-08-29 — wall 0:00 — pre-Cortex run, migrated; died in ~1 s under the CWD trap — no run directory, no output, no .out log on the mirror + where: output/legacy_wrong/ +- 342010: legacy_wrong — gpu — submitted 2026-08-29 — wall 0:00 — pre-Cortex run, migrated; died in ~1 s under the CWD trap — no run directory, no output, no .out log on the mirror + where: output/legacy_wrong/ + +## Ruling + +R-20260901-04 — drop diff --git a/phases/inference_programme/inference_refs_v1_redo.md b/phases/inference_programme/inference_refs_v1_redo.md new file mode 100644 index 0000000..73b83f7 --- /dev/null +++ b/phases/inference_programme/inference_refs_v1_redo.md @@ -0,0 +1,76 @@ +# Inference_programme — phase 10: InferenceRefs_v1 redo — the Phase 1 restart wave (array 342091) + +Project: inference_programme +Phase: 10 +State: awaiting-ruling +Gates: +Witness: all 10 array tasks deliver on their own evidence — wall inside the 6:00 budget, a result row carrying version 2026.8.17.1 and a matching target_id, `.err` free of Tracebacks, `.out` ending "Finished." with a `.completed` marker, zero "Fit Already Completed" and no overflow signature +Budget: 6:00 +Runs: 342091 +Ruling: +Lane: local-dev +Review-minutes: 20 +Epic: jax-inference-profiling +Filed: 2026-08-31 +Migrated-from: PyAutoMind/batches/2026-08-31-pm.md § member jax-inference-phase1-refs + +## Question + +The restart point the REWIND named: Phase 1, the `InferenceRefs_v1` reference baselines, redone +step by step under batch-and-review. Do the 10 cells of the refs array deliver clean reference +rows on the current stack? + +Dispatched from the laptop 2026-08-31T21:08Z as RAL job 342091, +`sbatch --array=0-8,10 --requeue` from `hpc/batch_gpu/`. Task 9 (`mge_pos`) was held back under +the MGE reuse rule and became phase 7. + +These are pre-Cortex runs, so they enter the ledger as `legacy` lines with `where:` pointing at +the active mirror tree they actually landed in (not a quarantine path) and `pulled_to:` the same. + +**No ruling yet.** R-20260901-03 spent the mesh rows' citability — a mesh row produced without +a `PositionsLH` positions penalty is not a citable reference — so this wave's mesh rows must be +checked against that before certification. The human rules the wave in the next slot. + +## Witness + +All 10 tasks COMPLETED 0:0 on euclid-ral-gpu-2, each classified DELIVERED from artefacts rather +than `sacct`: wall 10–23 % of the 6 h budget; result row with version `2026.8.17.1` and matching +`target_id`; `.err` carrying only the 222 B mask-padding UserWarning and no Traceback; `.out` +ending "Finished." with a `.completed` marker; zero "Fit Already Completed"; no overflow +signature. + +maxLL / logZ against truth 31,521.744: + +| task | cell | maxLL | logZ | +|---|---|---:|---:| +| t0 | pixelization | 29,835.4509 | 29,778.0569 | +| t1 | delaunay_nn | 30,650.8283 | 30,591.1280 | +| t2 | delaunay_nn pos-0.3 | 31,351.3909 | 31,275.2300 | +| t3 | slam_source_pix | 31,026.6354 | 30,951.7885 | +| t4 | slam_source_pix pos-auto | 31,547.2396 | 31,452.0997 | +| t5 | slam_source_pix_nn | 30,750.9893 | 30,684.1146 | +| t6 | slam_source_pix_nn pos-auto | 31,405.6282 | 31,324.8382 | +| t7 | knn | 30,062.6117 | 29,995.6676 | +| t8 | delaunay_matern | 30,675.9711 | 30,615.1603 | +| t10 | delaunay pos-auto | 31,338.4344 | 31,264.8345 | + +t4 is the only row above truth (+25.50). The stale-knn flag is CLOSED: task 7's real run +replaced the stashed 30,077.028 with 30,062.612 — 14.4 nats lower still and ~494 below the +Prodigy arm bar 30,557.03, so the knn cell genuinely underperforms. + +## Where to look + +- `inference_programme` (project row): `output/searches/nautilus/imaging/` — the 10 run trees +- `logs/output/output.342091_{0-8,10}.out` and the matching `logs/error/` files on the mirror +- `autolens_profiling/results/notes/inference/PROGRAMME.md` — the rewind restart point +- autolens_profiling#200 (queue anchor), #201 (this leg), docs-only PR #202 + +## Runs + +- 342091_[0-8,10]: legacy — gpu — submitted 2026-08-31 — wall 1:24 — pre-Cortex run, migrated; 10 tasks run serially on one A100, 22:00:39 BST 31 Aug → 08:08:54 BST 1 Sep (10 h 08 m end to end); wall shown is the longest single task. Task 9 (mge_pos) held back under the MGE reuse rule — see phase 7 + where: output/searches/nautilus/imaging + pulled_to: output/searches/nautilus/imaging + +## Ruling + +(none) diff --git a/phases/inference_programme/knn_rerun_342016_7.md b/phases/inference_programme/knn_rerun_342016_7.md new file mode 100644 index 0000000..c516365 --- /dev/null +++ b/phases/inference_programme/knn_rerun_342016_7.md @@ -0,0 +1,45 @@ +# Inference_programme — phase 3: knn reference re-run after the AdaptSplitPower stack boundary + +Project: inference_programme +Phase: 3 +State: dropped +Gates: +Witness: the re-run knn reference row replaces the withdrawn one and its logL deficit against the Prodigy arm bar is measured rather than inferred from a stale row +Budget: 6:00 +Runs: 342016 +Ruling: R-20260831-03 +Lane: local-dev +Review-minutes: 15 +Epic: jax-inference-profiling +Filed: 2026-08-30 +Migrated-from: PyAutoMind/batches/reviews/2026-08-31-am.md + +## Question + +The `knn` reference row was withdrawn as a bar on 2026-08-29 (its 480-nat deficit is the +overflow pathology at lower amplitude) and queued to re-run under the capped +`al.reg.AdaptSplitPower` target. Does the re-run close the deficit, or is the knn cell +genuinely underperforming? + +Hand-dispatched 2026-08-29 night, pre-framework; carried into the 2026-08-31-am packet. +A mesh cell, so the quarantine put it in `output/legacy_wrong/`. + +## Witness + +A fresh knn `hpc_a100_fp64_ref` result row on the current stack, comparable with the Prodigy +arm bar (30,557.03) — replacing the withdrawn one rather than being read beside it. + +## Where to look + +- `inference_programme` (project row): `output/legacy_wrong/searches/nautilus/imaging/knn/hst/hpc_a100_fp64_ref/39b5ccc228ad4b4389b1129a7e5cf605` +- `logs/output/output.342016_7.out` on the mirror +- `PyAutoMind/batches/reviews/2026-08-31-am.md` — the human's note on this member + +## Runs + +- 342016_7: legacy_wrong — gpu — submitted 2026-08-30 — wall 0:58 — pre-Cortex run, migrated + where: output/legacy_wrong/searches/nautilus/imaging/knn/hst/hpc_a100_fp64_ref/39b5ccc228ad4b4389b1129a7e5cf605 + +## Ruling + +R-20260831-03 — drop diff --git a/phases/inference_programme/mge_fp64_retro_baseline.md b/phases/inference_programme/mge_fp64_retro_baseline.md new file mode 100644 index 0000000..58adcc4 --- /dev/null +++ b/phases/inference_programme/mge_fp64_retro_baseline.md @@ -0,0 +1,54 @@ +# Inference_programme — phase 8: the retro-adopted mge_fp64 InferenceRefs_v1 baseline + +Project: inference_programme +Phase: 8 +State: accepted +Gates: +Witness: the retro-adopted mge_fp64 baseline still stands under the redo standard — no fresh `_ref` run is needed for the InferenceRefs_v1 mge_fp64 row +Budget: 6:00 +Runs: 339070 +Ruling: R-20260901-02 +Lane: local-dev +Review-minutes: 4 +Epic: jax-inference-profiling +Filed: 2026-08-24 +Migrated-from: autolens_profiling/results/notes/inference/DECISIONS.md § 2026-09-01 — Batch 2026-08-31-pm review rulings (InferenceRefs_v1 legacy-reuse members) + +## Question + +`results/baselines/InferenceRefs_v1/mge_fp64/` was adopted **retroactively** +(`certified_by: "retro"`, 2026-08-24) from an existing Nautilus run rather than a fresh +long-run re-fit. Does that adoption still stand under the redo standard, or does the rewound +programme owe the row a fresh `_ref` run? + +The run behind the row is RAL job **339070** (24 Aug 2026). Identified, not assumed: the +baseline's `reference.json` names `source_artifact: +results/searches/nautilus/imaging/mge/hst/hpc_hpc_a100_fp64.json`, and exactly one job log on +the mirror writes that file — `logs/output/output.339070.out` ("Results JSON saved to: … +hpc_hpc_a100_fp64.json"), whose `Total wall: 775.11 s` matches the baseline's +`total_wall_s: 775.11` and `sampler_wall_s: 706.50`. + +## Witness + +The adopted row reproduces the Nautilus MGE truth bar the programme used throughout Phases +0–3: `max_log_likelihood = 31786.63`, `log_evidence = 31690.50`, `likelihood_evals = 62208`, +version `2026.8.17.1`, `af.Nautilus` n_live=200 / n_batch=64. + +A future Phase-1 refresh should still run at ≥ 2× this row's `n_live` for a tighter reference +posterior (`SUBMIT_LIST.md`). + +## Where to look + +- `autolens_profiling/results/baselines/InferenceRefs_v1/mge_fp64/{README.md,reference.json,target.json}` +- `inference_programme` (project row): `output/legacy/searches/nautilus/imaging/mge/hst/hpc_a100_fp64/181b13114ba3c2298191185ff74f90d8` +- `logs/output/output.339070.out` on the mirror — the log that wrote the source artifact + +## Runs + +- 339070: legacy — gpu — submitted 2026-08-24 — wall 0:12 — pre-Cortex run, migrated + where: output/legacy/searches/nautilus/imaging/mge/hst/hpc_a100_fp64/181b13114ba3c2298191185ff74f90d8 + pulled_to: output/legacy/searches/nautilus/imaging/mge/hst/hpc_a100_fp64/181b13114ba3c2298191185ff74f90d8 + +## Ruling + +R-20260901-02 — accept diff --git a/phases/inference_programme/mge_pos_ref_reuse.md b/phases/inference_programme/mge_pos_ref_reuse.md new file mode 100644 index 0000000..20d6a9f --- /dev/null +++ b/phases/inference_programme/mge_pos_ref_reuse.md @@ -0,0 +1,53 @@ +# Inference_programme — phase 7: reuse 340210_9 as the mge_pos_fp64 InferenceRefs_v1 reference + +Project: inference_programme +Phase: 7 +State: accepted +Gates: +Witness: the quarantined-but-reusable 340210_9 run is at the same target/config as the `mge_pos_fp64` reference and can be adopted rather than resubmitted +Budget: 6:00 +Runs: 340210 +Ruling: R-20260901-01 +Lane: local-dev +Review-minutes: 5 +Epic: jax-inference-profiling +Filed: 2026-08-25 +Migrated-from: autolens_profiling/results/notes/inference/DECISIONS.md § 2026-09-01 — Batch 2026-08-31-pm review rulings (InferenceRefs_v1 legacy-reuse members) + +## Question + +The MGE reuse rule binds the redo: before submitting any MGE run, check `output/legacy/searches/` +(RAL **and** the mirror) for an existing result at the same target/config. If one is present and +has not been ruled on in a batch review packet, do not resubmit — surface it as a packet member +for a human ruling. + +`340210_9` (2026-08-25) is such a run: the `mge_pos_fp64` cell, positions on at +`tauto0.2_f1e8`, fp64. Is it adopted as the InferenceRefs_v1 `mge_pos_fp64` reference, or does +the redo need a fresh `_ref` run? + +It is why array task 9 was held back from the 342091 refs wave. + +## Witness + +The run's result row matches the `mge_pos_fp64` target/config, on a stack the redo standard +accepts — enough that adopting it costs the programme no evidence a fresh run would add. + +If accepted, the laptop action is to cite it and `mv` it from `output/legacy/searches/…` back +into the active tree on RAL and on the mirror; array task 9 stays excluded from the refs array. + +## Where to look + +- `inference_programme` (project row): `output/legacy/searches/nautilus/imaging/mge/hst/pos_tauto0.2_f1e8/hpc_a100_fp64_ref_pos_tauto0.2_f1e8/dc42087fb0524e78fd43eced7706b365` +- `logs/output/output.340210_9.out` on the mirror +- `autolens_profiling/results/notes/inference/DECISIONS.md` — the MGE reuse rule in the 2026-08-31 REWIND entry +- autolens_profiling#201 — the Phase-1 redo leg that routed this as a ruling member + +## Runs + +- 340210_9: legacy — gpu — submitted 2026-08-25 — wall 0:16 — pre-Cortex run, migrated + where: output/legacy/searches/nautilus/imaging/mge/hst/pos_tauto0.2_f1e8/hpc_a100_fp64_ref_pos_tauto0.2_f1e8/dc42087fb0524e78fd43eced7706b365 + pulled_to: output/legacy/searches/nautilus/imaging/mge/hst/pos_tauto0.2_f1e8/hpc_a100_fp64_ref_pos_tauto0.2_f1e8/dc42087fb0524e78fd43eced7706b365 + +## Ruling + +R-20260901-01 — accept diff --git a/phases/inference_programme/refs_5_6_342016_56.md b/phases/inference_programme/refs_5_6_342016_56.md new file mode 100644 index 0000000..551059e --- /dev/null +++ b/phases/inference_programme/refs_5_6_342016_56.md @@ -0,0 +1,50 @@ +# Inference_programme — phase 4: InferenceRefs_v1 rows 5 and 6 — slam_source_pix_nn, positions off and on + +Project: inference_programme +Phase: 4 +State: dropped +Gates: +Witness: InferenceRefs_v1 rows 5 and 6 (slam_source_pix_nn, positions off and pos_tauto0.2_f1e8) land clean and are certifiable as reference baselines +Budget: 6:00 +Runs: 342016 +Ruling: R-20260831-04 +Lane: local-dev +Review-minutes: 15 +Epic: jax-inference-profiling +Filed: 2026-08-30 +Migrated-from: PyAutoMind/batches/reviews/2026-08-31-am.md + +## Question + +Do InferenceRefs_v1 rows 5 and 6 — `slam_source_pix_nn` with positions off, and the same cell +under `pos_tauto0.2_f1e8` — land clean enough to be certified as reference baselines? + +Hand-dispatched 2026-08-29 night, pre-framework; carried into the 2026-08-31-am packet as +HEALTHY. Mesh rows, so the quarantine put them in `output/legacy_wrong/`. + +## Witness + +Two `_ref` result rows under `slam_source_pix_nn/hst/`, on the current stack, with matching +`target_id`s and clean `.err`/`.out`, ready for certification into +`results/baselines/InferenceRefs_v1/`. + +Ruled before certification: the rewound epic takes precedence, so refs 5, 6 and 7 were **not** +certified. + +## Where to look + +- `inference_programme` (project row): `output/legacy_wrong/searches/nautilus/imaging/slam_source_pix_nn/hst/hpc_a100_fp64_ref/7ced3989425354c2a53d4a79d7da2b11` +- `output/legacy_wrong/searches/nautilus/imaging/slam_source_pix_nn/hst/pos_tauto0.2_f1e8/hpc_a100_fp64_ref_pos_tauto0.2_f1e8/35b83ae5f6eab98575809e7c5dc5f3af` +- `logs/output/output.342016_{5,6}.out` on the mirror +- `PyAutoMind/batches/reviews/2026-08-31-am.md` — the human's note on this member + +## Runs + +- 342016_5: legacy_wrong — gpu — submitted 2026-08-29 — wall 1:21 — pre-Cortex run, migrated + where: output/legacy_wrong/searches/nautilus/imaging/slam_source_pix_nn/hst/hpc_a100_fp64_ref/7ced3989425354c2a53d4a79d7da2b11 +- 342016_6: legacy_wrong — gpu — submitted 2026-08-30 — wall 1:14 — pre-Cortex run, migrated + where: output/legacy_wrong/searches/nautilus/imaging/slam_source_pix_nn/hst/pos_tauto0.2_f1e8/hpc_a100_fp64_ref_pos_tauto0.2_f1e8/35b83ae5f6eab98575809e7c5dc5f3af + +## Ruling + +R-20260831-04 — drop diff --git a/phases/inference_programme/rewind_2026_08_31.md b/phases/inference_programme/rewind_2026_08_31.md new file mode 100644 index 0000000..2926861 --- /dev/null +++ b/phases/inference_programme/rewind_2026_08_31.md @@ -0,0 +1,68 @@ +# Inference_programme — phase 1: programme rewind — quarantine the runs, restart at Phase 1 under batch-and-review + +Project: inference_programme +Phase: 1 +State: dropped +Gates: +Witness: the mesh/pixelization evidence is out of the active tree: `output/searches/` holds only the point_source and cluster trees, on RAL and on the mirror +Budget: 6:00 +Runs: 341908, 341874 +Ruling: R-20260831-01 +Lane: local-dev +Review-minutes: 5 +Epic: jax-inference-profiling +Filed: 2026-08-31 +Migrated-from: autolens_profiling/results/notes/inference/DECISIONS.md § 2026-08-31 — Programme rewind (human directive) + +## Question + +The synthetic phase that carries the 2026-08-31 programme-wide directive. It asks nothing new: +it exists so the REWIND has a phase to be a ruling *of*, and so the two runs the directive +names by id are held in the ledger rather than only in prose. + +The directive itself: the inference programme rewinds. The epic ran largely without human +review and every mesh / pixelization result it produced goes to an unphysical solution — the +λ⁴ non-PD overflow flood (341908_5), 59 % of Phase-8B best points clustered at the |e| = +1.41421 box corner of the pixelized cells' box-clipped `ell_comps` geometry, and a +`cond ~ 1e15–1e19` NaN population. Beyond the failures, the human had lost track of what the +programme had actually established. + +MGE evidence moved to `output/legacy/` (reusable, pending batch review); mesh / pixelization +evidence moved to `output/legacy_wrong/` (quarantined, never reused or cited except as +failure-mode documentation). All shipped source and registry code stands; only run evidence +and phase/gate state rewind. Gates A, B pt 1 and B pt 2 become **provisional**. + +Follow-up filed from the same finding: the `ell_comps` box-corner clustering is +PyAutoGalaxy#594. + +## Witness + +The quarantine is executed identically on RAL (`/mnt/ral/jnightin/autolens_profiling/output/`) +and on the laptop mirror, directory structure preserved: + +- `output/legacy/searches/…` — **MGE**, reusable pending batch review: 83 run-config dirs + (nautilus 16, nss 13, nuts 2, smc 3, multi_start_prodigy 4, multi_start_prodigy_autoconv 45). +- `output/legacy_wrong/searches/…` — **mesh / pixelization**, unphysical, quarantined: 71 run-config + dirs (nautilus 21, nss 3, multi_start_prodigy 47), plus RAL-only `slogdet_ab_harvest` and + `_void_341874_24`. + +The active `output/searches/` then holds only the point_source and cluster trees. `hpc/sync pull` +scopes to `output/searches` and passes no `--delete`, so it can never reach into `legacy*/`. + +## Where to look + +- `autolens_profiling/results/notes/inference/DECISIONS.md` — the 2026-08-31 REWIND entry (the directive of record) +- `autolens_profiling/results/notes/inference/PROGRAMME.md` §"2026-08-31 REWIND" + the annotated phase/gate table +- `inference_programme` (project row): `output/legacy/searches/` and `output/legacy_wrong/searches/` +- autolens_profiling#200 — the queue-of-record anchor for the redo + +## Runs + +- 341908_5: legacy_wrong — gpu — submitted 2026-08-27 — wall 0:00 — pre-Cortex run, migrated; killed by the likelihood-overflow flood — checkpoint.hdf5 is the only artifact it left + where: output/legacy_wrong/searches/nautilus/imaging/slam_source_pix_nn/hst/hpc_a100_fp64_ref/c16ff7b02fc168edd8eb9b4aaa4f4bc7 +- 341874_24: legacy_wrong — gpu — submitted 2026-08-28 — wall 2:00 — pre-Cortex run, migrated; the RAL-side copy is quarantined as _void_341874_24 + where: output/legacy_wrong/searches/multi_start_prodigy/imaging/knn/hst/n16_s3000_seed4/knn_auto_none_seed4 + +## Ruling + +R-20260831-01 — drop diff --git a/phases/inference_programme/slogdet_ab_342017.md b/phases/inference_programme/slogdet_ab_342017.md new file mode 100644 index 0000000..8d76182 --- /dev/null +++ b/phases/inference_programme/slogdet_ab_342017.md @@ -0,0 +1,51 @@ +# Inference_programme — phase 5: slogdet A/B — Cholesky vs slogdet log-det on slam_source_pix_nn (overflow-flood rung 4) + +Project: inference_programme +Phase: 5 +State: dropped +Gates: +Witness: the Cholesky and slogdet log-det arms of the same slam_source_pix_nn cell differ (or do not) by more than the run-to-run scatter, giving a verdict on the log-det route +Budget: 6:00 +Runs: 342017 +Ruling: R-20260831-05 +Lane: local-dev +Review-minutes: 12 +Epic: jax-inference-profiling +Filed: 2026-08-30 +Migrated-from: PyAutoMind/batches/reviews/2026-08-31-am.md + +## Question + +Rung 4 of the overflow-flood fix wave: does swapping the regularization log-determinant from a +Cholesky factorisation to `slogdet` change the fit of the same `slam_source_pix_nn` cell by +more than run-to-run scatter — and is either arm immune to the non-PD flood? + +Hand-dispatched 2026-08-29 night, pre-framework; carried into the 2026-08-31-am packet as +HEALTHY. Mesh rows, so the quarantine put them in `output/legacy_wrong/`. + +## Witness + +Two paired result rows, `ld_cholesky` and `ld_slogdet`, on the same cell and stack, whose +logZ / maxLL difference is readable against the arm's own scatter — enough for a written +slogdet verdict. + +Ruled before the verdict was written: the rewound epic takes precedence, so no slogdet verdict +was recorded. + +## Where to look + +- `inference_programme` (project row): `output/legacy_wrong/searches/nautilus/imaging/slam_source_pix_nn/hst/ld_cholesky/…` +- `output/legacy_wrong/searches/nautilus/imaging/slam_source_pix_nn/hst/ld_slogdet/…` +- `logs/output/output.342017_{0,1}.out` on the mirror; RAL-only `slogdet_ab_harvest` +- `PyAutoMind/batches/reviews/2026-08-31-am.md` — the human's note on this member + +## Runs + +- 342017_0: legacy_wrong — gpu — submitted 2026-08-30 — wall 1:23 — pre-Cortex run, migrated + where: output/legacy_wrong/searches/nautilus/imaging/slam_source_pix_nn/hst/ld_cholesky/hpc_a100_fp64_slogdet_ab_cholesky/ae454f54e14847ca62929c70033c11a5 +- 342017_1: legacy_wrong — gpu — submitted 2026-08-30 — wall 1:14 — pre-Cortex run, migrated + where: output/legacy_wrong/searches/nautilus/imaging/slam_source_pix_nn/hst/ld_slogdet/hpc_a100_fp64_slogdet_ab_slogdet/7eb80135de1db2112c5bb5383b32bc49 + +## Ruling + +R-20260831-05 — drop diff --git a/phases/inference_programme/smc_probe_342018.md b/phases/inference_programme/smc_probe_342018.md new file mode 100644 index 0000000..d00bb65 --- /dev/null +++ b/phases/inference_programme/smc_probe_342018.md @@ -0,0 +1,52 @@ +# Inference_programme — phase 2: SMC probe on MGE — warm-start MALA, warm-start HMC, cold MALA + +Project: inference_programme +Phase: 2 +State: dropped +Gates: +Witness: the SMC arm's warm-start MALA/HMC rows on MGE reach the Nautilus logZ bar and the acceptance rates are ingestable as a PROGRAMME Phase-7 readout +Budget: 6:00 +Runs: 342018 +Ruling: R-20260831-02 +Lane: local-dev +Review-minutes: 25 +Epic: jax-inference-profiling +Filed: 2026-08-30 +Migrated-from: PyAutoMind/batches/reviews/2026-08-31-am.md + +## Question + +Does the `af.SMC` arm — warm-start MALA, warm-start HMC and cold MALA at n=256 on the MGE +cell — reach the Nautilus bar, and are its acceptance rates worth ingesting as a PROGRAMME +Phase-7 readout? + +Hand-dispatched on 2026-08-29 night, before the batch framework shipped; carried into the +2026-08-31-am packet as a review member. The runs are MGE, so the quarantine put them in +`output/legacy/` (reusable) rather than `output/legacy_wrong/`. + +## Witness + +The three SMC rows write result JSONs under `output/legacy/searches/smc/imaging/mge/hst/` +with acceptance rates and logZ readable against the Nautilus MGE truth bar. + +Ruled before the readout was ingested: the rewound epic takes precedence, so no rates were +ingested and no ledger write was made. + +## Where to look + +- `inference_programme` (project row): `output/legacy/searches/smc/imaging/mge/hst/` +- `logs/output/output.342018_{0,1,2}.out` on the mirror +- `PyAutoMind/batches/reviews/2026-08-31-am.md` — the human's note on this member + +## Runs + +- 342018_0: legacy — gpu — submitted 2026-08-30 — wall 0:10 — pre-Cortex run, migrated + where: output/legacy/searches/smc/imaging/mge/hst/n256_mala_m5_seed0_warmebe2e28d_massprior_scaled/hpc_a100_fp64_n256_mala_warm/7fe1b8d8baee5edc4a5c485a8d48b45b +- 342018_1: legacy — gpu — submitted 2026-08-30 — wall 0:12 — pre-Cortex run, migrated + where: output/legacy/searches/smc/imaging/mge/hst/n256_hmc_m5_i8_seed0_warmebe2e28d_massprior_scaled/hpc_a100_fp64_n256_hmc_warm/3b4ab5e2d6ff3b453680eee171bb6f4b +- 342018_2: legacy — gpu — submitted 2026-08-30 — wall 0:31 — pre-Cortex run, migrated + where: output/legacy/searches/smc/imaging/mge/hst/n256_mala_m5_seed0_cold/hpc_a100_fp64_n256_mala_cold/1e3dea2f7a4d385eac93f5bbc80de7cf + +## Ruling + +R-20260831-02 — drop diff --git a/phases/slope_hierarchy/n25_scale_up.md b/phases/slope_hierarchy/n25_scale_up.md new file mode 100644 index 0000000..572dda0 --- /dev/null +++ b/phases/slope_hierarchy/n25_scale_up.md @@ -0,0 +1,84 @@ +# Slope_hierarchy — phase 1: scale the hierarchical slope recovery to N=25–50 + +Project: slope_hierarchy +Phase: 1 +State: planned +Gates: +Witness: an N=25–50 parity table in the same shape as the N=5 one, committed under `results/`, plus the scaling measurements (VRAM ceiling, JAX compile time vs model size, sampler ladder at 3N+1 dims, gradient-utilisation sanity) +Budget: 48:00 +Runs: +Ruling: +Lane: local-dev +Review-minutes: 20 +Epic: graphical-ep +Filed: 2026-07-22 +Migrated-from: PyAutoMind/draft/research/graphical_ep/slope_hierarchy_n25_scale_up.md + +## Question + +Ready when: graphical-ep phase 2 (Mind) is issued — add its ref to `Gates:` and move to `gated`. + +The `slope_hierarchy` project answered all four of its goals at N=5 and was wrapped up on +2026-07-22. This is the *optional* scale-up that was left on the table, not a blocker or an open +defect. Final N=5 parity (converged, `results/ep_history_n5_maxsteps12/`): + +| parent | truth | draws | NUTS | EP (converged) | +|---|---|---|---|---| +| mean | 2.0 | 2.023 | 2.028 [2.000, 2.063] | 2.051 ± 0.0001 | +| sigma | 0.1 | 0.099 | 0.143 [0.117, 0.185] | 0.026 ± 0.00001 | + +Does the picture hold at survey-relevant N? + +- Does **NUTS** stay the trustworthy method for the parent scatter as N grows — and does the + scatter estimate tighten toward truth 0.1 the way more data should make it? +- Does **EP**'s ~4×-low scatter get *better* with N (more groups → more information in the + parent factor) or *worse* (more per-group messages to over-shrink)? Either answer is + publishable: the framing is **NUTS headline, EP cautionary**, and the EP arm is here to + characterise the failure mode at scale, not to be rescued. +- Does the collapse basin documented in PyAutoFit#1405 show up more or less often at larger N? + +**The traps** (carried from the retired active.md entry): edit the simulator's `N` and +re-simulate; submit with the `submit_*` scripts using `--array`; `rm output//*` before +refits, or stale output silently resumes and returns the old answer; force-sync the truth files +to the cluster, or the recovery scoring goes quietly wrong; verify the RAL PyAutoFit mirror +commit before trusting a run (`HPCPullPyAuto`); `export JAX_ENABLE_X64=True` explicitly in the +sbatch script — it is ambient locally but **not** inherited by `sbatch`, and float32 silently +ruins a gradient run (verify `grep -c "truncated to dtype float32" *.err` == 0); the repo is on +`autonerves`. + +Cost note: RAL GPU contention has been severe (multi-day queues as of 2026-07-20). N=25–50 × +per-lens fits is a large array job — check the queue before committing to it, and consider the +CPU `ral` partition for the EP arm (finiteness, not throughput). + +## Witness + +An N=25–50 parity table in the same shape as the N=5 one, committed under `results/`, plus an +issue comment on slope_hierarchy#1. + +Scaling measurements to capture while the runs are up anyway (the EP campaign needs them): + +- **VRAM ceiling** — peak GPU memory vs N on the A100 arm; the N at which the joint fit no + longer fits. +- **JAX compile time vs model size** — wall time of the first likelihood/grad call vs N; is + compilation a fixed cost worth caring about at N=100+? (This is the *graphical factor-graph* + trace, a different graph from the lensing-side compile-time work.) +- **Sampler ladder at high dimension** — at 3N+1 ≈ 76–151 dims, time at least one alternative + gradient sampler beside NUTS on the same problem. +- **Gradient utilisation sanity** — confirm the run exercises the JAX gradients end to end (x64 + on, no silent float32, no numpy fallback in the hot path). + +Each number goes in the committed results table beside the parity numbers. + +## Where to look + +- `slope_hierarchy` (project row): `results/ep_history_n5_maxsteps12/` — the N=5 baseline +- Jammy2211/slope_hierarchy#1 — the project issue the parity table is commented on +- Mind `research/graphical_ep/ep_campaign.md` phase 3 — the campaign rows this feeds +- Mind `draft/research/graphical_ep/slope_hierarchy_methods_writeup.md` — the write-up it feeds +- PyAutoFit#1405 — the collapse basin + +## Runs + +## Ruling + +(none) diff --git a/phases/subhalo_validation/delaunay_adapt_split_pl_eff_0.md b/phases/subhalo_validation/delaunay_adapt_split_pl_eff_0.md new file mode 100644 index 0000000..f166142 --- /dev/null +++ b/phases/subhalo_validation/delaunay_adapt_split_pl_eff_0.md @@ -0,0 +1,53 @@ +# Subhalo_validation — phase 2: delaunay_adapt_split on pl_eff_0 — the clumpy-source lens + +Project: subhalo_validation +Phase: 2 +State: awaiting-ruling +Gates: +Witness: evidence_increase < 5 in results/delaunay_adapt_split/pl_eff_0_no_subhalo.json (test_mode false) +Budget: 48:00 +Runs: 342027 +Ruling: R-20260831-07 +Lane: local-dev +Review-minutes: 8 +Epic: +Filed: 2026-08-29 +Migrated-from: PyAutoMind/batches/reviews/2026-08-31-am.md + +## Question + +Does `delaunay_adapt_split` report a false subhalo detection on `pl_eff_0` — six +`ElsonFreeFall` clumps inside the tangential caustic, quadruply imaged, and no subhalo? + +Three simulated HST-like lenses (0.05" pixels, 301×301, PSF sigma 0.05", 300 s), identical +`PowerLaw` + shear mass and bulge+disk lens light, differing only in the source. The pipeline is +the full SLaM chain (source_lp → source_pix → light → mass → subhalo grid search), source mesh +at 1250 pixels, subhalo grid 3.0" across 2 steps per axis, then a single-plane refine. + +Every dataset contains **no** subhalo, so a large `evidence_increase` is a false positive; +≳ 5 is conventionally a detection. + +## Witness + +`evidence_increase < 5` in `results/delaunay_adapt_split/pl_eff_0_no_subhalo.json`, with +`test_mode: false`. + +Landed 2026-08-31 19:04 local (2026-09-01 00:04 BST): **−0.2425** (logL +0.6093, M200 1.213e8 +M⊙ at (−1.3016, −2.5132)"). Under the < 5 null. The witness came from `subhalo[2]` and +predates the `subhalo_stage` key. + +## Where to look + +- `subhalo_validation` (project row): `output/subhalo/detect/delaunay_adapt_split/pl_eff_0_no_subhalo/` +- `results/delaunay_adapt_split/pl_eff_0_no_subhalo.json` — the witness +- `wiki/project/results_summary.md`, `wiki/project/state.md` + +## Runs + +- 342027_1: legacy — ral — submitted 2026-08-29 — wall 45:10 — pre-Cortex run, migrated; COMPLETED 0:0 in 1-21:10:24, ended 00:04:55 BST 2026-09-01 — inside the 2-day wall after all; witness predates the subhalo_stage key + where: output/subhalo/detect/delaunay_adapt_split/pl_eff_0_no_subhalo/ + pulled_to: output/subhalo/detect/delaunay_adapt_split/pl_eff_0_no_subhalo/ + +## Ruling + +R-20260831-07 — leave-to-finish diff --git a/phases/subhalo_validation/delaunay_adapt_split_pl_eff_1_outer.md b/phases/subhalo_validation/delaunay_adapt_split_pl_eff_1_outer.md new file mode 100644 index 0000000..19d4eb2 --- /dev/null +++ b/phases/subhalo_validation/delaunay_adapt_split_pl_eff_1_outer.md @@ -0,0 +1,75 @@ +# Subhalo_validation — phase 3: delaunay_adapt_split on pl_eff_1_outer — the outer-component lens, plus the AdaptSplit fix rerun + +Project: subhalo_validation +Phase: 3 +State: awaiting-ruling +Gates: +Witness: evidence_increase < 5 in results/delaunay_adapt_split/pl_eff_1_outer_no_subhalo.json (test_mode false) +Budget: 48:00 +Runs: 342027, 342093 +Ruling: R-20260831-08 +Lane: local-dev +Review-minutes: 8 +Epic: +Filed: 2026-08-29 +Migrated-from: PyAutoMind/batches/reviews/2026-08-31-am.md; PyAutoMind/active/follow_up_wave_adapt_split_and_rectangular.md + +## Question + +Two questions on one lens — `pl_eff_1_outer`, which is `pl_eff_0` plus a compact component at +(0.0, −0.4) outside the tangential caustic, doubly imaged at x ≈ +1.5" / −2.3". + +1. Does `delaunay_adapt_split` report a false subhalo detection on it (`342027_2`)? +2. Was the poor `pix[1]` solution the am review flagged caused by `ConstantSplit` + regularization at `source_pix[1]`? The `_adapt_split_fix` rerun (`342093_0`) swaps it for + `AdaptSplit` on the same lens, same mesh, same free parameters — the human inspects the + reconstruction and decides the default recipe's `pix[1]` regularization from it. Nothing + downstream moves until that judgement lands. + +Three simulated HST-like lenses (0.05" pixels, 301×301, PSF sigma 0.05", 300 s), identical +`PowerLaw` + shear mass and bulge+disk lens light, differing only in the source. The pipeline is +the full SLaM chain (source_lp → source_pix → light → mass → subhalo grid search), source mesh +at 1250 pixels, subhalo grid 3.0" across 2 steps per axis, then a single-plane refine. + +Every dataset contains **no** subhalo, so a large `evidence_increase` is a false positive; +≳ 5 is conventionally a detection. + +## Witness + +`evidence_increase < 5` in `results/delaunay_adapt_split/pl_eff_1_outer_no_subhalo.json`, with +`test_mode: false`. + +Landed 2026-08-31 20:59 local (2026-09-01 01:59 BST): **+2.7361** (logL +15.3716, M200 8.196e8 +M⊙ at (1.4117, 1.4193)"). Under the < 5 null, but the least emphatic of the three — +2.7361 +against a +15.37 likelihood increase — and the largest false positive on the board so far. + +The `_adapt_split_fix` rerun has no witness JSON by design (`--stop_after=source_pix_1`). Its +readout, like for like against the ConstantSplit baseline `source_pix[1]` (same lens, 600-px +Delaunay, `areas_factor` 0.5, 10 free parameters, same dataset): log evidence **18,312.473 → +19,390.673 (+1,078.20)**, maxLL **18,377.142 → 19,462.741 (+1,085.60)**; regularization goes +from a flat 1.321 to `inner_coefficient` 1.0e-4 / `outer_coefficient` 79.76 (3σ 44.3–113.6) / +`signal_scale` 0.0164. The compact source is **not** smoothed away. Caveat, recorded honestly: +the coherent ring residuals persist in *both* fits at comparable amplitude (±7.40 → ±7.95; +chi² map max 54.69 → 63.20), so the +1,078 nats come from the adaptive smoothing prior, not +from resolving the ring residuals away. + +## Where to look + +- `subhalo_validation` (project row): `output/subhalo/detect/delaunay_adapt_split/pl_eff_1_outer_no_subhalo/` +- `output/subhalo/detect/delaunay_adapt_split/pl_eff_1_outer_no_subhalo_adapt_split_fix/source_pix[1]/085f85ba6970436a1b8b115b6c9b3fbf` +- `results/delaunay_adapt_split/pl_eff_1_outer_no_subhalo.json` — the witness +- `results/figures/adapt_split_fix_{regularization,source_plane,residuals}_before_after.jpg` +- `wiki/project/2026-08-31-adapt-split-fix-and-rectangular.md`, `wiki/project/state.md` + +## Runs + +- 342027_2: legacy — ral — submitted 2026-08-29 — wall 47:05 — pre-Cortex run, migrated; COMPLETED 0:0 in 1-23:05:05, ended 01:59:36 BST 2026-09-01 with 55 min of wall to spare; witness predates the subhalo_stage key + where: output/subhalo/detect/delaunay_adapt_split/pl_eff_1_outer_no_subhalo/ + pulled_to: output/subhalo/detect/delaunay_adapt_split/pl_eff_1_outer_no_subhalo/ +- 342093_0: legacy — ral — submitted 2026-08-31 — wall 1:16 — pre-Cortex run, migrated; the source_pix[1] AdaptSplit rerun the am review asked for — recipe delaunay_adapt_split_fix, --output_suffix=_adapt_split_fix --stop_after=source_pix_1, so it writes no witness JSON by design; awaiting the human's judgement + where: output/subhalo/detect/delaunay_adapt_split/pl_eff_1_outer_no_subhalo_adapt_split_fix/ + pulled_to: output/subhalo/detect/delaunay_adapt_split/pl_eff_1_outer_no_subhalo_adapt_split_fix/ + +## Ruling + +R-20260831-08 — leave-to-finish diff --git a/phases/subhalo_validation/delaunay_adapt_split_pl_sersic_0.md b/phases/subhalo_validation/delaunay_adapt_split_pl_sersic_0.md new file mode 100644 index 0000000..bbbdf77 --- /dev/null +++ b/phases/subhalo_validation/delaunay_adapt_split_pl_sersic_0.md @@ -0,0 +1,58 @@ +# Subhalo_validation — phase 1: delaunay_adapt_split on pl_sersic_0 — the false-positive null + +Project: subhalo_validation +Phase: 1 +State: accepted +Gates: +Witness: evidence_increase < 5 in results/delaunay_adapt_split/pl_sersic_0_no_subhalo.json (test_mode false) +Budget: 48:00 +Runs: 342027 +Ruling: R-20260831-06 +Lane: local-dev +Review-minutes: 10 +Epic: +Filed: 2026-08-29 +Migrated-from: PyAutoMind/batches/reviews/2026-08-31-am.md + +## Question + +Does the `delaunay_adapt_split` SLaM chain report a false subhalo detection on `pl_sersic_0`, +a simulated lens with a single `SersicCore` source and no subhalo? + +Three simulated HST-like lenses (0.05" pixels, 301×301, PSF sigma 0.05", 300 s), identical +`PowerLaw` + shear mass and bulge+disk lens light, differing only in the source. The pipeline is +the full SLaM chain (source_lp → source_pix → light → mass → subhalo grid search), source mesh +at 1250 pixels, subhalo grid 3.0" across 2 steps per axis, then a single-plane refine. + +Every dataset contains **no** subhalo, so a large `evidence_increase` is a false positive; +≳ 5 is conventionally a detection. + +## Witness + +`evidence_increase < 5` in `results/delaunay_adapt_split/pl_sersic_0_no_subhalo.json`, with +`test_mode: false`. + +Landed: **−0.3009** — a textbook null. The subhalo buys +0.9668 nats of likelihood for 3 extra +parameters and the evidence falls; the nuisance subhalo parks at (2.1457, −0.2410)", outside +the ring, M200 1.493e9 M⊙. All four subhalo[1] grid cells sit below the no-subhalo baseline +(−0.60 to −0.79) — a uniform null. + +Measured stage costs (the reason subhalo[2] was later switched off): `source_lp[1]` 1 h 10 m, +`source_pix[1]` 29 min, `source_pix[2]` 10 min, `light[1]` 74 min, `mass_total[1]` 168 min, +`subhalo[1]` grid 21.2 h, `subhalo[2]` refine 8.25 h — 29.4 h of the 34 h chain. + +## Where to look + +- `subhalo_validation` (project row): `output/subhalo/detect/delaunay_adapt_split/pl_sersic_0_no_subhalo/` +- `results/delaunay_adapt_split/pl_sersic_0_no_subhalo.json` — the witness +- `wiki/project/results_summary.md` — the scientific commentary and the provenance row + +## Runs + +- 342027_0: legacy — ral — submitted 2026-08-29 — wall 34:08 — pre-Cortex run, migrated; MaxRSS 27.0 GB, autolens 2026.8.17.1 + PyAutoFit 56fb8b63b (the PR #1548 pool fix); witness predates the subhalo_stage key — read it as subhalo[2]_[single_plane_refine] + where: output/subhalo/detect/delaunay_adapt_split/pl_sersic_0_no_subhalo/ + pulled_to: output/subhalo/detect/delaunay_adapt_split/pl_sersic_0_no_subhalo/ + +## Ruling + +R-20260831-06 — accept diff --git a/phases/subhalo_validation/rectangular_adapt_pl_sersic_0.md b/phases/subhalo_validation/rectangular_adapt_pl_sersic_0.md new file mode 100644 index 0000000..8aa2944 --- /dev/null +++ b/phases/subhalo_validation/rectangular_adapt_pl_sersic_0.md @@ -0,0 +1,74 @@ +# Subhalo_validation — phase 4: rectangular_adapt on pl_sersic_0 — the RectangularBilinear comparison + +Project: subhalo_validation +Phase: 4 +State: awaiting-ruling +Gates: +Witness: evidence_increase < 5 in results/rectangular_adapt/pl_sersic_0_no_subhalo.json (test_mode false) +Budget: 48:00 +Runs: 342094, 342095 +Ruling: +Lane: local-dev +Review-minutes: 15 +Epic: +Filed: 2026-08-31 +Migrated-from: PyAutoMind/active/follow_up_wave_adapt_split_and_rectangular.md + +## Question + +The mesh/regularization comparison the am review asked for: does `RectangularBilinear` (recipe +`rectangular_adapt`) behave like `delaunay_adapt_split` on the lens that already gave a clean +null, `pl_sersic_0`? + +Run as the mandatory two-job split — job A (`342094_0`) refits `source_lp[1]` under JAX in the +rectangular tree, job B (`342095_0`) runs the numba chain `afterok`. It is the first run with +`subhalo[2]` gated off behind `--subhalo_refine`, so its witness comes from the `subhalo[1]` +grid rather than a refine and is **not** like-for-like with the three delaunay rows. + +The second `rectangular_adapt` lens stays PARKED: which `pl_eff` (if either) goes next is a +human decision that follows the rulings on phases 2 and 3. + +Three simulated HST-like lenses (0.05" pixels, 301×301, PSF sigma 0.05", 300 s), identical +`PowerLaw` + shear mass and bulge+disk lens light, differing only in the source. The pipeline is +the full SLaM chain (source_lp → source_pix → light → mass → subhalo grid search), source mesh +at 1250 pixels, subhalo grid 3.0" across 2 steps per axis, then a single-plane refine. + +Every dataset contains **no** subhalo, so a large `evidence_increase` is a false positive; +≳ 5 is conventionally a detection. + +## Witness + +`evidence_increase < 5` in `results/rectangular_adapt/pl_sersic_0_no_subhalo.json`, with +`test_mode: false`. + +Landed 2026-09-01 12:54 local (17:54 BST): **+0.2538** (logL +4.1732, M200 2.360e10 M⊙ at +(1.5843, −0.0448)"), `subhalo_stage: subhalo[1]_[search_lens_plane]` — the first grid-derived +witness in the project. Under the < 5 null. + +Its results JSON is deliberately **untracked** in git (the human's ruling), so it lives only on +the laptop at `results/rectangular_adapt/pl_sersic_0_no_subhalo.json`. + +Indicative only, not a like-for-like comparison: `source_pix[2]` confirms +`RectangularBilinearAdaptImage` 32×32 + Adapt (N=3) at evidence 21,418.501 against the delaunay +column's 21,386.823 — different mesh size and free-parameter count. + +## Where to look + +- `subhalo_validation` (project row): `output/subhalo/detect/rectangular_adapt/pl_sersic_0_no_subhalo/` +- `output/subhalo/detect/rectangular_adapt/pl_sersic_0_no_subhalo/source_lp[1]/3e9ced20d8180a15d972b1874636c5c1` — job A +- `results/rectangular_adapt/pl_sersic_0_no_subhalo.json` — the witness (untracked; laptop only) +- `wiki/project/2026-08-31-adapt-split-fix-and-rectangular.md`, `wiki/project/results_summary.md` + +## Runs + +- 342094_0: legacy — ral — submitted 2026-08-31 — wall 1:44 — pre-Cortex run, migrated; job A, JAX source_lp[1] under the rectangular tree (4c / 32gb / 12 h) — COMPLETED 0:0 in 01:44:40, ended 23:52:17 BST + where: output/subhalo/detect/rectangular_adapt/pl_sersic_0_no_subhalo/ + pulled_to: output/subhalo/detect/rectangular_adapt/pl_sersic_0_no_subhalo/ +- 342095_0: legacy — ral — submitted 2026-08-31 — wall 18:01 — pre-Cortex run, migrated; job B, the numba chain (8c / 96gb / 48 h) with subhalo[2] off — witness landed 2026-09-01 12:54 local (17:54 BST) + where: output/subhalo/detect/rectangular_adapt/pl_sersic_0_no_subhalo/ + after: 342094_0 + pulled_to: output/subhalo/detect/rectangular_adapt/pl_sersic_0_no_subhalo/ + +## Ruling + +(none) diff --git a/rulings/2026/08/R-20260831-01.md b/rulings/2026/08/R-20260831-01.md new file mode 100644 index 0000000..301c73d --- /dev/null +++ b/rulings/2026/08/R-20260831-01.md @@ -0,0 +1,100 @@ +# R-20260831-01 — drop inference_programme phase 1 + +Project: inference_programme +Phase: phases/inference_programme/rewind_2026_08_31.md +Runs: 341908, 341874 +Ruling: drop +Reviewed-at: 2026-08-31T13:34Z +Follow-ups: PyAutoGalaxy#594 +Migrated-from: autolens_profiling/results/notes/inference/DECISIONS.md § 2026-08-31 — Programme rewind (human directive) + +## Ruling + +**Decision (human, 2026-08-31):** the inference programme **rewinds**. The epic +ran largely without human review, and every mesh / pixelization result it +produced goes to an unphysical solution. This ledger already carries the three +symptoms: the λ⁴ non-PD **overflow flood** (the regularization matrix goes non-PD +from c~1e4, fp64 Cholesky returns finite garbage, and Nautilus accepts `log_l` up +to **3e+303** as a best point — the 341908_5 entry); **59 %** of Phase-8B best +points (23 of 39) clustered at the **|e| = 1.41421** box corner of the pixelized +cells' box-clipped `ell_comps` geometry (W10); and a `cond ~ 1e15–1e19` NaN +population where the reconstruction solve NaNs despite finite log-det terms +(CP-4/W7). Beyond the failures, the human had lost track of what the programme +had actually established. The rewind quarantines the bad evidence and restarts +the review discipline. + +**What moved where (executed 2026-08-31, identically on RAL +`/mnt/ral/jnightin/autolens_profiling/output/` and the laptop mirror +`/mnt/c/Users/Jammy/Science/inference_programme/output/`, directory structure +preserved):** + +| Tree | Contents | Run-config dirs | +|---|---|---| +| `output/legacy/searches/…` | **MGE** — reusable, pending batch review | **83**: nautilus 16, nss 13, nuts 2, smc 3, multi_start_prodigy 4, multi_start_prodigy_autoconv 45 | +| `output/legacy_wrong/searches/…` | **mesh / pixelization** — unphysical, quarantined | **71**: nautilus 21 (delaunay, delaunay_matern, delaunay_nn, knn, pixelization, slam_source_pix, slam_source_pix_nn), nss 3, multi_start_prodigy 47 (delaunay_adapt_split, knn); plus RAL-only `slogdet_ab_harvest` and `_void_341874_24` | + +The active `output/searches/` now holds only the **point_source** and **cluster** +trees. `hpc/sync pull` scopes to `output/searches` and passes no `--delete`, so it +can never reach into `legacy*/`. + +**Restart point: Phase 1 — the `InferenceRefs_v1` reference baselines**, the +first phase whose design does RAL runs. Phase 0 outcomes stand, and **all shipped +source and registry code stands** — the PositionsLH fix (PyAutoLens#699/#700), +blackjax 1.6.2 on cloud/local/RAL, `_targets.py`, benchmark schema v2, `af.SMC`, +the NUTS plumbing. Nothing is reverted. Only **run evidence** and **phase/gate +state** rewind. Re-execution is human-driven and step-by-step under the +batch-and-review workflow: human-declared `review-at:` slots, review packets at +`PyAutoMind/batches/packets/` (spec `batches/packets/TEMPLATE.md`), with the +laptop mirror as the review surface. + +**Gates → PROVISIONAL.** Each call below stands as a *working assumption* and may +continue to be used as one, but the legacy MGE evidence beneath it must be +surfaced in batch review packets and ruled on by a human before the gate is +re-confirmed: + +- **Gate A** (2026-08-24 — Nautilus stays the nested baseline; `af.NSS` is the + tuned alternative) → **provisional**. Re-confirmation needs the Phase-2 inner-steps + / logZ-bias scan reviewed in packets; note its Delaunay comparison re-baselines + now sit in `legacy_wrong` and cannot be cited. +- **Gate B pt 1** (2026-08-23 — Prodigy n=256, prior_box, autoconv, positions-off + ratified as global MAP searcher on MGE) → **provisional**. Re-confirmation needs + the Phase-3 reliability wave (p̂_hit, fresh-seed tiers) reviewed in packets. +- **Gate B pt 2** (2026-08-27 — PositionsLH safe for gradient MAP at factor ≤ 1e5 + on MGE, 1e8 rejected) → **provisional**. Re-confirmation needs the Phase-4 + Stage 2/3 arms reviewed in packets, with the six caveats that ride with the call + restated rather than assumed. + +Gates C–F remain open and unstarted. + +**MGE reuse rule (binding for the redo).** Before submitting any MGE run during +the redo, check `output/legacy/searches/…` (RAL **and** the mirror) for an +existing result at the same target/config. If one is present and has not yet been +ruled on in a batch review packet, **do not resubmit** — surface it as a packet +member (question → witness → health evidence → readout → where-to-look) for a +human ruling. Accepted → cite it and `mv` it back into the active tree; rejected → +queue a fresh run. `legacy_wrong` results are **never** reused or cited except as +failure-mode documentation. + +**Resume-trap note.** The quarantine doubles as the mandated resume-trap +relocation. With the old run directories off the active path, a redo run launched +with identical knobs starts fresh instead of silently re-emitting the old fit — +reuse means *reading from* `legacy/`, never resuming in place. + +**Records:** autolens_profiling#200 (queue-of-record anchor for the redo — redo +slices ship as autolens_profiling issues/PRs per the epics rule); +`PROGRAMME.md` §"2026-08-31 REWIND" + the annotated phase/gate table; +`PyAutoMind/epics.md` (JAX profiling epic notes). + +--- + +Follow-up filed from the same finding (2026-09-01): the 59 % Phase-8B clustering at the +|e| = 1.41421 box corner of the pixelized cells' box-clipped `ell_comps` geometry is +PyAutoGalaxy#594 — the joint disk constraint on `ell_comps`, with the 8B crash channel +PyAutoFit#1535 beside it. + +## Evidence + +- `autolens_profiling/results/notes/inference/DECISIONS.md` — the 2026-08-31 REWIND entry (the directive of record) +- `autolens_profiling/results/notes/inference/PROGRAMME.md` §"2026-08-31 REWIND" + the annotated phase/gate table +- `inference_programme` (project row): `output/legacy/searches/` and `output/legacy_wrong/searches/` +- autolens_profiling#200 — the queue-of-record anchor for the redo diff --git a/rulings/2026/08/R-20260831-02.md b/rulings/2026/08/R-20260831-02.md new file mode 100644 index 0000000..ce5a36c --- /dev/null +++ b/rulings/2026/08/R-20260831-02.md @@ -0,0 +1,19 @@ +# R-20260831-02 — drop inference_programme phase 2 + +Project: inference_programme +Phase: phases/inference_programme/smc_probe_342018.md +Runs: 342018 +Ruling: drop +Batch: 2026-08-31-am +Reviewed-at: 2026-08-31T13:34Z +Migrated-from: PyAutoMind/batches/reviews/2026-08-31-am.md + +## Ruling + +I have updated the JAX gradient epic to do alot of phases back from the start, so wee will basically discard this and start again. Dont change anything just let that recent intake that precendent. + +## Evidence + +- `inference_programme` (project row): `output/legacy/searches/smc/imaging/mge/hst/` +- `logs/output/output.342018_{0,1,2}.out` on the mirror +- `PyAutoMind/batches/reviews/2026-08-31-am.md` — the human's note on this member diff --git a/rulings/2026/08/R-20260831-03.md b/rulings/2026/08/R-20260831-03.md new file mode 100644 index 0000000..d28e1af --- /dev/null +++ b/rulings/2026/08/R-20260831-03.md @@ -0,0 +1,19 @@ +# R-20260831-03 — drop inference_programme phase 3 + +Project: inference_programme +Phase: phases/inference_programme/knn_rerun_342016_7.md +Runs: 342016 +Ruling: drop +Batch: 2026-08-31-am +Reviewed-at: 2026-08-31T13:34Z +Migrated-from: PyAutoMind/batches/reviews/2026-08-31-am.md + +## Ruling + +I have updated the JAX gradient epic to do alot of phases back from the start, so wee will basically discard this and start again. Dont change anything just let that recent intake that precendent. + +## Evidence + +- `inference_programme` (project row): `output/legacy_wrong/searches/nautilus/imaging/knn/hst/hpc_a100_fp64_ref/39b5ccc228ad4b4389b1129a7e5cf605` +- `logs/output/output.342016_7.out` on the mirror +- `PyAutoMind/batches/reviews/2026-08-31-am.md` — the human's note on this member diff --git a/rulings/2026/08/R-20260831-04.md b/rulings/2026/08/R-20260831-04.md new file mode 100644 index 0000000..a2585b5 --- /dev/null +++ b/rulings/2026/08/R-20260831-04.md @@ -0,0 +1,20 @@ +# R-20260831-04 — drop inference_programme phase 4 + +Project: inference_programme +Phase: phases/inference_programme/refs_5_6_342016_56.md +Runs: 342016 +Ruling: drop +Batch: 2026-08-31-am +Reviewed-at: 2026-08-31T13:34Z +Migrated-from: PyAutoMind/batches/reviews/2026-08-31-am.md + +## Ruling + +I have updated the JAX gradient epic to do alot of phases back from the start, so wee will basically discard this and start again. Dont change anything just let that recent intake that precendent. + +## Evidence + +- `inference_programme` (project row): `output/legacy_wrong/searches/nautilus/imaging/slam_source_pix_nn/hst/hpc_a100_fp64_ref/7ced3989425354c2a53d4a79d7da2b11` +- `output/legacy_wrong/searches/nautilus/imaging/slam_source_pix_nn/hst/pos_tauto0.2_f1e8/hpc_a100_fp64_ref_pos_tauto0.2_f1e8/35b83ae5f6eab98575809e7c5dc5f3af` +- `logs/output/output.342016_{5,6}.out` on the mirror +- `PyAutoMind/batches/reviews/2026-08-31-am.md` — the human's note on this member diff --git a/rulings/2026/08/R-20260831-05.md b/rulings/2026/08/R-20260831-05.md new file mode 100644 index 0000000..326d662 --- /dev/null +++ b/rulings/2026/08/R-20260831-05.md @@ -0,0 +1,20 @@ +# R-20260831-05 — drop inference_programme phase 5 + +Project: inference_programme +Phase: phases/inference_programme/slogdet_ab_342017.md +Runs: 342017 +Ruling: drop +Batch: 2026-08-31-am +Reviewed-at: 2026-08-31T13:34Z +Migrated-from: PyAutoMind/batches/reviews/2026-08-31-am.md + +## Ruling + +I have updated the JAX gradient epic to do alot of phases back from the start, so wee will basically discard this and start again. Dont change anything just let that recent intake that precendent. + +## Evidence + +- `inference_programme` (project row): `output/legacy_wrong/searches/nautilus/imaging/slam_source_pix_nn/hst/ld_cholesky/…` +- `output/legacy_wrong/searches/nautilus/imaging/slam_source_pix_nn/hst/ld_slogdet/…` +- `logs/output/output.342017_{0,1}.out` on the mirror; RAL-only `slogdet_ab_harvest` +- `PyAutoMind/batches/reviews/2026-08-31-am.md` — the human's note on this member diff --git a/rulings/2026/08/R-20260831-06.md b/rulings/2026/08/R-20260831-06.md new file mode 100644 index 0000000..c6819c5 --- /dev/null +++ b/rulings/2026/08/R-20260831-06.md @@ -0,0 +1,23 @@ +# R-20260831-06 — accept subhalo_validation phase 1 + +Project: subhalo_validation +Phase: phases/subhalo_validation/delaunay_adapt_split_pl_sersic_0.md +Runs: 342027 +Ruling: accept +Batch: 2026-08-31-am +Reviewed-at: 2026-08-31T13:34Z +Migrated-from: PyAutoMind/batches/reviews/2026-08-31-am.md + +## Ruling + +Review on numba likelihood function stuff: 1) End to end runs are good in general, with numba speed up run times roughly x8 faster than old code, incredible. Do an intake to investigate if the linear algebra approach taken for this is the same as the sparse operator mode alreasdy implemented for JAX, and if it is not ask whether it is amenable to GPU JAX speed up or if there is some sparsity exploitation that means this is suited to CPU. 2) The pl_eff_1_outer_no_subhalo task did not get a good result, the outer source galaxy messed up the solution. I think the issue is that source_pix[1] used ConstantSplit, not AdaptSplit, so can you rerun source_pix[1] on this lens with a suffix "_adapt_split_fix" and I will inspect the result. + +We dont need subhalo[2] for these currently, knowing if theres a detection in subhalo[1] is sufficent so dont run subhalo[2] for future runs. + +Lets also do a RectangularBilinear run for the two successful lenses in the next batch / task. + +## Evidence + +- `subhalo_validation` (project row): `output/subhalo/detect/delaunay_adapt_split/pl_sersic_0_no_subhalo/` +- `results/delaunay_adapt_split/pl_sersic_0_no_subhalo.json` — the witness +- `wiki/project/results_summary.md` — the scientific commentary and the provenance row diff --git a/rulings/2026/08/R-20260831-07.md b/rulings/2026/08/R-20260831-07.md new file mode 100644 index 0000000..e73b6c1 --- /dev/null +++ b/rulings/2026/08/R-20260831-07.md @@ -0,0 +1,29 @@ +# R-20260831-07 — leave-to-finish subhalo_validation phase 2 + +Project: subhalo_validation +Phase: phases/subhalo_validation/delaunay_adapt_split_pl_eff_0.md +Runs: 342027 +Ruling: leave-to-finish +Batch: 2026-08-31-am +Reviewed-at: 2026-08-31T13:34Z +Migrated-from: PyAutoMind/batches/reviews/2026-08-31-am.md + +## Ruling + +Review on numba likelihood function stuff: 1) End to end runs are good in general, with numba speed up run times roughly x8 faster than old code, incredible. Do an intake to investigate if the linear algebra approach taken for this is the same as the sparse operator mode alreasdy implemented for JAX, and if it is not ask whether it is amenable to GPU JAX speed up or if there is some sparsity exploitation that means this is suited to CPU. 2) The pl_eff_1_outer_no_subhalo task did not get a good result, the outer source galaxy messed up the solution. I think the issue is that source_pix[1] used ConstantSplit, not AdaptSplit, so can you rerun source_pix[1] on this lens with a suffix "_adapt_split_fix" and I will inspect the result. + +We dont need subhalo[2] for these currently, knowing if theres a detection in subhalo[1] is sufficent so dont run subhalo[2] for future runs. + +Lets also do a RectangularBilinear run for the two successful lenses in the next batch / task. + +--- + +Filed on both `pl_eff` phases with the same body: at the review both `342027_1` and +`342027_2` were still inside `subhalo[2]` with about three hours of wall left, and the +human left them to finish rather than ruling on an unfinished chain. + +## Evidence + +- `subhalo_validation` (project row): `output/subhalo/detect/delaunay_adapt_split/pl_eff_0_no_subhalo/` +- `results/delaunay_adapt_split/pl_eff_0_no_subhalo.json` — the witness +- `wiki/project/results_summary.md`, `wiki/project/state.md` diff --git a/rulings/2026/08/R-20260831-08.md b/rulings/2026/08/R-20260831-08.md new file mode 100644 index 0000000..205bfe4 --- /dev/null +++ b/rulings/2026/08/R-20260831-08.md @@ -0,0 +1,31 @@ +# R-20260831-08 — leave-to-finish subhalo_validation phase 3 + +Project: subhalo_validation +Phase: phases/subhalo_validation/delaunay_adapt_split_pl_eff_1_outer.md +Runs: 342027, 342093 +Ruling: leave-to-finish +Batch: 2026-08-31-am +Reviewed-at: 2026-08-31T13:34Z +Migrated-from: PyAutoMind/batches/reviews/2026-08-31-am.md + +## Ruling + +Review on numba likelihood function stuff: 1) End to end runs are good in general, with numba speed up run times roughly x8 faster than old code, incredible. Do an intake to investigate if the linear algebra approach taken for this is the same as the sparse operator mode alreasdy implemented for JAX, and if it is not ask whether it is amenable to GPU JAX speed up or if there is some sparsity exploitation that means this is suited to CPU. 2) The pl_eff_1_outer_no_subhalo task did not get a good result, the outer source galaxy messed up the solution. I think the issue is that source_pix[1] used ConstantSplit, not AdaptSplit, so can you rerun source_pix[1] on this lens with a suffix "_adapt_split_fix" and I will inspect the result. + +We dont need subhalo[2] for these currently, knowing if theres a detection in subhalo[1] is sufficent so dont run subhalo[2] for future runs. + +Lets also do a RectangularBilinear run for the two successful lenses in the next batch / task. + +--- + +Filed on both `pl_eff` phases with the same body: at the review both `342027_1` and +`342027_2` were still inside `subhalo[2]` with about three hours of wall left, and the +human left them to finish rather than ruling on an unfinished chain. + +## Evidence + +- `subhalo_validation` (project row): `output/subhalo/detect/delaunay_adapt_split/pl_eff_1_outer_no_subhalo/` +- `output/subhalo/detect/delaunay_adapt_split/pl_eff_1_outer_no_subhalo_adapt_split_fix/source_pix[1]/085f85ba6970436a1b8b115b6c9b3fbf` +- `results/delaunay_adapt_split/pl_eff_1_outer_no_subhalo.json` — the witness +- `results/figures/adapt_split_fix_{regularization,source_plane,residuals}_before_after.jpg` +- `wiki/project/2026-08-31-adapt-split-fix-and-rectangular.md`, `wiki/project/state.md` diff --git a/rulings/2026/09/R-20260901-01.md b/rulings/2026/09/R-20260901-01.md new file mode 100644 index 0000000..3cbe3c9 --- /dev/null +++ b/rulings/2026/09/R-20260901-01.md @@ -0,0 +1,24 @@ +# R-20260901-01 — accept inference_programme phase 7 + +Project: inference_programme +Phase: phases/inference_programme/mge_pos_ref_reuse.md +Runs: 340210 +Ruling: accept +Batch: 2026-08-31-pm +Reviewed-at: 2026-09-01T15:13Z +Migrated-from: autolens_profiling/results/notes/inference/DECISIONS.md § 2026-09-01 — Batch 2026-08-31-pm review rulings (InferenceRefs_v1 legacy-reuse members) + +## Ruling + +**mge-pos-ref-reuse — ACCEPTED.** The quarantined-but-reusable `340210_9` run +(2026-08-25) is adopted as the `mge_pos_fp64` InferenceRefs_v1 reference. Per +the MGE reuse rule: cite it and `mv` it from `output/legacy/searches/…` back +into the active tree (RAL **and** mirror) — laptop action, queued for the next +local-dev slot. Array task 9 stays excluded from the refs array. + +## Evidence + +- `inference_programme` (project row): `output/legacy/searches/nautilus/imaging/mge/hst/pos_tauto0.2_f1e8/hpc_a100_fp64_ref_pos_tauto0.2_f1e8/dc42087fb0524e78fd43eced7706b365` +- `logs/output/output.340210_9.out` on the mirror +- `autolens_profiling/results/notes/inference/DECISIONS.md` — the MGE reuse rule in the 2026-08-31 REWIND entry +- autolens_profiling#201 — the Phase-1 redo leg that routed this as a ruling member diff --git a/rulings/2026/09/R-20260901-02.md b/rulings/2026/09/R-20260901-02.md new file mode 100644 index 0000000..55bfddc --- /dev/null +++ b/rulings/2026/09/R-20260901-02.md @@ -0,0 +1,20 @@ +# R-20260901-02 — accept inference_programme phase 8 + +Project: inference_programme +Phase: phases/inference_programme/mge_fp64_retro_baseline.md +Runs: 339070 +Ruling: accept +Batch: 2026-08-31-pm +Reviewed-at: 2026-09-01T15:13Z +Migrated-from: autolens_profiling/results/notes/inference/DECISIONS.md § 2026-09-01 — Batch 2026-08-31-pm review rulings (InferenceRefs_v1 legacy-reuse members) + +## Ruling + +**mge-fp64-retro-baseline — ACCEPTED.** The retro-adopted `mge_fp64` baseline +stands under the redo standard; no fresh `_ref` run needed. + +## Evidence + +- `autolens_profiling/results/baselines/InferenceRefs_v1/mge_fp64/{README.md,reference.json,target.json}` +- `inference_programme` (project row): `output/legacy/searches/nautilus/imaging/mge/hst/hpc_a100_fp64/181b13114ba3c2298191185ff74f90d8` +- `logs/output/output.339070.out` on the mirror — the log that wrote the source artifact diff --git a/rulings/2026/09/R-20260901-03.md b/rulings/2026/09/R-20260901-03.md new file mode 100644 index 0000000..92c0db7 --- /dev/null +++ b/rulings/2026/09/R-20260901-03.md @@ -0,0 +1,44 @@ +# R-20260901-03 — drop inference_programme phase 9 + +Project: inference_programme +Phase: phases/inference_programme/delaunay_fp64_retro_baseline.md +Runs: 339071 +Ruling: drop +Batch: 2026-08-31-pm +Reviewed-at: 2026-09-01T15:13Z +Follow-ups: autolens_profiling#203, autolens_profiling#205 +Migrated-from: autolens_profiling/results/notes/inference/DECISIONS.md § 2026-09-01 — Batch 2026-08-31-pm review rulings (InferenceRefs_v1 legacy-reuse members) + +## Ruling + +**delaunay-fp64-retro-baseline — REJECTED, with a programme-wide finding.** +The human inspected +`output/legacy_wrong/searches/nautilus/imaging/delaunay/hst/hpc_a100_fp64/` +and identified the solution as a **demagnified-source unphysical solution** — +the classic Inversion bias `PositionsLH` exists to remove +(https://pyautolens.readthedocs.io/en/latest/general/demagnified_solutions.html). +Human, verbatim: *"This is why we basically need to rerun loads of JAX +gradient runs (MGE sources are not effect by this so safe) wit the position +penalty set up and used. The problem is it is not used in all these JAX +Gradient runs!"* + +**Binding consequence for the redo (from the rejection):** every mesh / +pixelization run — the refs array's mesh rows included — must carry a +`PositionsLH` positions penalty (positions + threshold per the workspace +convention, e.g. 0.3"). MGE-source rows are unaffected and stay valid. The +human also flagged that the "Inference Baselines v1" summary carries the same +unphysical-solution issue. Mesh rows produced without the penalty are not +citable as references. The 342091 wave's mesh rows should be checked against +this before certification — a mesh row without the penalty is spent evidence. + +**Output-tree consequence (human directive, verbatim intent):** clear the RAL +active `output/` of all mesh results, the `image_plane`/`source_plane` point +source fits, the `Cluster` folder and the `point_source` folder — moved (not +deleted) to a new `output/legacy_point/` folder, so the active tree is clean and +updates with each task. Laptop action, queued as a local-dev prompt in the Mind. + +## Evidence + +- `autolens_profiling/results/baselines/InferenceRefs_v1/delaunay_fp64/{README.md,reference.json,target.json}` +- `inference_programme` (project row): `output/legacy_wrong/searches/nautilus/imaging/delaunay/hst/hpc_a100_fp64/` — the tree the human inspected +- `logs/output/output.339071.out` on the mirror — the log that wrote the source artifact diff --git a/rulings/2026/09/R-20260901-04.md b/rulings/2026/09/R-20260901-04.md new file mode 100644 index 0000000..6946549 --- /dev/null +++ b/rulings/2026/09/R-20260901-04.md @@ -0,0 +1,28 @@ +# R-20260901-04 — drop inference_programme phase 6 + +Project: inference_programme +Phase: phases/inference_programme/failed_submissions_342008_10.md +Runs: 342008, 342009, 342010 +Ruling: drop +Reviewed-at: 2026-09-01T15:13Z +Migrated-from: PyAutoMind/batches/2026-08-31-am.md § member failed-submissions-342008-10 + +## Ruling + +Carried, then superseded by the rewind. + +The three submissions were the one member of the 2026-08-31-am packet the human left +untouched (`UNREVIEWED, carried to next packet`), and they were carried again past the +pm slot. There was never anything to rule on: each job died in about a second under the +CWD trap, wrote no run directory and left no `.out` log on the mirror, so there is no +evidence to accept and nothing to reuse. The question they were dispatched to answer is +in any case one the 2026-08-31 REWIND retired — the programme restarts at Phase 1. + +Dropped for the record, so the ids stop being carried forward. Revival is a new phase +number. + +## Evidence + +- `inference_programme` (project row): nothing under `output/` — the runs wrote no directory +- `PyAutoMind/batches/2026-08-31-am.md` — member `failed-submissions-342008-10`, `UNREVIEWED, carried to next packet` +- `autolens_profiling/results/notes/inference/DECISIONS.md` — the 2026-08-31 REWIND entry that superseded the question diff --git a/scripts/cortex.py b/scripts/cortex.py index c215e48..41ba794 100644 --- a/scripts/cortex.py +++ b/scripts/cortex.py @@ -83,11 +83,11 @@ PHASE_KEYS = ( "Project", "Phase", "State", "Gates", "Gates-cleared", "Gate-override", "Reset", "Witness", "Budget", "Runs", "Ruling", "Lane", "Review-minutes", - "Epic", "Filed", + "Epic", "Filed", "Migrated-from", ) RULING_KEYS = ( "Project", "Phase", "Runs", "Ruling", "Supersedes", "Batch", "Reviewed-at", - "Review-minutes-actual", "Follow-ups", + "Review-minutes-actual", "Follow-ups", "Migrated-from", ) PHASE_SECTIONS = ("Question", "Witness", "Where to look", "Runs", "Ruling") RULING_SECTIONS = ("Ruling", "Evidence") diff --git a/tests/test_cortex.py b/tests/test_cortex.py index d6aa08b..b717fc2 100644 --- a/tests/test_cortex.py +++ b/tests/test_cortex.py @@ -302,6 +302,21 @@ def test_illegal_state_unknown_key_and_missing_section(tmp_path): "missing section ## Where to look") +def test_migrated_from_is_legal_on_a_phase_and_a_ruling(tmp_path): + """Decision 53: the provenance key joins both key tables; everything else + outside them is still drift.""" + root = _copy(tmp_path) + _edit(root, P["01_scope"], "Filed: 2026-08-25\n", + "Filed: 2026-08-25\nMigrated-from: PyAutoMind/draft/research/example/scope.md\n") + _edit(root, R + "R-20260901-05.md", "Ruling: drop\n", + "Ruling: drop\nMigrated-from: PyAutoMind/batches/reviews/2026-09-01-pm.md\n") + assert _problems(root) == [] + _edit(root, P["01_scope"], "Migrated-from: PyAutoMind", "Migrated-to: PyAutoMind") + _edit(root, R + "R-20260901-05.md", "Migrated-from: PyAutoMind", "Migrated-to: PyAutoMind") + _assert_drift(root, "01_scope.md: unknown header key Migrated-to:", + "R-20260901-05.md: unknown header key Migrated-to:") + + def test_owner_repo_gate_form_is_refused(tmp_path): root = _copy(tmp_path) _edit(root, P["02_gated_on_dev"], "Gates: PyAutoArray#431,", "Gates: PyAutoLabs/PyAutoArray#431,")
ProjectStatusPartitionSyncWitness