From f597b1f72ffb32943423c592b1289d96746d3ea8 Mon Sep 17 00:00:00 2001 From: Jammy2211 Date: Thu, 29 Oct 2026 13:43:01 +0000 Subject: [PATCH 1/3] projects.yaml: development projects' ral_root under RAL development/ MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit On the human's decision (2026-10-29) the RAL trees of analytic_gaussian, ep_toy_gaussian, ic50_workspace and slope_hierarchy_scale moved from /mnt/ral/jnightin/active/ to /mnt/ral/jnightin/development/, mirroring the laptop (decision 61); their hpc/sync.conf HPC_BASE follow and `hpc/sync check` passes for each. Also: euclid_dr1's ral_root was stale (top level) — its tree and running jobs are at active/euclid_dr1; and the sync_verbs of analytic_gaussian, ep_toy_gaussian, slope_hierarchy_scale now list every verb their hpc/sync dispatches. Each moved project's ledger notes the move; dashboard re-rendered. `cortex drift`: 0 errors; the development ral_category warnings are gone. Co-Authored-By: Claude Opus 5.5 --- dashboard.html | 44 +++++++++++++++---------------- dashboard.md | 42 ++++++++++++++--------------- projects.yaml | 16 +++++------ projects/analytic_gaussian.md | 1 + projects/ep_toy_gaussian.md | 1 + projects/ic50_workspace.md | 1 + projects/slope_hierarchy_scale.md | 1 + state.json | 2 +- 8 files changed, 56 insertions(+), 52 deletions(-) diff --git a/dashboard.html b/dashboard.html index f425aae..771882a 100644 --- a/dashboard.html +++ b/dashboard.html @@ -332,7 +332,7 @@ - Open ic50_workspace: https://github.com/Jammy2211/ic50_workspace - Open autolens_inference: https://github.com/PyAutoLabs/autolens_inference - Open autofit_inference: https://github.com/PyAutoLabs/autofit_inference -- Open euclid_detailed: https://github.com/PyAutoLabs/euclid_detailed
Last updated 2026-10-29 12:49:17 UTC
↻ Update

+- Open euclid_detailed: https://github.com/PyAutoLabs/euclid_detailed
Last updated 2026-10-29 13:43:01 UTC
↻ Update

@@ -341,10 +341,10 @@ ProjectRunningOpenLast update subhalo_validation052026-09-07 euclid_dr116422026-10-10 -analytic_gaussian002026-10-23 -ep_toy_gaussian002026-10-14 -slope_hierarchy_scale002026-10-14 -ic50_workspace002026-10-14 +analytic_gaussian002026-10-29 +ep_toy_gaussian002026-10-29 +slope_hierarchy_scale002026-10-29 +ic50_workspace002026-10-29 autolens_inference002026-10-21 autofit_inference002026-10-16 euclid_detailed002026-10-15 @@ -392,7 +392,7 @@

Runs (5)

Registered:
2026-09-11
Local:
/mnt/c/Users/Jammy/Workspace/Science/active/euclid_dr1
-
RAL:
/mnt/ral/jnightin/euclid_dr1
+
RAL:
/mnt/ral/jnightin/active/euclid_dr1

Current position

The top-1000 DR1 sep1 vis_lp array finished 971 of 1000; 29 hit the 18-hour walltime. Array 344645 is now running vis_pix with the Numba CPU sparse route for those 971 completed lenses only.

@@ -476,12 +476,12 @@

Runs (58)

-

analytic_gaussian

Graphical and EP against a closed-form Gaussian posterior, at ensemble scaleno runs recorded · updated 2026-10-23 +

analytic_gaussian

Graphical and EP against a closed-form Gaussian posterior, at ensemble scaleno runs recorded · updated 2026-10-29
Registered:
not recorded
Local:
/mnt/c/Users/Jammy/Workspace/Science/development/analytic_gaussian
-
RAL:
/mnt/ral/jnightin/active/analytic_gaussian
+
RAL:
/mnt/ral/jnightin/development/analytic_gaussian

Current position

Wave 1 (342413, 200 seeds at N=5) is accepted as the baseline: autofit EP is exact on the Gaussian leg, leg B sigma misses as pre-registered (78/200), collapse rate 0/200. On criterion 2, astra and an independent Opus review both favour an under-calibrated mu threshold (calibrated on five unrepresentative seeds) and found no defect in the minimal-EP control; both flag the per-site sigma>0 clip (analytic_ep_minimal.py:333) as the one audit target, and astra will not clear the control without an independent reconstruction of the EP fixed point. Next: run the planned diagnostic (archive/tasks/analytic_gaussian/minimal_ep_legb_mu_threshold.md) on a passing, an edge and the worst mu seed; the N=25 rung stays written but unsubmitted until it lands.

@@ -489,13 +489,13 @@

Current position

Runs (0)

nothing on the cluster

-
Recent log — 5 of 9 entries +
Recent log — 5 of 10 entries
    +
  • 2026-10-29 — note — moved to Science/development/ and RAL /mnt/ral/jnightin/development/analytic_gaussian on the human's decision (2026-10-29); hpc/sync.conf HPC_BASE updated, hpc/sync check passes
  • 2026-10-23 — note —
    second wave-1 finding closed (PyAutoFit#1713, PR #1714 merged, autofit_workspace_test#116): the `invalid value encountered in add` warnings on seeds 25 and 172 are a readout-only defect — under the mode projection an accepted truncated-nor…show less

    second wave-1 finding closed (PyAutoFit#1713, PR #1714 merged, autofit_workspace_test#116): the `invalid value encountered in add` warnings on seeds 25 and 172 are a readout-only defect — under the mode projection an accepted truncated-normal scatter message sits ~19 sigma into its tail, its log_partition underflows to −inf (true log Z ≈ −187.9), the sigma evidence goes +inf and the EP log_evidence NaN from sweep 4; posteriors bit-identical to the ens_n5 sidecars. The library now names the term once per fit and returns an explicit NaN. The cure (tail-stable log_partition via log_ndtr) is message maths, filed in the Mind as draft/bug/autofit/truncated_normal_log_partition_tail_underflow.md for the human.

  • 2026-10-23 — note —
    criterion 6 amended (PyAutoFit#1711, PR #1712): the 19/400 zero-SUCCESS seed-legs of wave 1 split into 13 NO_CHANGE-only prior factors (zero-iteration Laplace convergence at the start point, projection accepted, closed-form match a 5e-5 /…show less

    criterion 6 amended (PyAutoFit#1711, PR #1712): the 19/400 zero-SUCCESS seed-legs of wave 1 split into 13 NO_CHANGE-only prior factors (zero-iteration Laplace convergence at the start point, projection accepted, closed-form match a 5e-5 / b 2e-5 on seed 140 leg A — a converged update, not stale; library reporting aligned, no EP maths changed) and 6 BAD_PROJECTION-only hierarchical factors (the #1405 scatter state, cured by projection="moments" #1656; seeds 29 and 71 re-run under moments give HierarchicalFactor SUCCESS 28 and 23). Criterion 6 now counts SUCCESS or NO_CHANGE as a completed update; the matching ensemble_seed.py any_factor_never_succeeded change is the human's to apply in the science checkout. Side finding: seed 29 leg B under moments still misses the sigma cell (a 0.225 / b 0.416) while sigma stays inside [q05, q95].

  • 2026-10-14 — note —
    PyAutoFit#1656 (merged 2026-09-30, released 2026.10.7.1) ports the minimal-EP referee's moment matching as LaplaceOptimiser(projection="moments"); on analytic_gaussian.py leg B autofit EP sigma 6.3401 ± 2.4609 vs closed form 6.5667 ± 2.883…show less

    PyAutoFit#1656 (merged 2026-09-30, released 2026.10.7.1) ports the minimal-EP referee's moment matching as LaplaceOptimiser(projection="moments"); on analytic_gaussian.py leg B autofit EP sigma 6.3401 ± 2.4609 vs closed form 6.5667 ± 2.8832 (mode path 9.3722, BAD_PROJECTION=19). A wave-2 ensemble on the moments path needs a private PyAutoFit checkout on RAL: the shared mirror is 404b3e5f7 and lacks it. The criterion-2 mu-threshold diagnostic is still unrun.

  • 2026-10-14 — note — output/ vaulted to z_vault/ on laptop and RAL (mv, nothing deleted), anticipating reruns; output/ left empty, results/ untouched; z_vault/ gitignored locally
    • -
  • 2026-09-30 — note —
    astra (Codex gpt-6-astra) on criterion 2, verbatim: 'Neither is established as "wrong" by this result alone. I favour inadequate calibration of the threshold, with moderate confidence, but would not yet certify the control.' Decisive diagn…show less

    astra (Codex gpt-6-astra) on criterion 2, verbatim: 'Neither is established as "wrong" by this result alone. I favour inadequate calibration of the threshold, with moderate confidence, but would not yet certify the control.' Decisive diagnostic: independently reconstruct the same EP factorisation at the fixed point on a passing, an edge and the worst mu seed and compare tilted vs q moments; audit the per-site sigma>0 clip (analytic_ep_minimal.py:333). Opus reviewer concurs independently (a_mu = 0.041*(E_ref[mu]-m0)/std_ref over 200 seeds, corr 0.98; calibration seeds 0-4 had |a_mu| 0.002-0.020 vs 95th pct 0.077). Full answer: PyAutoCortex#50 comment https://github.com/PyAutoLabs/PyAutoCortex/issues/50#issuecomment-5908448194

full log

active · ral · PyAutoCortex#34

@@ -503,12 +503,12 @@

Runs (0)

-

ep_toy_gaussian

NUTS versus three EP fits on the #1405 collapse toy at N=50no runs recorded · updated 2026-10-14 +

ep_toy_gaussian

NUTS versus three EP fits on the #1405 collapse toy at N=50no runs recorded · updated 2026-10-29
Registered:
not recorded
Local:
/mnt/c/Users/Jammy/Workspace/Science/development/ep_toy_gaussian
-
RAL:
/mnt/ral/jnightin/active/ep_toy_gaussian
+
RAL:
/mnt/ral/jnightin/development/ep_toy_gaussian

Current position

Wave 1 (n50_seed42: 342639 + 342640_[0-2]) failed on infrastructure — EP x3 on EMFILE (fixed by PyAutoFit#1632/#1634, both in the RAL mirror at 404b3e5f7), NUTS on OOM at the 8 GB cap while XLA compiled the 50-factor window-adaptation scan, which grows with N (≥5.4 GB at N=20 locally). Wave 2 is prepared but not submitted: sample n50_seed42_w2 (free locally and on RAL; still needs its toy.SAMPLES entry), submit_nuts raised to 64 GB, submit_ep unchanged at 8 GB (peak ~590 MB). The collapse question at N=50 is unanswered; only n5_smoke speaks (EP RECOVER 3/3, NUTS 49.89/11.14). Next: human go, add the sample entry, push and submit NUTS then EP with SAMPLE=n50_seed42_w2 exported.

@@ -516,13 +516,13 @@

Current position

Runs (0)

nothing on the cluster

-
Recent log — 5 of 10 entries +
Recent log — 5 of 11 entries
    +
  • 2026-10-29 — note — moved to Science/development/ and RAL /mnt/ral/jnightin/development/ep_toy_gaussian on the human's decision (2026-10-29); hpc/sync.conf HPC_BASE updated, hpc/sync check passes
  • 2026-10-14 — note — Wave 2 still unsubmitted. RAL shared mirror 404b3e5f7 already carries the EMFILE fixes (#1632/#1634) wave 2 needs; it lacks #1655 (ProjectionException recovery), so a private PyAutoFit checkout is optional here but advisable.
  • 2026-10-14 — note — output/ vaulted to z_vault/ on laptop and RAL (mv, nothing deleted), anticipating reruns; output/ left empty, results/ untouched; z_vault/ gitignored locally
  • 2026-09-30 — note —
    RAL mirror PyAutoFit now 404b3e5f7 (2026-09-27), contains #1632 (b82fb3f69) and #1634 (fa2d540ac); 5 commits behind origin/main, none in blackjax/graphical/dynesty; deps at floor except anesthetic 2.8.14 < 2.9.0 (plots only). Sample n50_se…show less

    RAL mirror PyAutoFit now 404b3e5f7 (2026-09-27), contains #1632 (b82fb3f69) and #1634 (fa2d540ac); 5 commits behind origin/main, none in blackjax/graphical/dynesty; deps at floor except anesthetic 2.8.14 < 2.9.0 (plots only). Sample n50_seed42_w2 is free locally and on RAL but must be added to toy.SAMPLES (scripts/toy.py) before submission, and SAMPLE must be exported to the job (hpc/sync submit does not forward laptop env)

  • 2026-09-30 — note —
    wave-2 prep: 342639 NUTS OOM was XLA compilation of the vmapped window-adaptation scan, not sampling (sacct MaxRSS 8,180,536K at ReqMem 8G; py-spy on a local N=50 run sits in backend_compile_and_load; local probes N=10 3.2 GB peak / 132 s…show less

    wave-2 prep: 342639 NUTS OOM was XLA compilation of the vmapped window-adaptation scan, not sampling (sacct MaxRSS 8,180,536K at ReqMem 8G; py-spy on a local N=50 run sits in backend_compile_and_load; local probes N=10 3.2 GB peak / 132 s warmup compile, N=20 >=5.4 GB / 247 s, N=50 still compiling at 5 min) — submit_nuts --mem raised 8gb -> 64gb locally, uncommitted (est. peak 15-25 GB on RAL; ral nodes ~921 GB); EP peak ~590 MB, submit_ep stays 8gb

    • -
  • 2026-09-24 — note —
    per-search overhead profile on this toy (2026-09-24, laptop, N=5, max_steps=3): 2.45 s per Dynesty factor search, ~1.47 s dynesty run_nested + ~0.98 s autofit wrapper (plots, a redundant second run_nested pass, samples writes); inside run_…show less

    per-search overhead profile on this toy (2026-09-24, laptop, N=5, max_steps=3): 2.45 s per Dynesty factor search, ~1.47 s dynesty run_nested + ~0.98 s autofit wrapper (plots, a redundant second run_nested pass, samples writes); inside run_nested instance_from_vector is 57 % of the likelihood callback for a 1-parameter model. Speed-up prompt filed in the Mind: draft/refactor/autofit/ep_factor_search_wrapper_overhead.md

full log

active · ral · ep_toy_gaussian#1

@@ -530,12 +530,12 @@

Runs (0)

-

slope_hierarchy_scale

Hierarchical slope recovery with graphical and EP at N=25 to 50no runs recorded · updated 2026-10-14 +

slope_hierarchy_scale

Hierarchical slope recovery with graphical and EP at N=25 to 50no runs recorded · updated 2026-10-29
Registered:
not recorded
Local:
/mnt/c/Users/Jammy/Workspace/Science/development/slope_hierarchy_scale
-
RAL:
/mnt/ral/jnightin/active/slope_hierarchy_scale
+
RAL:
/mnt/ral/jnightin/development/slope_hierarchy_scale

Current position

The N=25 graphical baselines are in: per-lens fits (342348) give a naive parent of 1.999 / 0.089 (deconvolved 0.085), and the joint NUTS fit (342350_0) recovers mean 1.994 [1.976, 2.014] and sigma 0.087 [0.074, 0.104] against truth 2.0 / 0.1; its summary JSON still needs writing offline from samples.csv (now under z_vault/sample_n25_seed42). The EP arm returned the prior on the Laplace projection (343299: hierarchical factor 0/50 SUCCESS). The cure has landed: PyAutoFit#1656 (merged 2026-09-30, released 2026.10.7.1) adds af.LaplaceOptimiser(projection="moments"), which takes the tight toy from 0 to 3/3 SUCCESS. The shared RAL PyAutoFit mirror is still 404b3e5f7 (2026-09-27, kept frozen for euclid_dr1) and lacks both #1655 and #1656, so any rerun needs a private PyAutoFit checkout on RAL. All earlier output is vaulted in z_vault/ on laptop and RAL, output/ is empty for the rerun. Next: rerun the N=25 EP arm with projection="moments" on a private PyAutoFit checkout and compare its scatter against the NUTS baseline.

@@ -543,13 +543,13 @@

Current position

Runs (0)

nothing on the cluster

-
Recent log — 5 of 14 entries +
Recent log — 5 of 15 entries
    +
  • 2026-10-29 — note — moved to Science/development/ and RAL /mnt/ral/jnightin/development/slope_hierarchy_scale on the human's decision (2026-10-29); hpc/sync.conf HPC_BASE updated, hpc/sync check passes
  • 2026-10-14 — note — output/ vaulted to z_vault/ on laptop and RAL (mv, nothing deleted), anticipating reruns; output/ left empty, results/ untouched; z_vault/ gitignored locally
  • 2026-09-30 — note —
    graphical baselines recorded late at the 2026-09-30 check-in. 342348_[0-24] per-lens (one_by_one Nautilus, A100, 25/25, 6.9-8.8 min/task, 0 float32 truncations): naive parent mean 1.9992 +/- 0.0178 (SEM), std of medians 0.0889, deconvolved…show less

    graphical baselines recorded late at the 2026-09-30 check-in. 342348_[0-24] per-lens (one_by_one Nautilus, A100, 25/25, 6.9-8.8 min/task, 0 float32 truncations): naive parent mean 1.9992 +/- 0.0178 (SEM), std of medians 0.0889, deconvolved scatter 0.0852 (rms 1-sigma half-width 0.0255); truth inside the 1-sigma interval for 17/25 lenses; per-lens JSONs on RAL at results/one_by_one_sample_n25_seed42/dataset_*.json (not pulled; aggregate never written by the array). 342350_0 joint hierarchical NUTS: science completed despite the post-processing OOM — 500 warm-up (8h08m) + 1000 samples (7h17m), 1 chain, 0 divergences, acceptance 0.84, rhat_max 1.012, ess_min 133/302 params; parent mean 1.9936 [1.9761, 2.0136] 68%, sigma 0.0871 [0.0744, 0.1041] 68% (95% [0.0636, 0.1190]) vs truth 2.0/0.1 and realised draws 1.9917/0.0864 — the N=5 high-scatter bias (0.143) is gone at N=25. Computed from RAL samples.csv (graphical_nuts/f16c32e17f570603a1dc061f95792798/files); results/graphical_nuts_sample_n25_seed42.json never written (killed at 64 GB in latent/arviz post-processing) — pull samples.csv and write the summary offline

  • 2026-09-30 — run —
    342351_0 failed — wall 27:41: n25_scale_up: EP arm on the CPU partition, hpc/batch_cpu/submit_ep, max_steps 12; RAL PyAutoFit mirror 68ff9bd57 predates PyAutoFit#1580, so this arm runs without the stale-mask fixed-point fix — CANCELLED 202…show less

    342351_0 failed — wall 27:41: n25_scale_up: EP arm on the CPU partition, hpc/batch_cpu/submit_ep, max_steps 12; RAL PyAutoFit mirror 68ff9bd57 predates PyAutoFit#1580, so this arm runs without the stale-mask fixed-point fix — CANCELLED 2026-09-09 20:56:19 BST (sacct 1-03:41:06): log stops at the first Nautilus status line 2026-09-08 17:15:36, no factor finished; superseded by 342410 then 343299, output parked as ep_dead_342351; recorded late at the 2026-09-30 check-in

  • 2026-09-30 — run —
    342350_0 failed — wall 16:29: n25_scale_up: joint hierarchical NUTS fit, hpc/batch_gpu/submit_graphical — OUT_OF_MEMORY 2026-09-09 13:06:31 BST (sacct 0:125, cgroup oom-kill): NUTS warm-up 500 + sampling 1000 steps completed and samples.cs…show less

    342350_0 failed — wall 16:29: n25_scale_up: joint hierarchical NUTS fit, hpc/batch_gpu/submit_graphical — OUT_OF_MEMORY 2026-09-09 13:06:31 BST (sacct 0:125, cgroup oom-kill): NUTS warm-up 500 + sampling 1000 steps completed and samples.csv was written 12:14 (output/sample_n25_seed42/graphical/graphical_nuts/f16c32e17f570603a1dc061f95792798/files/samples.csv), then killed during latent samples / arviz post-processing; recorded late at the 2026-09-30 check-in

    • -
  • 2026-09-30 — run —
    342348_[0-24] finished — wall 3:22: n25_scale_up: one lens per array task, hpc/batch_gpu/submit_one_by_one, sample_n25_seed42 — all 25 tasks COMPLETED 2026-09-08 17:14-20:37 BST (sacct, 6:54-8:50 per task, serial on the GPU): per-lens one_…show less

    342348_[0-24] finished — wall 3:22: n25_scale_up: one lens per array task, hpc/batch_gpu/submit_one_by_one, sample_n25_seed42 — all 25 tasks COMPLETED 2026-09-08 17:14-20:37 BST (sacct, 6:54-8:50 per task, serial on the GPU): per-lens one_by_one slopes are the baseline, e.g. dataset_0 2.0417 [2.0130, 2.0730] (truth 2.0305), dataset_24 2.0333 [2.0205, 2.0463] (truth 2.0439); recorded late at the 2026-09-30 check-in

full log

active · both · slope_hierarchy_scale#2

@@ -557,12 +557,12 @@

Runs (0)

-

ic50_workspace

EP against the graphical joint fit on IC50 dose-response data, scaling upno runs recorded · updated 2026-10-14 +

ic50_workspace

EP against the graphical joint fit on IC50 dose-response data, scaling upno runs recorded · updated 2026-10-29
Registered:
not recorded
Local:
/mnt/c/Users/Jammy/Workspace/Science/development/ic50_workspace
-
RAL:
/mnt/ral/jnightin/active/ic50_workspace
+
RAL:
/mnt/ral/jnightin/development/ic50_workspace

Current position

The scale ladder is in. EP passes N=5/10/25 with coef_mean within 3σ and cost growing about linearly per sweep (26/50/140 s); it died at N=50 on a projection assert. That fix has landed: PyAutoFit#1655 (merged 2026-09-30, released 2026.10.7.1) raises ProjectionException, which EP factor_step now recovers from instead of killing the run. Graphical is cheaper but overconfident beyond N=25, and EP hill_coef widths are not yet comparable because util.py reports factor-message widths. The shared RAL PyAutoFit mirror is still 404b3e5f7 (2026-09-27, kept frozen for euclid_dr1) and lacks both #1655 and #1656, so any rerun needs a private PyAutoFit checkout on RAL. All earlier output is vaulted in z_vault/ on laptop and RAL, output/ is empty for the rerun. Next: rerun the EP ladder on a private PyAutoFit checkout with output on and a fixed seed, report hill_coef from the mean field, then give ep_lbfgs_jax a witness as the scale lever.

@@ -570,13 +570,13 @@

Current position

Runs (0)

nothing on the cluster

-
Recent log — 5 of 12 entries +
Recent log — 5 of 13 entries
    +
  • 2026-10-29 — note — moved to Science/development/ and RAL /mnt/ral/jnightin/development/ic50_workspace on the human's decision (2026-10-29); hpc/sync.conf HPC_BASE updated, hpc/sync check passes
  • 2026-10-14 — note — output/ vaulted to z_vault/ on laptop and RAL (mv, nothing deleted), anticipating reruns; output/ left empty, results/ untouched; z_vault/ gitignored locally
  • 2026-09-30 — note —
    Ladder read: EP s/sweep 26/50/140/~300 s at N=5/10/25/50 vs graphical wall 21/42/119/281 s. EP hill_coef widths flat at ~0.8 because util.py reports factor-message (likelihood) widths and the global factor freezes hill_coef; graphical widt…show less

    Ladder read: EP s/sweep 26/50/140/~300 s at N=5/10/25/50 vs graphical wall 21/42/119/281 s. EP hill_coef widths flat at ~0.8 because util.py reports factor-message (likelihood) widths and the global factor freezes hill_coef; graphical widths shrink but fall to 56/75 and 65/150 within 3 sigma at N=25/50 (overconfident). coef_mean passes 3 sigma for both methods on every completed rung.

  • 2026-09-30 — note —
    342411 N=50 failure diagnosed: AssertionError at autofit/messages/abstract.py:316 while projecting the global factor after EP sweep 4 (mirror 66f9f8d5d); the 4th global search had max logL -337 (vs -69 at sweep 3) and logz +/- nan. Library…show less

    342411 N=50 failure diagnosed: AssertionError at autofit/messages/abstract.py:316 while projecting the global factor after EP sweep 4 (mirror 66f9f8d5d); the 4th global search had max logL -337 (vs -69 at sweep 3) and logz +/- nan. Library defect: project asserts on non-finite suff_stats and factor_step (optimiser.py:150) does not catch AssertionError, so one bad projection kills the run; no later commit touches the path. A local rerun at 66f9f8d5d converged in sweep 3 without failing (unseeded Dynesty, non-deterministic). Mind bug prompt draft/bug/autofit/ep_project_nonfinite_suff_stats_ic50_n50.md

  • 2026-09-09 — run —
    342411 failed — wall 0:38: ep_scale_up: EP scale ladder, sim rungs 5/10/25/50, nlive 150 max_steps 12 — finished 2026-09-09 23:19:29 BST (sacct COMPLETED 00:38:49, the ladder catches rung failures): rungs N=5/10/25 passed the global coef_m…show less

    342411 failed — wall 0:38: ep_scale_up: EP scale ladder, sim rungs 5/10/25/50, nlive 150 max_steps 12 — finished 2026-09-09 23:19:29 BST (sacct COMPLETED 00:38:49, the ladder catches rung failures): rungs N=5/10/25 passed the global coef_mean assertions; rung N=50 FAILED — AssertionError: assert np.isfinite(suff_stats).all() in autofit/messages/abstract.py:316 project (via mapper/prior/abstract.py:237), error.342411.err

    • -
  • 2026-09-09 — run —
    342412 finished — wall 0:07: ep_scale_up: graphical scale ladder, sim rungs 5/10/25/50, nlive 150 — COMPLETED 2026-09-09 22:48:42 BST (sacct 00:07:59): all four rungs N=5/10/25/50 passed the global coef_mean assertions (within 3σ); results…show less

    342412 finished — wall 0:07: ep_scale_up: graphical scale ladder, sim rungs 5/10/25/50, nlive 150 — COMPLETED 2026-09-09 22:48:42 BST (sacct 00:07:59): all four rungs N=5/10/25/50 passed the global coef_mean assertions (within 3σ); results/graphical_ladder_n*_summary.json

full log

active · ral · PyAutoCortex#36

diff --git a/dashboard.md b/dashboard.md index cb5a2b8..8f6898d 100644 --- a/dashboard.md +++ b/dashboard.md @@ -44,10 +44,10 @@ Last check-in: 2026-10-17T11:18Z. |---|---:|---:|---| | subhalo_validation | 0 | 5 | 2026-09-07 | | euclid_dr1 | 16 | 42 | 2026-10-10 | -| analytic_gaussian | 0 | 0 | 2026-10-23 | -| ep_toy_gaussian | 0 | 0 | 2026-10-14 | -| slope_hierarchy_scale | 0 | 0 | 2026-10-14 | -| ic50_workspace | 0 | 0 | 2026-10-14 | +| analytic_gaussian | 0 | 0 | 2026-10-29 | +| ep_toy_gaussian | 0 | 0 | 2026-10-29 | +| slope_hierarchy_scale | 0 | 0 | 2026-10-29 | +| ic50_workspace | 0 | 0 | 2026-10-29 | | autolens_inference | 0 | 0 | 2026-10-21 | | autofit_inference | 0 | 0 | 2026-10-16 | | euclid_detailed | 0 | 0 | 2026-10-15 | @@ -117,7 +117,7 @@ active · ral · [subhalo_validation#2](https://github.com/PyAutoLabs/subhalo_va **Local:** `` /mnt/c/Users/Jammy/Workspace/Science/active/euclid_dr1 `` -**RAL:** `` /mnt/ral/jnightin/euclid_dr1 `` +**RAL:** `` /mnt/ral/jnightin/active/euclid_dr1 `` **Current position** @@ -258,13 +258,13 @@ active · ral · no issue yet
-
analytic_gaussian — Graphical and EP against a closed-form Gaussian posterior, at ensemble scale · no runs recorded · updated 2026-10-23 +
analytic_gaussian — Graphical and EP against a closed-form Gaussian posterior, at ensemble scale · no runs recorded · updated 2026-10-29 **Registered:** not recorded **Local:** `` /mnt/c/Users/Jammy/Workspace/Science/development/analytic_gaussian `` -**RAL:** `` /mnt/ral/jnightin/active/analytic_gaussian `` +**RAL:** `` /mnt/ral/jnightin/development/analytic_gaussian `` **Current position** @@ -282,13 +282,13 @@ Use the cortex skill. — resume analytic_gaussian: read PyAutoCortex projects/a _nothing on the cluster_ -
Recent log — 5 of 9 entries +
Recent log — 5 of 10 entries +- 2026-10-29 — *note* — moved to Science/development/ and RAL /mnt/ral/jnightin/development/analytic_gaussian on the human's decision (2026-10-29); hpc/sync.conf HPC_BASE updated, hpc/sync check passes - 2026-10-23 — *note* — second wave-1 finding closed (PyAutoFit#1713, PR #1714 merged, autofit_workspace_test#116): the `invalid value encountered in add` warnings on seeds 25 and 172 are a readout-only defect — under the mode projection an accepted truncated-normal scatter message sits ~19 sigma into its tail, its log_partition underflows to −inf (true log Z ≈ −187.9), the sigma evidence goes +inf and the EP log_evidence NaN from sweep 4; posteriors bit-identical to the ens_n5 sidecars. The library now names the term once per fit and returns an explicit NaN. The cure (tail-stable log_partition via log_ndtr) is message maths, filed in the Mind as draft/bug/autofit/truncated_normal_log_partition_tail_underflow.md for the human. - 2026-10-23 — *note* — criterion 6 amended (PyAutoFit#1711, PR #1712): the 19/400 zero-SUCCESS seed-legs of wave 1 split into 13 NO_CHANGE-only prior factors (zero-iteration Laplace convergence at the start point, projection accepted, closed-form match a 5e-5 / b 2e-5 on seed 140 leg A — a converged update, not stale; library reporting aligned, no EP maths changed) and 6 BAD_PROJECTION-only hierarchical factors (the #1405 scatter state, cured by projection="moments" #1656; seeds 29 and 71 re-run under moments give HierarchicalFactor SUCCESS 28 and 23). Criterion 6 now counts SUCCESS or NO_CHANGE as a completed update; the matching ensemble_seed.py any_factor_never_succeeded change is the human's to apply in the science checkout. Side finding: seed 29 leg B under moments still misses the sigma cell (a 0.225 / b 0.416) while sigma stays inside [q05, q95]. - 2026-10-14 — *note* — PyAutoFit#1656 (merged 2026-09-30, released 2026.10.7.1) ports the minimal-EP referee's moment matching as LaplaceOptimiser(projection="moments"); on analytic_gaussian.py leg B autofit EP sigma 6.3401 ± 2.4609 vs closed form 6.5667 ± 2.8832 (mode path 9.3722, BAD_PROJECTION=19). A wave-2 ensemble on the moments path needs a private PyAutoFit checkout on RAL: the shared mirror is 404b3e5f7 and lacks it. The criterion-2 mu-threshold diagnostic is still unrun. - 2026-10-14 — *note* — output/ vaulted to z_vault/ on laptop and RAL (mv, nothing deleted), anticipating reruns; output/ left empty, results/ untouched; z_vault/ gitignored locally -- 2026-09-30 — *note* — astra (Codex gpt-6-astra) on criterion 2, verbatim: 'Neither is established as "wrong" by this result alone. I favour inadequate calibration of the threshold, with moderate confidence, but would not yet certify the control.' Decisive diagnostic: independently reconstruct the same EP factorisation at the fixed point on a passing, an edge and the worst mu seed and compare tilted vs q moments; audit the per-site sigma>0 clip (analytic_ep_minimal.py:333). Opus reviewer concurs independently (a_mu = 0.041*(E_ref[mu]-m0)/std_ref over 200 seeds, corr 0.98; calibration seeds 0-4 had |a_mu| 0.002-0.020 vs 95th pct 0.077). Full answer: PyAutoCortex#50 comment https://github.com/PyAutoLabs/PyAutoCortex/issues/50#issuecomment-5908448194 [full log](https://github.com/PyAutoLabs/PyAutoCortex/blob/main/projects/analytic_gaussian.md) @@ -298,13 +298,13 @@ active · ral · [PyAutoCortex#34](https://github.com/PyAutoLabs/PyAutoCortex/is
-
ep_toy_gaussian — NUTS versus three EP fits on the #1405 collapse toy at N=50 · no runs recorded · updated 2026-10-14 +
ep_toy_gaussian — NUTS versus three EP fits on the #1405 collapse toy at N=50 · no runs recorded · updated 2026-10-29 **Registered:** not recorded **Local:** `` /mnt/c/Users/Jammy/Workspace/Science/development/ep_toy_gaussian `` -**RAL:** `` /mnt/ral/jnightin/active/ep_toy_gaussian `` +**RAL:** `` /mnt/ral/jnightin/development/ep_toy_gaussian `` **Current position** @@ -322,13 +322,13 @@ Use the cortex skill. — resume ep_toy_gaussian: read PyAutoCortex projects/ep_ _nothing on the cluster_ -
Recent log — 5 of 10 entries +
Recent log — 5 of 11 entries +- 2026-10-29 — *note* — moved to Science/development/ and RAL /mnt/ral/jnightin/development/ep_toy_gaussian on the human's decision (2026-10-29); hpc/sync.conf HPC_BASE updated, hpc/sync check passes - 2026-10-14 — *note* — Wave 2 still unsubmitted. RAL shared mirror 404b3e5f7 already carries the EMFILE fixes (#1632/#1634) wave 2 needs; it lacks #1655 (ProjectionException recovery), so a private PyAutoFit checkout is optional here but advisable. - 2026-10-14 — *note* — output/ vaulted to z_vault/ on laptop and RAL (mv, nothing deleted), anticipating reruns; output/ left empty, results/ untouched; z_vault/ gitignored locally - 2026-09-30 — *note* — RAL mirror PyAutoFit now 404b3e5f7 (2026-09-27), contains #1632 (b82fb3f69) and #1634 (fa2d540ac); 5 commits behind origin/main, none in blackjax/graphical/dynesty; deps at floor except anesthetic 2.8.14 < 2.9.0 (plots only). Sample n50_seed42_w2 is free locally and on RAL but must be added to toy.SAMPLES (scripts/toy.py) before submission, and SAMPLE must be exported to the job (hpc/sync submit does not forward laptop env) - 2026-09-30 — *note* — wave-2 prep: 342639 NUTS OOM was XLA compilation of the vmapped window-adaptation scan, not sampling (sacct MaxRSS 8,180,536K at ReqMem 8G; py-spy on a local N=50 run sits in backend_compile_and_load; local probes N=10 3.2 GB peak / 132 s warmup compile, N=20 >=5.4 GB / 247 s, N=50 still compiling at 5 min) — submit_nuts --mem raised 8gb -> 64gb locally, uncommitted (est. peak 15-25 GB on RAL; ral nodes ~921 GB); EP peak ~590 MB, submit_ep stays 8gb -- 2026-09-24 — *note* — per-search overhead profile on this toy (2026-09-24, laptop, N=5, max_steps=3): 2.45 s per Dynesty factor search, ~1.47 s dynesty run_nested + ~0.98 s autofit wrapper (plots, a redundant second run_nested pass, samples writes); inside run_nested instance_from_vector is 57 % of the likelihood callback for a 1-parameter model. Speed-up prompt filed in the Mind: draft/refactor/autofit/ep_factor_search_wrapper_overhead.md [full log](https://github.com/PyAutoLabs/PyAutoCortex/blob/main/projects/ep_toy_gaussian.md) @@ -338,13 +338,13 @@ active · ral · [ep_toy_gaussian#1](https://github.com/PyAutoLabs/ep_toy_gaussi
-
slope_hierarchy_scale — Hierarchical slope recovery with graphical and EP at N=25 to 50 · no runs recorded · updated 2026-10-14 +
slope_hierarchy_scale — Hierarchical slope recovery with graphical and EP at N=25 to 50 · no runs recorded · updated 2026-10-29 **Registered:** not recorded **Local:** `` /mnt/c/Users/Jammy/Workspace/Science/development/slope_hierarchy_scale `` -**RAL:** `` /mnt/ral/jnightin/active/slope_hierarchy_scale `` +**RAL:** `` /mnt/ral/jnightin/development/slope_hierarchy_scale `` **Current position** @@ -362,13 +362,13 @@ Use the cortex skill. — resume slope_hierarchy_scale: read PyAutoCortex projec _nothing on the cluster_ -
Recent log — 5 of 14 entries +
Recent log — 5 of 15 entries +- 2026-10-29 — *note* — moved to Science/development/ and RAL /mnt/ral/jnightin/development/slope_hierarchy_scale on the human's decision (2026-10-29); hpc/sync.conf HPC_BASE updated, hpc/sync check passes - 2026-10-14 — *note* — output/ vaulted to z_vault/ on laptop and RAL (mv, nothing deleted), anticipating reruns; output/ left empty, results/ untouched; z_vault/ gitignored locally - 2026-09-30 — *note* — graphical baselines recorded late at the 2026-09-30 check-in. 342348_[0-24] per-lens (one_by_one Nautilus, A100, 25/25, 6.9-8.8 min/task, 0 float32 truncations): naive parent mean 1.9992 +/- 0.0178 (SEM), std of medians 0.0889, deconvolved scatter 0.0852 (rms 1-sigma half-width 0.0255); truth inside the 1-sigma interval for 17/25 lenses; per-lens JSONs on RAL at results/one_by_one_sample_n25_seed42/dataset_*.json (not pulled; aggregate never written by the array). 342350_0 joint hierarchical NUTS: science completed despite the post-processing OOM — 500 warm-up (8h08m) + 1000 samples (7h17m), 1 chain, 0 divergences, acceptance 0.84, rhat_max 1.012, ess_min 133/302 params; parent mean 1.9936 [1.9761, 2.0136] 68%, sigma 0.0871 [0.0744, 0.1041] 68% (95% [0.0636, 0.1190]) vs truth 2.0/0.1 and realised draws 1.9917/0.0864 — the N=5 high-scatter bias (0.143) is gone at N=25. Computed from RAL samples.csv (graphical_nuts/f16c32e17f570603a1dc061f95792798/files); results/graphical_nuts_sample_n25_seed42.json never written (killed at 64 GB in latent/arviz post-processing) — pull samples.csv and write the summary offline - 2026-09-30 — *run* — 342351_0 failed — wall 27:41: n25_scale_up: EP arm on the CPU partition, hpc/batch_cpu/submit_ep, max_steps 12; RAL PyAutoFit mirror 68ff9bd57 predates PyAutoFit#1580, so this arm runs without the stale-mask fixed-point fix — CANCELLED 2026-09-09 20:56:19 BST (sacct 1-03:41:06): log stops at the first Nautilus status line 2026-09-08 17:15:36, no factor finished; superseded by 342410 then 343299, output parked as ep_dead_342351; recorded late at the 2026-09-30 check-in - 2026-09-30 — *run* — 342350_0 failed — wall 16:29: n25_scale_up: joint hierarchical NUTS fit, hpc/batch_gpu/submit_graphical — OUT_OF_MEMORY 2026-09-09 13:06:31 BST (sacct 0:125, cgroup oom-kill): NUTS warm-up 500 + sampling 1000 steps completed and samples.csv was written 12:14 (output/sample_n25_seed42/graphical/graphical_nuts/f16c32e17f570603a1dc061f95792798/files/samples.csv), then killed during latent samples / arviz post-processing; recorded late at the 2026-09-30 check-in -- 2026-09-30 — *run* — 342348_[0-24] finished — wall 3:22: n25_scale_up: one lens per array task, hpc/batch_gpu/submit_one_by_one, sample_n25_seed42 — all 25 tasks COMPLETED 2026-09-08 17:14-20:37 BST (sacct, 6:54-8:50 per task, serial on the GPU): per-lens one_by_one slopes are the baseline, e.g. dataset_0 2.0417 [2.0130, 2.0730] (truth 2.0305), dataset_24 2.0333 [2.0205, 2.0463] (truth 2.0439); recorded late at the 2026-09-30 check-in [full log](https://github.com/PyAutoLabs/PyAutoCortex/blob/main/projects/slope_hierarchy_scale.md) @@ -378,13 +378,13 @@ active · both · [slope_hierarchy_scale#2](https://github.com/PyAutoLabs/slope_
-
ic50_workspace — EP against the graphical joint fit on IC50 dose-response data, scaling up · no runs recorded · updated 2026-10-14 +
ic50_workspace — EP against the graphical joint fit on IC50 dose-response data, scaling up · no runs recorded · updated 2026-10-29 **Registered:** not recorded **Local:** `` /mnt/c/Users/Jammy/Workspace/Science/development/ic50_workspace `` -**RAL:** `` /mnt/ral/jnightin/active/ic50_workspace `` +**RAL:** `` /mnt/ral/jnightin/development/ic50_workspace `` **Current position** @@ -402,13 +402,13 @@ Use the cortex skill. — resume ic50_workspace: read PyAutoCortex projects/ic50 _nothing on the cluster_ -
Recent log — 5 of 12 entries +
Recent log — 5 of 13 entries +- 2026-10-29 — *note* — moved to Science/development/ and RAL /mnt/ral/jnightin/development/ic50_workspace on the human's decision (2026-10-29); hpc/sync.conf HPC_BASE updated, hpc/sync check passes - 2026-10-14 — *note* — output/ vaulted to z_vault/ on laptop and RAL (mv, nothing deleted), anticipating reruns; output/ left empty, results/ untouched; z_vault/ gitignored locally - 2026-09-30 — *note* — Ladder read: EP s/sweep 26/50/140/~300 s at N=5/10/25/50 vs graphical wall 21/42/119/281 s. EP hill_coef widths flat at ~0.8 because util.py reports factor-message (likelihood) widths and the global factor freezes hill_coef; graphical widths shrink but fall to 56/75 and 65/150 within 3 sigma at N=25/50 (overconfident). coef_mean passes 3 sigma for both methods on every completed rung. - 2026-09-30 — *note* — 342411 N=50 failure diagnosed: AssertionError at autofit/messages/abstract.py:316 while projecting the global factor after EP sweep 4 (mirror 66f9f8d5d); the 4th global search had max logL -337 (vs -69 at sweep 3) and logz +/- nan. Library defect: project asserts on non-finite suff_stats and factor_step (optimiser.py:150) does not catch AssertionError, so one bad projection kills the run; no later commit touches the path. A local rerun at 66f9f8d5d converged in sweep 3 without failing (unseeded Dynesty, non-deterministic). Mind bug prompt draft/bug/autofit/ep_project_nonfinite_suff_stats_ic50_n50.md - 2026-09-09 — *run* — 342411 failed — wall 0:38: ep_scale_up: EP scale ladder, sim rungs 5/10/25/50, nlive 150 max_steps 12 — finished 2026-09-09 23:19:29 BST (sacct COMPLETED 00:38:49, the ladder catches rung failures): rungs N=5/10/25 passed the global coef_mean assertions; rung N=50 FAILED — AssertionError: assert np.isfinite(suff_stats).all() in autofit/messages/abstract.py:316 project (via mapper/prior/abstract.py:237), error.342411.err -- 2026-09-09 — *run* — 342412 finished — wall 0:07: ep_scale_up: graphical scale ladder, sim rungs 5/10/25/50, nlive 150 — COMPLETED 2026-09-09 22:48:42 BST (sacct 00:07:59): all four rungs N=5/10/25/50 passed the global coef_mean assertions (within 3σ); results/graphical_ladder_n*_summary.json [full log](https://github.com/PyAutoLabs/PyAutoCortex/blob/main/projects/ic50_workspace.md) diff --git a/projects.yaml b/projects.yaml index 6779120..e53ef49 100644 --- a/projects.yaml +++ b/projects.yaml @@ -118,7 +118,7 @@ euclid_dr1_prelim: euclid_dr1: remote: none local_path: /mnt/c/Users/Jammy/Workspace/Science/active/euclid_dr1 - ral_root: /mnt/ral/jnightin/euclid_dr1 + ral_root: /mnt/ral/jnightin/active/euclid_dr1 mirror: none sync_cli: hpc/sync sync_verbs: [push, pull, logs, sync, push-data-init, pull-full, status, submit, push-submit, jobs, sacct, cancel, wait-and-pull, tail, du, check, clear-logs] @@ -132,10 +132,10 @@ euclid_dr1: analytic_gaussian: remote: none local_path: /mnt/c/Users/Jammy/Workspace/Science/development/analytic_gaussian - ral_root: /mnt/ral/jnightin/active/analytic_gaussian + ral_root: /mnt/ral/jnightin/development/analytic_gaussian mirror: none sync_cli: hpc/sync - sync_verbs: [push, pull, logs, sync, status, submit, push-submit, jobs, sacct, cancel, tail, du, check] + sync_verbs: [push, pull, logs, sync, push-data-init, pull-full, status, submit, push-submit, jobs, sacct, cancel, wait-and-pull, tail, du, check, clear-logs] ledger: wiki/project/state.md assistant: autofit_assistant witness_file: results/**/*.json @@ -146,10 +146,10 @@ analytic_gaussian: ep_toy_gaussian: remote: PyAutoLabs/ep_toy_gaussian local_path: /mnt/c/Users/Jammy/Workspace/Science/development/ep_toy_gaussian - ral_root: /mnt/ral/jnightin/active/ep_toy_gaussian + ral_root: /mnt/ral/jnightin/development/ep_toy_gaussian mirror: none sync_cli: hpc/sync - sync_verbs: [push, pull, logs, sync, status, submit, push-submit, jobs, sacct, cancel, tail, du, check] + sync_verbs: [push, pull, logs, sync, push-data-init, pull-full, status, submit, push-submit, jobs, sacct, cancel, wait-and-pull, tail, du, check, clear-logs] ledger: wiki/project/state.md assistant: autofit_assistant witness_file: results/**/*.json @@ -222,10 +222,10 @@ slope_hierarchy: slope_hierarchy_scale: remote: PyAutoLabs/slope_hierarchy_scale local_path: /mnt/c/Users/Jammy/Workspace/Science/development/slope_hierarchy_scale - ral_root: /mnt/ral/jnightin/active/slope_hierarchy_scale + ral_root: /mnt/ral/jnightin/development/slope_hierarchy_scale mirror: none sync_cli: hpc/sync - sync_verbs: [push, pull, sync, status, submit, push-submit, jobs, tail, check] + sync_verbs: [push, pull, logs, sync, push-data-init, pull-full, status, submit, push-submit, jobs, sacct, cancel, wait-and-pull, tail, du, check, clear-logs] ledger: wiki/project/state.md assistant: autolens_assistant witness_file: results/**/*.json @@ -264,7 +264,7 @@ concr: ic50_workspace: remote: Jammy2211/ic50_workspace local_path: /mnt/c/Users/Jammy/Workspace/Science/development/ic50_workspace - ral_root: /mnt/ral/jnightin/active/ic50_workspace + ral_root: /mnt/ral/jnightin/development/ic50_workspace mirror: none sync_cli: hpc/sync sync_verbs: [push, pull, logs, sync, push-data-init, pull-full, status, submit, push-submit, jobs, sacct, cancel, wait-and-pull, tail, du, check, clear-logs] diff --git a/projects/analytic_gaussian.md b/projects/analytic_gaussian.md index c394571..7275f42 100644 --- a/projects/analytic_gaussian.md +++ b/projects/analytic_gaussian.md @@ -11,6 +11,7 @@ Wave 1 (342413, 200 seeds at N=5) is accepted as the baseline: autofit EP is exa ## Log +- 2026-10-29 — note — moved to Science/development/ and RAL /mnt/ral/jnightin/development/analytic_gaussian on the human's decision (2026-10-29); hpc/sync.conf HPC_BASE updated, hpc/sync check passes - 2026-10-23 — note — second wave-1 finding closed (PyAutoFit#1713, PR #1714 merged, autofit_workspace_test#116): the `invalid value encountered in add` warnings on seeds 25 and 172 are a readout-only defect — under the mode projection an accepted truncated-normal scatter message sits ~19 sigma into its tail, its log_partition underflows to −inf (true log Z ≈ −187.9), the sigma evidence goes +inf and the EP log_evidence NaN from sweep 4; posteriors bit-identical to the ens_n5 sidecars. The library now names the term once per fit and returns an explicit NaN. The cure (tail-stable log_partition via log_ndtr) is message maths, filed in the Mind as draft/bug/autofit/truncated_normal_log_partition_tail_underflow.md for the human. - 2026-10-23 — note — criterion 6 amended (PyAutoFit#1711, PR #1712): the 19/400 zero-SUCCESS seed-legs of wave 1 split into 13 NO_CHANGE-only prior factors (zero-iteration Laplace convergence at the start point, projection accepted, closed-form match a 5e-5 / b 2e-5 on seed 140 leg A — a converged update, not stale; library reporting aligned, no EP maths changed) and 6 BAD_PROJECTION-only hierarchical factors (the #1405 scatter state, cured by projection="moments" #1656; seeds 29 and 71 re-run under moments give HierarchicalFactor SUCCESS 28 and 23). Criterion 6 now counts SUCCESS or NO_CHANGE as a completed update; the matching ensemble_seed.py any_factor_never_succeeded change is the human's to apply in the science checkout. Side finding: seed 29 leg B under moments still misses the sigma cell (a 0.225 / b 0.416) while sigma stays inside [q05, q95]. - 2026-10-14 — note — PyAutoFit#1656 (merged 2026-09-30, released 2026.10.7.1) ports the minimal-EP referee's moment matching as LaplaceOptimiser(projection="moments"); on analytic_gaussian.py leg B autofit EP sigma 6.3401 ± 2.4609 vs closed form 6.5667 ± 2.8832 (mode path 9.3722, BAD_PROJECTION=19). A wave-2 ensemble on the moments path needs a private PyAutoFit checkout on RAL: the shared mirror is 404b3e5f7 and lacks it. The criterion-2 mu-threshold diagnostic is still unrun. diff --git a/projects/ep_toy_gaussian.md b/projects/ep_toy_gaussian.md index 4f330ac..23e2141 100644 --- a/projects/ep_toy_gaussian.md +++ b/projects/ep_toy_gaussian.md @@ -12,6 +12,7 @@ Wave 1 (n50_seed42: 342639 + 342640_[0-2]) failed on infrastructure — EP x3 on ## Log +- 2026-10-29 — note — moved to Science/development/ and RAL /mnt/ral/jnightin/development/ep_toy_gaussian on the human's decision (2026-10-29); hpc/sync.conf HPC_BASE updated, hpc/sync check passes - 2026-10-14 — note — Wave 2 still unsubmitted. RAL shared mirror 404b3e5f7 already carries the EMFILE fixes (#1632/#1634) wave 2 needs; it lacks #1655 (ProjectionException recovery), so a private PyAutoFit checkout is optional here but advisable. - 2026-10-14 — note — output/ vaulted to z_vault/ on laptop and RAL (mv, nothing deleted), anticipating reruns; output/ left empty, results/ untouched; z_vault/ gitignored locally - 2026-09-30 — note — RAL mirror PyAutoFit now 404b3e5f7 (2026-09-27), contains #1632 (b82fb3f69) and #1634 (fa2d540ac); 5 commits behind origin/main, none in blackjax/graphical/dynesty; deps at floor except anesthetic 2.8.14 < 2.9.0 (plots only). Sample n50_seed42_w2 is free locally and on RAL but must be added to toy.SAMPLES (scripts/toy.py) before submission, and SAMPLE must be exported to the job (hpc/sync submit does not forward laptop env) diff --git a/projects/ic50_workspace.md b/projects/ic50_workspace.md index 0b5fd7b..f72ac2e 100644 --- a/projects/ic50_workspace.md +++ b/projects/ic50_workspace.md @@ -12,6 +12,7 @@ The scale ladder is in. EP passes N=5/10/25 with coef_mean within 3σ and cost g ## Log +- 2026-10-29 — note — moved to Science/development/ and RAL /mnt/ral/jnightin/development/ic50_workspace on the human's decision (2026-10-29); hpc/sync.conf HPC_BASE updated, hpc/sync check passes - 2026-10-14 — note — output/ vaulted to z_vault/ on laptop and RAL (mv, nothing deleted), anticipating reruns; output/ left empty, results/ untouched; z_vault/ gitignored locally - 2026-09-30 — note — Ladder read: EP s/sweep 26/50/140/~300 s at N=5/10/25/50 vs graphical wall 21/42/119/281 s. EP hill_coef widths flat at ~0.8 because util.py reports factor-message (likelihood) widths and the global factor freezes hill_coef; graphical widths shrink but fall to 56/75 and 65/150 within 3 sigma at N=25/50 (overconfident). coef_mean passes 3 sigma for both methods on every completed rung. - 2026-09-30 — note — 342411 N=50 failure diagnosed: AssertionError at autofit/messages/abstract.py:316 while projecting the global factor after EP sweep 4 (mirror 66f9f8d5d); the 4th global search had max logL -337 (vs -69 at sweep 3) and logz +/- nan. Library defect: project asserts on non-finite suff_stats and factor_step (optimiser.py:150) does not catch AssertionError, so one bad projection kills the run; no later commit touches the path. A local rerun at 66f9f8d5d converged in sweep 3 without failing (unseeded Dynesty, non-deterministic). Mind bug prompt draft/bug/autofit/ep_project_nonfinite_suff_stats_ic50_n50.md diff --git a/projects/slope_hierarchy_scale.md b/projects/slope_hierarchy_scale.md index 42f8210..11e3de7 100644 --- a/projects/slope_hierarchy_scale.md +++ b/projects/slope_hierarchy_scale.md @@ -12,6 +12,7 @@ The N=25 graphical baselines are in: per-lens fits (342348) give a naive parent ## Log +- 2026-10-29 — note — moved to Science/development/ and RAL /mnt/ral/jnightin/development/slope_hierarchy_scale on the human's decision (2026-10-29); hpc/sync.conf HPC_BASE updated, hpc/sync check passes - 2026-10-14 — note — output/ vaulted to z_vault/ on laptop and RAL (mv, nothing deleted), anticipating reruns; output/ left empty, results/ untouched; z_vault/ gitignored locally - 2026-09-30 — note — graphical baselines recorded late at the 2026-09-30 check-in. 342348_[0-24] per-lens (one_by_one Nautilus, A100, 25/25, 6.9-8.8 min/task, 0 float32 truncations): naive parent mean 1.9992 +/- 0.0178 (SEM), std of medians 0.0889, deconvolved scatter 0.0852 (rms 1-sigma half-width 0.0255); truth inside the 1-sigma interval for 17/25 lenses; per-lens JSONs on RAL at results/one_by_one_sample_n25_seed42/dataset_*.json (not pulled; aggregate never written by the array). 342350_0 joint hierarchical NUTS: science completed despite the post-processing OOM — 500 warm-up (8h08m) + 1000 samples (7h17m), 1 chain, 0 divergences, acceptance 0.84, rhat_max 1.012, ess_min 133/302 params; parent mean 1.9936 [1.9761, 2.0136] 68%, sigma 0.0871 [0.0744, 0.1041] 68% (95% [0.0636, 0.1190]) vs truth 2.0/0.1 and realised draws 1.9917/0.0864 — the N=5 high-scatter bias (0.143) is gone at N=25. Computed from RAL samples.csv (graphical_nuts/f16c32e17f570603a1dc061f95792798/files); results/graphical_nuts_sample_n25_seed42.json never written (killed at 64 GB in latent/arviz post-processing) — pull samples.csv and write the summary offline - 2026-09-30 — run — 342351_0 failed — wall 27:41: n25_scale_up: EP arm on the CPU partition, hpc/batch_cpu/submit_ep, max_steps 12; RAL PyAutoFit mirror 68ff9bd57 predates PyAutoFit#1580, so this arm runs without the stale-mask fixed-point fix — CANCELLED 2026-09-09 20:56:19 BST (sacct 1-03:41:06): log stops at the first Nautilus status line 2026-09-08 17:15:36, no factor finished; superseded by 342410 then 343299, output parked as ep_dead_342351; recorded late at the 2026-09-30 check-in diff --git a/state.json b/state.json index c71318c..7403fdf 100644 --- a/state.json +++ b/state.json @@ -4,7 +4,7 @@ "repo": "PyAutoCortex", "status": "yellow", "headline": "YELLOW \u2014 9 active \u00b7 16 running", - "updated": "2026-10-29T12:49:17Z", + "updated": "2026-10-29T13:43:01Z", "pages_url": "https://pyautolabs.github.io/PyAutoCortex/", "items": [ { From c215a2ac70605fd2aefe0e317771d8340b196c16 Mon Sep 17 00:00:00 2001 From: Jammy2211 Date: Thu, 29 Oct 2026 13:43:17 +0000 Subject: [PATCH 2/3] ledgers: name the move by category, not by absolute path Co-Authored-By: Claude Opus 5.5 --- dashboard.html | 10 +++++----- dashboard.md | 8 ++++---- projects/analytic_gaussian.md | 2 +- projects/ep_toy_gaussian.md | 2 +- projects/ic50_workspace.md | 2 +- projects/slope_hierarchy_scale.md | 2 +- state.json | 2 +- 7 files changed, 14 insertions(+), 14 deletions(-) diff --git a/dashboard.html b/dashboard.html index 771882a..c95a3bc 100644 --- a/dashboard.html +++ b/dashboard.html @@ -332,7 +332,7 @@ - Open ic50_workspace: https://github.com/Jammy2211/ic50_workspace - Open autolens_inference: https://github.com/PyAutoLabs/autolens_inference - Open autofit_inference: https://github.com/PyAutoLabs/autofit_inference -- Open euclid_detailed: https://github.com/PyAutoLabs/euclid_detailed
Last updated 2026-10-29 13:43:01 UTC
↻ Update

+- Open euclid_detailed: https://github.com/PyAutoLabs/euclid_detailed
Last updated 2026-10-29 13:43:17 UTC
↻ Update

@@ -491,7 +491,7 @@

Runs (0)

nothing on the cluster

Recent log — 5 of 10 entries
    -
  • 2026-10-29 — note — moved to Science/development/ and RAL /mnt/ral/jnightin/development/analytic_gaussian on the human's decision (2026-10-29); hpc/sync.conf HPC_BASE updated, hpc/sync check passes
    • +
  • 2026-10-29 — note — moved to the development/ category on the laptop and on RAL (ral_root in its row) on the human's decision (2026-10-29); hpc/sync.conf HPC_BASE updated, hpc/sync check passes
  • 2026-10-23 — note —
    second wave-1 finding closed (PyAutoFit#1713, PR #1714 merged, autofit_workspace_test#116): the `invalid value encountered in add` warnings on seeds 25 and 172 are a readout-only defect — under the mode projection an accepted truncated-nor…show less

    second wave-1 finding closed (PyAutoFit#1713, PR #1714 merged, autofit_workspace_test#116): the `invalid value encountered in add` warnings on seeds 25 and 172 are a readout-only defect — under the mode projection an accepted truncated-normal scatter message sits ~19 sigma into its tail, its log_partition underflows to −inf (true log Z ≈ −187.9), the sigma evidence goes +inf and the EP log_evidence NaN from sweep 4; posteriors bit-identical to the ens_n5 sidecars. The library now names the term once per fit and returns an explicit NaN. The cure (tail-stable log_partition via log_ndtr) is message maths, filed in the Mind as draft/bug/autofit/truncated_normal_log_partition_tail_underflow.md for the human.

  • 2026-10-23 — note —
    criterion 6 amended (PyAutoFit#1711, PR #1712): the 19/400 zero-SUCCESS seed-legs of wave 1 split into 13 NO_CHANGE-only prior factors (zero-iteration Laplace convergence at the start point, projection accepted, closed-form match a 5e-5 /…show less

    criterion 6 amended (PyAutoFit#1711, PR #1712): the 19/400 zero-SUCCESS seed-legs of wave 1 split into 13 NO_CHANGE-only prior factors (zero-iteration Laplace convergence at the start point, projection accepted, closed-form match a 5e-5 / b 2e-5 on seed 140 leg A — a converged update, not stale; library reporting aligned, no EP maths changed) and 6 BAD_PROJECTION-only hierarchical factors (the #1405 scatter state, cured by projection="moments" #1656; seeds 29 and 71 re-run under moments give HierarchicalFactor SUCCESS 28 and 23). Criterion 6 now counts SUCCESS or NO_CHANGE as a completed update; the matching ensemble_seed.py any_factor_never_succeeded change is the human's to apply in the science checkout. Side finding: seed 29 leg B under moments still misses the sigma cell (a 0.225 / b 0.416) while sigma stays inside [q05, q95].

  • 2026-10-14 — note —
    PyAutoFit#1656 (merged 2026-09-30, released 2026.10.7.1) ports the minimal-EP referee's moment matching as LaplaceOptimiser(projection="moments"); on analytic_gaussian.py leg B autofit EP sigma 6.3401 ± 2.4609 vs closed form 6.5667 ± 2.883…show less

    PyAutoFit#1656 (merged 2026-09-30, released 2026.10.7.1) ports the minimal-EP referee's moment matching as LaplaceOptimiser(projection="moments"); on analytic_gaussian.py leg B autofit EP sigma 6.3401 ± 2.4609 vs closed form 6.5667 ± 2.8832 (mode path 9.3722, BAD_PROJECTION=19). A wave-2 ensemble on the moments path needs a private PyAutoFit checkout on RAL: the shared mirror is 404b3e5f7 and lacks it. The criterion-2 mu-threshold diagnostic is still unrun.

    • @@ -518,7 +518,7 @@

    Runs (0)

    nothing on the cluster

    Recent log — 5 of 11 entries
      -
    • 2026-10-29 — note — moved to Science/development/ and RAL /mnt/ral/jnightin/development/ep_toy_gaussian on the human's decision (2026-10-29); hpc/sync.conf HPC_BASE updated, hpc/sync check passes
      • +
    • 2026-10-29 — note — moved to the development/ category on the laptop and on RAL (ral_root in its row) on the human's decision (2026-10-29); hpc/sync.conf HPC_BASE updated, hpc/sync check passes
    • 2026-10-14 — note — Wave 2 still unsubmitted. RAL shared mirror 404b3e5f7 already carries the EMFILE fixes (#1632/#1634) wave 2 needs; it lacks #1655 (ProjectionException recovery), so a private PyAutoFit checkout is optional here but advisable.
    • 2026-10-14 — note — output/ vaulted to z_vault/ on laptop and RAL (mv, nothing deleted), anticipating reruns; output/ left empty, results/ untouched; z_vault/ gitignored locally
    • 2026-09-30 — note —
      RAL mirror PyAutoFit now 404b3e5f7 (2026-09-27), contains #1632 (b82fb3f69) and #1634 (fa2d540ac); 5 commits behind origin/main, none in blackjax/graphical/dynesty; deps at floor except anesthetic 2.8.14 < 2.9.0 (plots only). Sample n50_se…show less

      RAL mirror PyAutoFit now 404b3e5f7 (2026-09-27), contains #1632 (b82fb3f69) and #1634 (fa2d540ac); 5 commits behind origin/main, none in blackjax/graphical/dynesty; deps at floor except anesthetic 2.8.14 < 2.9.0 (plots only). Sample n50_seed42_w2 is free locally and on RAL but must be added to toy.SAMPLES (scripts/toy.py) before submission, and SAMPLE must be exported to the job (hpc/sync submit does not forward laptop env)

      • @@ -545,7 +545,7 @@

      Runs (0)

      nothing on the cluster

      Recent log — 5 of 15 entries
        -
      • 2026-10-29 — note — moved to Science/development/ and RAL /mnt/ral/jnightin/development/slope_hierarchy_scale on the human's decision (2026-10-29); hpc/sync.conf HPC_BASE updated, hpc/sync check passes
        • +
      • 2026-10-29 — note — moved to the development/ category on the laptop and on RAL (ral_root in its row) on the human's decision (2026-10-29); hpc/sync.conf HPC_BASE updated, hpc/sync check passes
      • 2026-10-14 — note — output/ vaulted to z_vault/ on laptop and RAL (mv, nothing deleted), anticipating reruns; output/ left empty, results/ untouched; z_vault/ gitignored locally
      • 2026-09-30 — note —
        graphical baselines recorded late at the 2026-09-30 check-in. 342348_[0-24] per-lens (one_by_one Nautilus, A100, 25/25, 6.9-8.8 min/task, 0 float32 truncations): naive parent mean 1.9992 +/- 0.0178 (SEM), std of medians 0.0889, deconvolved…show less

        graphical baselines recorded late at the 2026-09-30 check-in. 342348_[0-24] per-lens (one_by_one Nautilus, A100, 25/25, 6.9-8.8 min/task, 0 float32 truncations): naive parent mean 1.9992 +/- 0.0178 (SEM), std of medians 0.0889, deconvolved scatter 0.0852 (rms 1-sigma half-width 0.0255); truth inside the 1-sigma interval for 17/25 lenses; per-lens JSONs on RAL at results/one_by_one_sample_n25_seed42/dataset_*.json (not pulled; aggregate never written by the array). 342350_0 joint hierarchical NUTS: science completed despite the post-processing OOM — 500 warm-up (8h08m) + 1000 samples (7h17m), 1 chain, 0 divergences, acceptance 0.84, rhat_max 1.012, ess_min 133/302 params; parent mean 1.9936 [1.9761, 2.0136] 68%, sigma 0.0871 [0.0744, 0.1041] 68% (95% [0.0636, 0.1190]) vs truth 2.0/0.1 and realised draws 1.9917/0.0864 — the N=5 high-scatter bias (0.143) is gone at N=25. Computed from RAL samples.csv (graphical_nuts/f16c32e17f570603a1dc061f95792798/files); results/graphical_nuts_sample_n25_seed42.json never written (killed at 64 GB in latent/arviz post-processing) — pull samples.csv and write the summary offline

      • 2026-09-30 — run —
        342351_0 failed — wall 27:41: n25_scale_up: EP arm on the CPU partition, hpc/batch_cpu/submit_ep, max_steps 12; RAL PyAutoFit mirror 68ff9bd57 predates PyAutoFit#1580, so this arm runs without the stale-mask fixed-point fix — CANCELLED 202…show less

        342351_0 failed — wall 27:41: n25_scale_up: EP arm on the CPU partition, hpc/batch_cpu/submit_ep, max_steps 12; RAL PyAutoFit mirror 68ff9bd57 predates PyAutoFit#1580, so this arm runs without the stale-mask fixed-point fix — CANCELLED 2026-09-09 20:56:19 BST (sacct 1-03:41:06): log stops at the first Nautilus status line 2026-09-08 17:15:36, no factor finished; superseded by 342410 then 343299, output parked as ep_dead_342351; recorded late at the 2026-09-30 check-in

        • @@ -572,7 +572,7 @@

        Runs (0)

        nothing on the cluster

        Recent log — 5 of 13 entries
          -
        • 2026-10-29 — note — moved to Science/development/ and RAL /mnt/ral/jnightin/development/ic50_workspace on the human's decision (2026-10-29); hpc/sync.conf HPC_BASE updated, hpc/sync check passes
          • +
        • 2026-10-29 — note — moved to the development/ category on the laptop and on RAL (ral_root in its row) on the human's decision (2026-10-29); hpc/sync.conf HPC_BASE updated, hpc/sync check passes
        • 2026-10-14 — note — output/ vaulted to z_vault/ on laptop and RAL (mv, nothing deleted), anticipating reruns; output/ left empty, results/ untouched; z_vault/ gitignored locally
        • 2026-09-30 — note —
          Ladder read: EP s/sweep 26/50/140/~300 s at N=5/10/25/50 vs graphical wall 21/42/119/281 s. EP hill_coef widths flat at ~0.8 because util.py reports factor-message (likelihood) widths and the global factor freezes hill_coef; graphical widt…show less

          Ladder read: EP s/sweep 26/50/140/~300 s at N=5/10/25/50 vs graphical wall 21/42/119/281 s. EP hill_coef widths flat at ~0.8 because util.py reports factor-message (likelihood) widths and the global factor freezes hill_coef; graphical widths shrink but fall to 56/75 and 65/150 within 3 sigma at N=25/50 (overconfident). coef_mean passes 3 sigma for both methods on every completed rung.

        • 2026-09-30 — note —
          342411 N=50 failure diagnosed: AssertionError at autofit/messages/abstract.py:316 while projecting the global factor after EP sweep 4 (mirror 66f9f8d5d); the 4th global search had max logL -337 (vs -69 at sweep 3) and logz +/- nan. Library…show less

          342411 N=50 failure diagnosed: AssertionError at autofit/messages/abstract.py:316 while projecting the global factor after EP sweep 4 (mirror 66f9f8d5d); the 4th global search had max logL -337 (vs -69 at sweep 3) and logz +/- nan. Library defect: project asserts on non-finite suff_stats and factor_step (optimiser.py:150) does not catch AssertionError, so one bad projection kills the run; no later commit touches the path. A local rerun at 66f9f8d5d converged in sweep 3 without failing (unseeded Dynesty, non-deterministic). Mind bug prompt draft/bug/autofit/ep_project_nonfinite_suff_stats_ic50_n50.md

          • diff --git a/dashboard.md b/dashboard.md index 8f6898d..d8b3928 100644 --- a/dashboard.md +++ b/dashboard.md @@ -284,7 +284,7 @@ _nothing on the cluster_
          Recent log — 5 of 10 entries -- 2026-10-29 — *note* — moved to Science/development/ and RAL /mnt/ral/jnightin/development/analytic_gaussian on the human's decision (2026-10-29); hpc/sync.conf HPC_BASE updated, hpc/sync check passes +- 2026-10-29 — *note* — moved to the development/ category on the laptop and on RAL (ral_root in its row) on the human's decision (2026-10-29); hpc/sync.conf HPC_BASE updated, hpc/sync check passes - 2026-10-23 — *note* — second wave-1 finding closed (PyAutoFit#1713, PR #1714 merged, autofit_workspace_test#116): the `invalid value encountered in add` warnings on seeds 25 and 172 are a readout-only defect — under the mode projection an accepted truncated-normal scatter message sits ~19 sigma into its tail, its log_partition underflows to −inf (true log Z ≈ −187.9), the sigma evidence goes +inf and the EP log_evidence NaN from sweep 4; posteriors bit-identical to the ens_n5 sidecars. The library now names the term once per fit and returns an explicit NaN. The cure (tail-stable log_partition via log_ndtr) is message maths, filed in the Mind as draft/bug/autofit/truncated_normal_log_partition_tail_underflow.md for the human. - 2026-10-23 — *note* — criterion 6 amended (PyAutoFit#1711, PR #1712): the 19/400 zero-SUCCESS seed-legs of wave 1 split into 13 NO_CHANGE-only prior factors (zero-iteration Laplace convergence at the start point, projection accepted, closed-form match a 5e-5 / b 2e-5 on seed 140 leg A — a converged update, not stale; library reporting aligned, no EP maths changed) and 6 BAD_PROJECTION-only hierarchical factors (the #1405 scatter state, cured by projection="moments" #1656; seeds 29 and 71 re-run under moments give HierarchicalFactor SUCCESS 28 and 23). Criterion 6 now counts SUCCESS or NO_CHANGE as a completed update; the matching ensemble_seed.py any_factor_never_succeeded change is the human's to apply in the science checkout. Side finding: seed 29 leg B under moments still misses the sigma cell (a 0.225 / b 0.416) while sigma stays inside [q05, q95]. - 2026-10-14 — *note* — PyAutoFit#1656 (merged 2026-09-30, released 2026.10.7.1) ports the minimal-EP referee's moment matching as LaplaceOptimiser(projection="moments"); on analytic_gaussian.py leg B autofit EP sigma 6.3401 ± 2.4609 vs closed form 6.5667 ± 2.8832 (mode path 9.3722, BAD_PROJECTION=19). A wave-2 ensemble on the moments path needs a private PyAutoFit checkout on RAL: the shared mirror is 404b3e5f7 and lacks it. The criterion-2 mu-threshold diagnostic is still unrun. @@ -324,7 +324,7 @@ _nothing on the cluster_
          Recent log — 5 of 11 entries -- 2026-10-29 — *note* — moved to Science/development/ and RAL /mnt/ral/jnightin/development/ep_toy_gaussian on the human's decision (2026-10-29); hpc/sync.conf HPC_BASE updated, hpc/sync check passes +- 2026-10-29 — *note* — moved to the development/ category on the laptop and on RAL (ral_root in its row) on the human's decision (2026-10-29); hpc/sync.conf HPC_BASE updated, hpc/sync check passes - 2026-10-14 — *note* — Wave 2 still unsubmitted. RAL shared mirror 404b3e5f7 already carries the EMFILE fixes (#1632/#1634) wave 2 needs; it lacks #1655 (ProjectionException recovery), so a private PyAutoFit checkout is optional here but advisable. - 2026-10-14 — *note* — output/ vaulted to z_vault/ on laptop and RAL (mv, nothing deleted), anticipating reruns; output/ left empty, results/ untouched; z_vault/ gitignored locally - 2026-09-30 — *note* — RAL mirror PyAutoFit now 404b3e5f7 (2026-09-27), contains #1632 (b82fb3f69) and #1634 (fa2d540ac); 5 commits behind origin/main, none in blackjax/graphical/dynesty; deps at floor except anesthetic 2.8.14 < 2.9.0 (plots only). Sample n50_seed42_w2 is free locally and on RAL but must be added to toy.SAMPLES (scripts/toy.py) before submission, and SAMPLE must be exported to the job (hpc/sync submit does not forward laptop env) @@ -364,7 +364,7 @@ _nothing on the cluster_
          Recent log — 5 of 15 entries -- 2026-10-29 — *note* — moved to Science/development/ and RAL /mnt/ral/jnightin/development/slope_hierarchy_scale on the human's decision (2026-10-29); hpc/sync.conf HPC_BASE updated, hpc/sync check passes +- 2026-10-29 — *note* — moved to the development/ category on the laptop and on RAL (ral_root in its row) on the human's decision (2026-10-29); hpc/sync.conf HPC_BASE updated, hpc/sync check passes - 2026-10-14 — *note* — output/ vaulted to z_vault/ on laptop and RAL (mv, nothing deleted), anticipating reruns; output/ left empty, results/ untouched; z_vault/ gitignored locally - 2026-09-30 — *note* — graphical baselines recorded late at the 2026-09-30 check-in. 342348_[0-24] per-lens (one_by_one Nautilus, A100, 25/25, 6.9-8.8 min/task, 0 float32 truncations): naive parent mean 1.9992 +/- 0.0178 (SEM), std of medians 0.0889, deconvolved scatter 0.0852 (rms 1-sigma half-width 0.0255); truth inside the 1-sigma interval for 17/25 lenses; per-lens JSONs on RAL at results/one_by_one_sample_n25_seed42/dataset_*.json (not pulled; aggregate never written by the array). 342350_0 joint hierarchical NUTS: science completed despite the post-processing OOM — 500 warm-up (8h08m) + 1000 samples (7h17m), 1 chain, 0 divergences, acceptance 0.84, rhat_max 1.012, ess_min 133/302 params; parent mean 1.9936 [1.9761, 2.0136] 68%, sigma 0.0871 [0.0744, 0.1041] 68% (95% [0.0636, 0.1190]) vs truth 2.0/0.1 and realised draws 1.9917/0.0864 — the N=5 high-scatter bias (0.143) is gone at N=25. Computed from RAL samples.csv (graphical_nuts/f16c32e17f570603a1dc061f95792798/files); results/graphical_nuts_sample_n25_seed42.json never written (killed at 64 GB in latent/arviz post-processing) — pull samples.csv and write the summary offline - 2026-09-30 — *run* — 342351_0 failed — wall 27:41: n25_scale_up: EP arm on the CPU partition, hpc/batch_cpu/submit_ep, max_steps 12; RAL PyAutoFit mirror 68ff9bd57 predates PyAutoFit#1580, so this arm runs without the stale-mask fixed-point fix — CANCELLED 2026-09-09 20:56:19 BST (sacct 1-03:41:06): log stops at the first Nautilus status line 2026-09-08 17:15:36, no factor finished; superseded by 342410 then 343299, output parked as ep_dead_342351; recorded late at the 2026-09-30 check-in @@ -404,7 +404,7 @@ _nothing on the cluster_
          Recent log — 5 of 13 entries -- 2026-10-29 — *note* — moved to Science/development/ and RAL /mnt/ral/jnightin/development/ic50_workspace on the human's decision (2026-10-29); hpc/sync.conf HPC_BASE updated, hpc/sync check passes +- 2026-10-29 — *note* — moved to the development/ category on the laptop and on RAL (ral_root in its row) on the human's decision (2026-10-29); hpc/sync.conf HPC_BASE updated, hpc/sync check passes - 2026-10-14 — *note* — output/ vaulted to z_vault/ on laptop and RAL (mv, nothing deleted), anticipating reruns; output/ left empty, results/ untouched; z_vault/ gitignored locally - 2026-09-30 — *note* — Ladder read: EP s/sweep 26/50/140/~300 s at N=5/10/25/50 vs graphical wall 21/42/119/281 s. EP hill_coef widths flat at ~0.8 because util.py reports factor-message (likelihood) widths and the global factor freezes hill_coef; graphical widths shrink but fall to 56/75 and 65/150 within 3 sigma at N=25/50 (overconfident). coef_mean passes 3 sigma for both methods on every completed rung. - 2026-09-30 — *note* — 342411 N=50 failure diagnosed: AssertionError at autofit/messages/abstract.py:316 while projecting the global factor after EP sweep 4 (mirror 66f9f8d5d); the 4th global search had max logL -337 (vs -69 at sweep 3) and logz +/- nan. Library defect: project asserts on non-finite suff_stats and factor_step (optimiser.py:150) does not catch AssertionError, so one bad projection kills the run; no later commit touches the path. A local rerun at 66f9f8d5d converged in sweep 3 without failing (unseeded Dynesty, non-deterministic). Mind bug prompt draft/bug/autofit/ep_project_nonfinite_suff_stats_ic50_n50.md diff --git a/projects/analytic_gaussian.md b/projects/analytic_gaussian.md index 7275f42..4fb56e0 100644 --- a/projects/analytic_gaussian.md +++ b/projects/analytic_gaussian.md @@ -11,7 +11,7 @@ Wave 1 (342413, 200 seeds at N=5) is accepted as the baseline: autofit EP is exa ## Log -- 2026-10-29 — note — moved to Science/development/ and RAL /mnt/ral/jnightin/development/analytic_gaussian on the human's decision (2026-10-29); hpc/sync.conf HPC_BASE updated, hpc/sync check passes +- 2026-10-29 — note — moved to the development/ category on the laptop and on RAL (ral_root in its row) on the human's decision (2026-10-29); hpc/sync.conf HPC_BASE updated, hpc/sync check passes - 2026-10-23 — note — second wave-1 finding closed (PyAutoFit#1713, PR #1714 merged, autofit_workspace_test#116): the `invalid value encountered in add` warnings on seeds 25 and 172 are a readout-only defect — under the mode projection an accepted truncated-normal scatter message sits ~19 sigma into its tail, its log_partition underflows to −inf (true log Z ≈ −187.9), the sigma evidence goes +inf and the EP log_evidence NaN from sweep 4; posteriors bit-identical to the ens_n5 sidecars. The library now names the term once per fit and returns an explicit NaN. The cure (tail-stable log_partition via log_ndtr) is message maths, filed in the Mind as draft/bug/autofit/truncated_normal_log_partition_tail_underflow.md for the human. - 2026-10-23 — note — criterion 6 amended (PyAutoFit#1711, PR #1712): the 19/400 zero-SUCCESS seed-legs of wave 1 split into 13 NO_CHANGE-only prior factors (zero-iteration Laplace convergence at the start point, projection accepted, closed-form match a 5e-5 / b 2e-5 on seed 140 leg A — a converged update, not stale; library reporting aligned, no EP maths changed) and 6 BAD_PROJECTION-only hierarchical factors (the #1405 scatter state, cured by projection="moments" #1656; seeds 29 and 71 re-run under moments give HierarchicalFactor SUCCESS 28 and 23). Criterion 6 now counts SUCCESS or NO_CHANGE as a completed update; the matching ensemble_seed.py any_factor_never_succeeded change is the human's to apply in the science checkout. Side finding: seed 29 leg B under moments still misses the sigma cell (a 0.225 / b 0.416) while sigma stays inside [q05, q95]. - 2026-10-14 — note — PyAutoFit#1656 (merged 2026-09-30, released 2026.10.7.1) ports the minimal-EP referee's moment matching as LaplaceOptimiser(projection="moments"); on analytic_gaussian.py leg B autofit EP sigma 6.3401 ± 2.4609 vs closed form 6.5667 ± 2.8832 (mode path 9.3722, BAD_PROJECTION=19). A wave-2 ensemble on the moments path needs a private PyAutoFit checkout on RAL: the shared mirror is 404b3e5f7 and lacks it. The criterion-2 mu-threshold diagnostic is still unrun. diff --git a/projects/ep_toy_gaussian.md b/projects/ep_toy_gaussian.md index 23e2141..849def6 100644 --- a/projects/ep_toy_gaussian.md +++ b/projects/ep_toy_gaussian.md @@ -12,7 +12,7 @@ Wave 1 (n50_seed42: 342639 + 342640_[0-2]) failed on infrastructure — EP x3 on ## Log -- 2026-10-29 — note — moved to Science/development/ and RAL /mnt/ral/jnightin/development/ep_toy_gaussian on the human's decision (2026-10-29); hpc/sync.conf HPC_BASE updated, hpc/sync check passes +- 2026-10-29 — note — moved to the development/ category on the laptop and on RAL (ral_root in its row) on the human's decision (2026-10-29); hpc/sync.conf HPC_BASE updated, hpc/sync check passes - 2026-10-14 — note — Wave 2 still unsubmitted. RAL shared mirror 404b3e5f7 already carries the EMFILE fixes (#1632/#1634) wave 2 needs; it lacks #1655 (ProjectionException recovery), so a private PyAutoFit checkout is optional here but advisable. - 2026-10-14 — note — output/ vaulted to z_vault/ on laptop and RAL (mv, nothing deleted), anticipating reruns; output/ left empty, results/ untouched; z_vault/ gitignored locally - 2026-09-30 — note — RAL mirror PyAutoFit now 404b3e5f7 (2026-09-27), contains #1632 (b82fb3f69) and #1634 (fa2d540ac); 5 commits behind origin/main, none in blackjax/graphical/dynesty; deps at floor except anesthetic 2.8.14 < 2.9.0 (plots only). Sample n50_seed42_w2 is free locally and on RAL but must be added to toy.SAMPLES (scripts/toy.py) before submission, and SAMPLE must be exported to the job (hpc/sync submit does not forward laptop env) diff --git a/projects/ic50_workspace.md b/projects/ic50_workspace.md index f72ac2e..c49151d 100644 --- a/projects/ic50_workspace.md +++ b/projects/ic50_workspace.md @@ -12,7 +12,7 @@ The scale ladder is in. EP passes N=5/10/25 with coef_mean within 3σ and cost g ## Log -- 2026-10-29 — note — moved to Science/development/ and RAL /mnt/ral/jnightin/development/ic50_workspace on the human's decision (2026-10-29); hpc/sync.conf HPC_BASE updated, hpc/sync check passes +- 2026-10-29 — note — moved to the development/ category on the laptop and on RAL (ral_root in its row) on the human's decision (2026-10-29); hpc/sync.conf HPC_BASE updated, hpc/sync check passes - 2026-10-14 — note — output/ vaulted to z_vault/ on laptop and RAL (mv, nothing deleted), anticipating reruns; output/ left empty, results/ untouched; z_vault/ gitignored locally - 2026-09-30 — note — Ladder read: EP s/sweep 26/50/140/~300 s at N=5/10/25/50 vs graphical wall 21/42/119/281 s. EP hill_coef widths flat at ~0.8 because util.py reports factor-message (likelihood) widths and the global factor freezes hill_coef; graphical widths shrink but fall to 56/75 and 65/150 within 3 sigma at N=25/50 (overconfident). coef_mean passes 3 sigma for both methods on every completed rung. - 2026-09-30 — note — 342411 N=50 failure diagnosed: AssertionError at autofit/messages/abstract.py:316 while projecting the global factor after EP sweep 4 (mirror 66f9f8d5d); the 4th global search had max logL -337 (vs -69 at sweep 3) and logz +/- nan. Library defect: project asserts on non-finite suff_stats and factor_step (optimiser.py:150) does not catch AssertionError, so one bad projection kills the run; no later commit touches the path. A local rerun at 66f9f8d5d converged in sweep 3 without failing (unseeded Dynesty, non-deterministic). Mind bug prompt draft/bug/autofit/ep_project_nonfinite_suff_stats_ic50_n50.md diff --git a/projects/slope_hierarchy_scale.md b/projects/slope_hierarchy_scale.md index 11e3de7..505f629 100644 --- a/projects/slope_hierarchy_scale.md +++ b/projects/slope_hierarchy_scale.md @@ -12,7 +12,7 @@ The N=25 graphical baselines are in: per-lens fits (342348) give a naive parent ## Log -- 2026-10-29 — note — moved to Science/development/ and RAL /mnt/ral/jnightin/development/slope_hierarchy_scale on the human's decision (2026-10-29); hpc/sync.conf HPC_BASE updated, hpc/sync check passes +- 2026-10-29 — note — moved to the development/ category on the laptop and on RAL (ral_root in its row) on the human's decision (2026-10-29); hpc/sync.conf HPC_BASE updated, hpc/sync check passes - 2026-10-14 — note — output/ vaulted to z_vault/ on laptop and RAL (mv, nothing deleted), anticipating reruns; output/ left empty, results/ untouched; z_vault/ gitignored locally - 2026-09-30 — note — graphical baselines recorded late at the 2026-09-30 check-in. 342348_[0-24] per-lens (one_by_one Nautilus, A100, 25/25, 6.9-8.8 min/task, 0 float32 truncations): naive parent mean 1.9992 +/- 0.0178 (SEM), std of medians 0.0889, deconvolved scatter 0.0852 (rms 1-sigma half-width 0.0255); truth inside the 1-sigma interval for 17/25 lenses; per-lens JSONs on RAL at results/one_by_one_sample_n25_seed42/dataset_*.json (not pulled; aggregate never written by the array). 342350_0 joint hierarchical NUTS: science completed despite the post-processing OOM — 500 warm-up (8h08m) + 1000 samples (7h17m), 1 chain, 0 divergences, acceptance 0.84, rhat_max 1.012, ess_min 133/302 params; parent mean 1.9936 [1.9761, 2.0136] 68%, sigma 0.0871 [0.0744, 0.1041] 68% (95% [0.0636, 0.1190]) vs truth 2.0/0.1 and realised draws 1.9917/0.0864 — the N=5 high-scatter bias (0.143) is gone at N=25. Computed from RAL samples.csv (graphical_nuts/f16c32e17f570603a1dc061f95792798/files); results/graphical_nuts_sample_n25_seed42.json never written (killed at 64 GB in latent/arviz post-processing) — pull samples.csv and write the summary offline - 2026-09-30 — run — 342351_0 failed — wall 27:41: n25_scale_up: EP arm on the CPU partition, hpc/batch_cpu/submit_ep, max_steps 12; RAL PyAutoFit mirror 68ff9bd57 predates PyAutoFit#1580, so this arm runs without the stale-mask fixed-point fix — CANCELLED 2026-09-09 20:56:19 BST (sacct 1-03:41:06): log stops at the first Nautilus status line 2026-09-08 17:15:36, no factor finished; superseded by 342410 then 343299, output parked as ep_dead_342351; recorded late at the 2026-09-30 check-in diff --git a/state.json b/state.json index 7403fdf..919bda9 100644 --- a/state.json +++ b/state.json @@ -4,7 +4,7 @@ "repo": "PyAutoCortex", "status": "yellow", "headline": "YELLOW \u2014 9 active \u00b7 16 running", - "updated": "2026-10-29T13:43:01Z", + "updated": "2026-10-29T13:43:17Z", "pages_url": "https://pyautolabs.github.io/PyAutoCortex/", "items": [ { From 0c334657a6b7deb1269a2a9f403041e394a9c0b5 Mon Sep 17 00:00:00 2001 From: Jammy2211 Date: Thu, 29 Oct 2026 13:59:19 +0000 Subject: [PATCH 3/3] projects.yaml: euclid ral_root under complete/; register euclid_group, cti, vignette retired; concr repointed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit On the human's decisions (2026-10-29): euclid's RAL tree moved from active/ to complete/ (hpc/sync.conf follows); the storage folders complete/euclid_group, archive/cti and archive/vignette registered as retired via `cortex.py birth --status retired`; concr's local_path repointed to complete/concr_cosmology (its tree; archive/concr is gone). archive/euclid_detailed_2 was deleted locally by human decision (all work is in euclid_detailed), so it gets no row. `cortex drift`: none — projects.yaml and the Science tree agree. Co-Authored-By: Claude Opus 5.5 --- dashboard.html | 16 +++++++++----- dashboard.md | 14 +++++++----- projects.yaml | 48 +++++++++++++++++++++++++++++++++++++--- projects/cti.md | 14 ++++++++++++ projects/euclid.md | 1 + projects/euclid_group.md | 14 ++++++++++++ projects/vignette.md | 14 ++++++++++++ state.json | 2 +- 8 files changed, 108 insertions(+), 15 deletions(-) create mode 100644 projects/cti.md create mode 100644 projects/euclid_group.md create mode 100644 projects/vignette.md diff --git a/dashboard.html b/dashboard.html index c95a3bc..f8f17cc 100644 --- a/dashboard.html +++ b/dashboard.html @@ -332,7 +332,7 @@ - Open ic50_workspace: https://github.com/Jammy2211/ic50_workspace - Open autolens_inference: https://github.com/PyAutoLabs/autolens_inference - Open autofit_inference: https://github.com/PyAutoLabs/autofit_inference -- Open euclid_detailed: https://github.com/PyAutoLabs/euclid_detailed
          Last updated 2026-10-29 13:43:17 UTC
          ↻ Update

          +- Open euclid_detailed: https://github.com/PyAutoLabs/euclid_detailed
          Last updated 2026-10-29 13:59:06 UTC
          ↻ Update

          @@ -661,20 +661,21 @@

          Runs (0)

          -

          euclid

          The original Euclid DR1 grade-AB catalogue project, superseded by euclid_dr1dormant · no runs recorded · updated 2026-10-15           +

          euclid

          The original Euclid DR1 grade-AB catalogue project, superseded by euclid_dr1dormant · no runs recorded · updated 2026-10-29
          Registered:
          not recorded
          Local:
          /mnt/c/Users/Jammy/Workspace/Science/complete/euclid
          -
          RAL:
          /mnt/ral/jnightin/active/euclid
          +
          RAL:
          /mnt/ral/jnightin/complete/euclid

          Current position

          Dormant: superseded by euclid_dr1 (euclid_dr1_prelim is defunct). The tree is kept under Science/active and RAL `active/euclid` (category layout, 2026-10-15); no question is open.

          Runs (0)

          nothing on the cluster

          -
          Recent log — 4 of 4 entries +
          Recent log — 5 of 5 entries
            +
          • 2026-10-29 — note — RAL tree moved to the complete/ category to mirror the laptop, on the human's decision (2026-10-29); hpc/sync.conf HPC_BASE updated
          • 2026-10-15 — note — superseded by euclid_dr1 (euclid_dr1_prelim defunct) per the human; stays dormant; folder under Science/active and RAL active/; ral_root repointed to the category layout (PyAutoCortex#67)
          • 2026-08-28 — note — question: Do we recover Sersic indices or is prior pile-up artefact — planned, never run [archive/tasks/euclid/sersic_index_recovery.md]
          • 2026-08-28 — note — question: Can a fitted Euclid lens be resimulated and recovered — planned, never run [archive/tasks/euclid/resimulate_fitted_lens_simulator.md]
            • @@ -685,7 +686,7 @@

            Runs (0)

          -
          8 retired
            +
            11 retired
            • inference_programme — The first inference programme, retired and restarted as autolens_inference — retired 2026-09-07: restarting from scratch: the project setup was unsatisfactory and it informed how the Cortex manages projects; RAL outputs stashed at /mnt/ral/jnightin/inference_programme_retired_2026-09-07
            • subhalo — SLaM subhalo modelling of the SLACS lens sample — retired 2026-10-29: defunct; local tree deleted by the human 2026-10-29
            • euclid_dr1_prelim — Refit the ten DR1-prelim tiles and rebuild the catalogue on RAL — retired 2026-09-25: retire euclid_dr1_prelim; tree moved to z_vault by the human
              • @@ -694,6 +695,9 @@

              Runs (0)

            • pj011646_final — Final masked PJ0116-24 power-law and multipole subhalo comparison — retired 2026-10-14: complete; tree moved to z_projects_complete 2026-10-14
            • cowls — euclid_detailed SLaM pipeline on COWLS JWST strong lenses, from COSJ100024+015334 — retired 2026-10-29: defunct; local tree deleted by the human 2026-10-29
            • cowls_ansofi — autolens_inference Delaunay-1250 numba CPU SLaM on COSJ100024+015334, F444W only — retired 2026-10-29: defunct; local tree deleted by the human 2026-10-29
              • +
            • euclid_group — Euclid group-scale lens modelling, complete; kept for reference — registered retired 2026-10-29 by human decision: storage tree, non-git; no RAL tree verified
              • +
            • cti — Charge transfer inefficiency modelling, archived COSMA-era storage — registered retired 2026-10-29 by human decision: archived CTI storage tree (COSMA era), non-git; no RAL tree verified
              • +
            • vignette — Strong-lens vignette figures and mosaic, archived — registered retired 2026-10-29 by human decision: archived vignette figures and DR1 material, non-git; no RAL tree

            No ledger

            @@ -702,7 +706,7 @@

            No ledger

            - +
            cowls_dianaretiredretired 2026-10-14: human moved the tree to z_vault 2026-10-14
            subhalo_simulationsretiredretired 2026-10-14: superseded by subhalo_validation; tree moved to z_vault 2026-10-14
            pj011646retiredretired 2026-10-02: Human requested both PJ011646 and aris_PJ011646 archived intact to z_vault on laptop and RAL; superseded by pj011646_final
            concrretiredretired 2026-10-14: human moved the tree to z_vault 2026-10-14
            concrretiredretired 2026-10-14: human moved the tree to z_vault 2026-10-14; the tree is Science/complete/concr_cosmology (local_path repointed 2026-10-29)