diff --git a/dashboard.html b/dashboard.html index abe27ec..3679ea0 100644 --- a/dashboard.html +++ b/dashboard.html @@ -4,7 +4,7 @@ PyAutoCortex Dashboard - + -

PyAutoCortexScience

Question. Run. Rule.

Explore science with your Cortex

Read the prompt
Read the prompt
Last updated 2026-10-18 08:17:23 UTC
↻ Update

- +- Open cowls: https://github.com/PyAutoLabs/cowls +- Open cowls_ansofi: https://github.com/PyAutoLabs/cowls_ansofi
Last updated 2026-10-19 19:05:59 UTC
↻ Update

+

-

Summary

20 Running · 49 Open
+

Summary

16 Running · 47 Open
@@ -347,12 +348,13 @@ - + +
ProjectRunningOpenLast update
subhalo_validation052026-09-07
ep_toy_gaussian002026-10-14
slope_hierarchy_scale002026-10-14
ic50_workspace002026-10-14
autolens_inference422026-10-17
autolens_inference002026-10-19
autofit_inference002026-10-16
euclid_detailed002026-10-15
cowls002026-10-15
cowls_ansofi002026-10-19
-

Projects

11
+

Projects

12

subhalo_validation

Do the SLaM subhalo pipelines false-detect on lenses with no subhalo0 running · 5 open · updated 2026-09-07
@@ -610,7 +612,7 @@

Runs (0)

-

autolens_inference

Do the PyAutoLens inference backends agree on the HST SLaM posterior4 running · 2 open · updated 2026-10-17 +

autolens_inference

Do the PyAutoLens inference backends agree on the HST SLaM posteriorno runs recorded · updated 2026-10-19
Registered:
not recorded
@@ -618,28 +620,18 @@

Runs (0)

RAL:
/mnt/ral/jnightin/inference/autolens_inference

Current position

-

Clean slate on pin 2026.10.17.1, campaign hst-imaging-mass-samplers, state read 2026-10-17T19:40Z. SLaM base arrays (seeds 0-1, --lens-light both): 412535 (jax_gpu dense) COMPLETED, rows and prepared problems on RAL, not yet pulled; 412537 (jax_gpu sparse), 412539 (jax_cpu dense) and 412541 (jax_cpu sparse) RUNNING since 16:46Z; 412563 (numba_cpu dense) and 412565 (numba_cpu sparse) on ral, 8 CPUs, submitted 18:58Z, waiting in-job on the shared source_lp[1] stage of hpc_a100_jax_cpu_dense_fp64 (rows mark it shared and exclude its wall from the numba chain wall). GPU sampler wave 1 (412543) attempt 1 failed on every arm in two classes: a row-writer TypeError on tuple-keyed parameter names after the nautilus and emcee fits completed, and A100 out-of-memory for NUTS (385.32GiB) and SMC (48.45GiB); pending arms 18-23 cancelled at 19:40Z, arms 9, 13, 17 still running then. Next: pull the A100 dense rows and read them; wave 1 attempt 2 (nautilus and emcee) after the writer fix in the filed Mind task, NUTS and SMC deferred with the OOM evidence; then the CPU wave once the CPU dense base rows land. No scientific acceptance implied.

+

Clean slate on pin 2026.10.17.1, campaign hst-imaging-mass-samplers, state read 2026-10-19 15:49:05Z. Nothing is queued: on the human's decision, 412541 (jax_cpu sparse) and 412565 (numba_cpu sparse) were cancelled ~15:51Z with partial stages left on disk (no CPU fixed-light row exists). 412535 and 412537 (A100) and 412539 (jax_cpu dense) completed; numba_cpu dense dropped. The project is on the flat layout (autolens_inference#36, c3ca4de; output/autolens/..., rows at results/slam/imaging/hst/<variant>/<config>/), and the wave-1 attempt-1 material is set aside. Next, each needing the human's go: confirm keeping prepared/; a Mind bug task and laptop repro for the jax_gpu sparse fixed-light mass_total[1] divergence; rerun the cancelled sparse remainder (jax_cpu sparse seed 1 from source_pix[2], the lightfixed mass_total[1] legs); GPU wave 1 attempt 2 (nautilus+emcee); commit the clean-slate rows in a project PR. No scientific acceptance implied.

-

Runs (6)

- - - - - - - - - - -
JobStatePartitionDateDescription
412537runninggpu2026-10-17
hst-imaging-mass-samplers clean slate: hpc_a100_jax_gpu_sparse_fp64 (submit_hst_cleanslate_jax_gpu_sparse), --lens-light both, seeds 0-1, pin 2026.10.17.1 (PYA…show less

hst-imaging-mass-samplers clean slate: hpc_a100_jax_gpu_sparse_fp64 (submit_hst_cleanslate_jax_gpu_sparse), --lens-light both, seeds 0-1, pin 2026.10.17.1 (PYAUTO_LIB_BASE=/mnt/ral/jnightin/inference/pins/2026.10.17.1), output root output/cleanslate_2026_10_17, results tag cleanslate_2026_10_17, prepared export output/prepared/cleanslate_2026_10_17/hpc_a100_jax_gpu_sparse_fp64/seed_<n>/<variant>/;
human-authorised 2026-10-17

412539runningral2026-10-17
hst-imaging-mass-samplers clean slate: hpc_a100_jax_cpu_dense_fp64 (submit_hst_cleanslate_jax_cpu_dense), --lens-light both, seeds 0-1, pin 2026.10.17.1 (PYAUT…show less

hst-imaging-mass-samplers clean slate: hpc_a100_jax_cpu_dense_fp64 (submit_hst_cleanslate_jax_cpu_dense), --lens-light both, seeds 0-1, pin 2026.10.17.1 (PYAUTO_LIB_BASE=/mnt/ral/jnightin/inference/pins/2026.10.17.1), output root output/cleanslate_2026_10_17, results tag cleanslate_2026_10_17, prepared export output/prepared/cleanslate_2026_10_17/hpc_a100_jax_cpu_dense_fp64/seed_<n>/<variant>/;
human-authorised 2026-10-17

412541runningral2026-10-17
hst-imaging-mass-samplers clean slate: hpc_a100_jax_cpu_sparse_fp64 (submit_hst_cleanslate_jax_cpu_sparse), --lens-light both, seeds 0-1, pin 2026.10.17.1 (PYA…show less

hst-imaging-mass-samplers clean slate: hpc_a100_jax_cpu_sparse_fp64 (submit_hst_cleanslate_jax_cpu_sparse), --lens-light both, seeds 0-1, pin 2026.10.17.1 (PYAUTO_LIB_BASE=/mnt/ral/jnightin/inference/pins/2026.10.17.1), output root output/cleanslate_2026_10_17, results tag cleanslate_2026_10_17, prepared export output/prepared/cleanslate_2026_10_17/hpc_a100_jax_cpu_sparse_fp64/seed_<n>/<variant>/;
human-authorised 2026-10-17

412543runninggpu2026-10-17
hst-imaging-mass-samplers wave 1 (cleanslate_2026_10_17_wave1, submit_hst_cleanslate_mass_samplers_jax_gpu_dense): 24-arm slice nautilus/emcee/nuts/smc x sampl…show less

hst-imaging-mass-samplers wave 1 (cleanslate_2026_10_17_wave1, submit_hst_cleanslate_mass_samplers_jax_gpu_dense): 24-arm slice nautilus/emcee/nuts/smc x sampler seeds 0-2 x both light modes on the base-seed-0 prepared problems of hpc_a100_jax_gpu_dense_fp64, array 0-23%4, 6 h containment, --dependency afterok:412535_0, pin 2026.10.17.1;
human-authorised 2026-10-17

412563openral2026-10-17
hst-imaging-mass-samplers clean slate: numba_cpu dense SLaM base (48gb, 8 CPUs), --lens-light both, seeds 0-1, from autolens_inference main b91ce9c (PR #32), p…show less

hst-imaging-mass-samplers clean slate: numba_cpu dense SLaM base (48gb, 8 CPUs), --lens-light both, seeds 0-1, from autolens_inference main b91ce9c (PR #32), pin 2026.10.17.1, output root output/cleanslate_2026_10_17, results tag cleanslate_2026_10_17, prepared export;
--stages-from hpc_a100_jax_cpu_dense_fp64:source_lp shares source_lp[1] from the jax_cpu dense chain with an in-job wait (hpc/wait_for_stage.sh);
rows mark that stage shared and exclude its wall from the numba chain wall;
submitted 2026-10-17T18:58Z by the architect;
human decisions 2026-10-17 (8 CPUs, shared stage, submit now)

412565openral2026-10-17
hst-imaging-mass-samplers clean slate: numba_cpu sparse SLaM base (64gb, 8 CPUs), --lens-light both, seeds 0-1, from autolens_inference main b91ce9c (PR #32),…show less

hst-imaging-mass-samplers clean slate: numba_cpu sparse SLaM base (64gb, 8 CPUs), --lens-light both, seeds 0-1, from autolens_inference main b91ce9c (PR #32), pin 2026.10.17.1, output root output/cleanslate_2026_10_17, results tag cleanslate_2026_10_17, prepared export;
--stages-from hpc_a100_jax_cpu_dense_fp64:source_lp shares source_lp[1] from the jax_cpu dense chain with an in-job wait (hpc/wait_for_stage.sh);
rows mark that stage shared and exclude its wall from the numba chain wall;
submitted 2026-10-17T18:58Z by the architect;
human decisions 2026-10-17 (8 CPUs, shared stage, submit now)

-
Recent log — 5 of 35 entries +

Runs (0)

+

nothing on the cluster

+
Recent log — 5 of 45 entries
    -
  • 2026-10-17 — note —
    412543 wave 1 attempt 1 (read 19:40Z): every arm fails. NUTS arms 2, 6, 10, 14 FAILED with JaxRuntimeError RESOURCE_EXHAUSTED, out of memory allocating 385.32GiB [jit_log_l]; SMC arms 3, 7, 11, 15 FAILED allocating 48.45GiB [jit_scan]; the…show less

    412543 wave 1 attempt 1 (read 19:40Z): every arm fails. NUTS arms 2, 6, 10, 14 FAILED with JaxRuntimeError RESOURCE_EXHAUSTED, out of memory allocating 385.32GiB [jit_log_l]; SMC arms 3, 7, 11, 15 FAILED allocating 48.45GiB [jit_scan]; their failed-status rows were written. Nautilus and emcee arms 0, 1, 4, 5, 8, 12, 16 completed their mass_total[1] fits (nautilus arm 0 about 13 min, emcee arm 1 about 37 min of job time) and then FAILED in the row writer, scripts/misc/experiments/run.py line 350, TypeError: keys must be str, int, float, bool or None, not tuple (tuple-keyed parameter names from the pinned PyAutoFit); no row written, output trees exist. The architect cancelled pending arms 18-23 at 19:40Z; arms 9, 13, 17 were still running. Fix task filed in Mind (draft/bug/autolens_inference/wave1_row_writer_tuple_keys_and_gpu_oom_reslice.md): fix the writer, re-run nautilus and emcee as attempt 2, defer NUTS and SMC with the OOM evidence.

    • -
  • 2026-10-17 — run —
    412565 submitted: hst-imaging-mass-samplers clean slate: numba_cpu sparse SLaM base (64gb, 8 CPUs), --lens-light both, seeds 0-1, from autolens_inference main b91ce9c (PR #32), pin 2026.10.17.1, output root output/cleanslate_2026_10_17, re…show less

    412565 submitted: hst-imaging-mass-samplers clean slate: numba_cpu sparse SLaM base (64gb, 8 CPUs), --lens-light both, seeds 0-1, from autolens_inference main b91ce9c (PR #32), pin 2026.10.17.1, output root output/cleanslate_2026_10_17, results tag cleanslate_2026_10_17, prepared export; --stages-from hpc_a100_jax_cpu_dense_fp64:source_lp shares source_lp[1] from the jax_cpu dense chain with an in-job wait (hpc/wait_for_stage.sh); rows mark that stage shared and exclude its wall from the numba chain wall; submitted 2026-10-17T18:58Z by the architect; human decisions 2026-10-17 (8 CPUs, shared stage, submit now)

    • -
  • 2026-10-17 — run —
    412563 submitted: hst-imaging-mass-samplers clean slate: numba_cpu dense SLaM base (48gb, 8 CPUs), --lens-light both, seeds 0-1, from autolens_inference main b91ce9c (PR #32), pin 2026.10.17.1, output root output/cleanslate_2026_10_17, res…show less

    412563 submitted: hst-imaging-mass-samplers clean slate: numba_cpu dense SLaM base (48gb, 8 CPUs), --lens-light both, seeds 0-1, from autolens_inference main b91ce9c (PR #32), pin 2026.10.17.1, output root output/cleanslate_2026_10_17, results tag cleanslate_2026_10_17, prepared export; --stages-from hpc_a100_jax_cpu_dense_fp64:source_lp shares source_lp[1] from the jax_cpu dense chain with an in-job wait (hpc/wait_for_stage.sh); rows mark that stage shared and exclude its wall from the numba chain wall; submitted 2026-10-17T18:58Z by the architect; human decisions 2026-10-17 (8 CPUs, shared stage, submit now)

    • -
  • 2026-10-17 — run —
    412535 finished — wall 1:01: hst-imaging-mass-samplers clean slate: hpc_a100_jax_gpu_dense_fp64 (submit_hst_cleanslate_jax_gpu_dense), --lens-light both, seeds 0-1, pin 2026.10.17.1 (PYAUTO_LIB_BASE=/mnt/ral/jnightin/inference/pins/2026.10…show less

    412535 finished — wall 1:01: hst-imaging-mass-samplers clean slate: hpc_a100_jax_gpu_dense_fp64 (submit_hst_cleanslate_jax_gpu_dense), --lens-light both, seeds 0-1, pin 2026.10.17.1 (PYAUTO_LIB_BASE=/mnt/ral/jnightin/inference/pins/2026.10.17.1), output root output/cleanslate_2026_10_17, results tag cleanslate_2026_10_17, prepared export output/prepared/cleanslate_2026_10_17/hpc_a100_jax_gpu_dense_fp64/seed_<n>/<variant>/; human-authorised 2026-10-17 — COMPLETED 0:0 both seeds on gpu, ended ~17:46Z (elapsed 01:00:33 seed 0 / 00:58:35 seed 1); rows on RAL under results/slam/imaging/hst/slam_base/ and slam_base_lightfixed/hpc_a100_jax_gpu_dense_fp64__cleanslate_2026_10_17/stages_seed{0,1}.json; prepared problems exported to output/prepared/cleanslate_2026_10_17/hpc_a100_jax_gpu_dense_fp64/seed_0/{slam_base,slam_base_lightfixed}/; rows not yet pulled or committed, no walls read yet

    • -
  • 2026-10-17 — run —
    412543 submitted: hst-imaging-mass-samplers wave 1 (cleanslate_2026_10_17_wave1, submit_hst_cleanslate_mass_samplers_jax_gpu_dense): 24-arm slice nautilus/emcee/nuts/smc x sampler seeds 0-2 x both light modes on the base-seed-0 prepared pr…show less

    412543 submitted: hst-imaging-mass-samplers wave 1 (cleanslate_2026_10_17_wave1, submit_hst_cleanslate_mass_samplers_jax_gpu_dense): 24-arm slice nautilus/emcee/nuts/smc x sampler seeds 0-2 x both light modes on the base-seed-0 prepared problems of hpc_a100_jax_gpu_dense_fp64, array 0-23%4, 6 h containment, --dependency afterok:412535_0, pin 2026.10.17.1; human-authorised 2026-10-17

    • +
  • 2026-10-19 — note —
    412539 (jax_cpu dense) elapsed from sacct, read 2026-10-19 ~17:25Z by the session architect: 412539_0 1-05:59:53 (ended 2026-10-18T23:46:01), 412539_1 1-02:57:03 (ended 2026-10-18T20:43:11). This supplements the 2026-10-19 finished entry,…show less

    412539 (jax_cpu dense) elapsed from sacct, read 2026-10-19 ~17:25Z by the session architect: 412539_0 1-05:59:53 (ended 2026-10-18T23:46:01), 412539_1 1-02:57:03 (ended 2026-10-18T20:43:11). This supplements the 2026-10-19 finished entry, which carries no wall. Its rows have still not been read.

    • +
  • 2026-10-19 — run —
    412543 failed — wall 0:47: hst-imaging-mass-samplers wave 1 (cleanslate_2026_10_17_wave1, submit_hst_cleanslate_mass_samplers_jax_gpu_dense): 24-arm slice nautilus/emcee/nuts/smc x sampler seeds 0-2 x both light modes on the base-seed-0 pr…show less

    412543 failed — wall 0:47: hst-imaging-mass-samplers wave 1 (cleanslate_2026_10_17_wave1, submit_hst_cleanslate_mass_samplers_jax_gpu_dense): 24-arm slice nautilus/emcee/nuts/smc x sampler seeds 0-2 x both light modes on the base-seed-0 prepared problems of hpc_a100_jax_gpu_dense_fp64, array 0-23%4, 6 h containment, --dependency afterok:412535_0, pin 2026.10.17.1; human-authorised 2026-10-17 — sacct read 2026-10-19 ~17:25Z by the session architect: tasks _0.._17 all FAILED on 2026-10-17 (ends 18:51:38-20:32:26Z, elapsed 00:04:15-00:47:10; wall given is the longest task); _18 CANCELLED after 00:04:00; _[19-23] CANCELLED unstarted 2026-10-17T20:01:03Z. Failure classes as in the 2026-10-17 note (NUTS/SMC A100 OOM; nautilus/emcee row-writer TypeError). Its output and the 8 failed NUTS/SMC rows were moved aside on 2026-10-19 to aside/autolens_inference_cleanslate_wave1_attempt1/ (RAL and laptop)

    • +
  • 2026-10-19 — note —
    HUMAN DECISION 2026-10-19, verbatim: "I want us to check the on going inference campaign on RAL, but I am ready for us to update folder structures on RAL and locally and clean things up so cancel on going jobs and we'll work on the clean u…show less

    HUMAN DECISION 2026-10-19, verbatim: "I want us to check the on going inference campaign on RAL, but I am ready for us to update folder structures on RAL and locally and clean things up so cancel on going jobs and we'll work on the clean up". Scheduler read 15:49:05Z: only 412541 and 412565 running; both scancelled ~15:51Z, queue empty; 412539 had COMPLETED. On "yes, go", autolens_inference#36 (flatten, closes #35) merged 15:53:33Z as c3ca4de; the RAL checkout fast-forwarded b91ce9c -> c3ca4de and the laptop to c3ca4de, after a full hpc/sync pull. flatten_cleanslate.py --apply: RAL 28 moves, 55 rewrites; laptop mirror 28 moves, 38 rewrites; both EXIT 0. New layout: output/autolens/imaging/hst/slam/<variant>/<config>/seed_<n>/<stage>/, output/autolens/prepared/<config>/seed_<n>/<variant>/, rows at results/slam/imaging/hst/<variant>/<config>/ (no __cleanslate_2026_10_17 suffix). RAL and laptop each hold 1974 files with identical paths and sizes; hpc/sync status dry run transfers 0. Human, verbatim: "check some older chats, think we dont need prepared, experiments and experiment_sources to exist anymore? Can we archive them somewhere separate to makr things easier to follow. Then do the apply". The wave-1 attempt-1 material (output/experiments 341M, output/experiment_sources 76M, results/searches/experiments with 8 failed NUTS/SMC rows) moved to /mnt/ral/jnightin/inference/aside/autolens_inference_cleanslate_wave1_attempt1/ on RAL and /mnt/c/Users/Jammy/Science/inference/aside/autolens_inference_cleanslate_wave1_attempt1/ on the laptop. prepared/ KEPT as the agent's proposal, not yet confirmed by the human: wave.py needs its manifests, and so does the sparse fixed-light repro. Partial stage dirs of the cancelled legs remain; they are not failures. Execution completion only. No compute submitted.

    • +
  • 2026-10-19 — run —
    412565 failed — wall 44:52: hst-imaging-mass-samplers clean slate: numba_cpu sparse SLaM base (64gb, 8 CPUs), --lens-light both, seeds 0-1, from autolens_inference main b91ce9c (PR #32), pin 2026.10.17.1, output root output/cleanslate_2026…show less

    412565 failed — wall 44:52: hst-imaging-mass-samplers clean slate: numba_cpu sparse SLaM base (64gb, 8 CPUs), --lens-light both, seeds 0-1, from autolens_inference main b91ce9c (PR #32), pin 2026.10.17.1, output root output/cleanslate_2026_10_17, results tag cleanslate_2026_10_17, prepared export; --stages-from hpc_a100_jax_cpu_dense_fp64:source_lp shares source_lp[1] from the jax_cpu dense chain with an in-job wait (hpc/wait_for_stage.sh); rows mark that stage shared and exclude its wall from the numba chain wall; submitted 2026-10-17T18:58Z by the architect; human decisions 2026-10-17 (8 CPUs, shared stage, submit now) — CANCELLED by scancel ~15:51Z on the human's decision, not a run failure (sacct CANCELLED, elapsed 1-20:52:11). Complete at cancel (.completed markers): seeds 0 and 1 slam_base light[1], source_pix[1], source_pix[2], mass_total[1] (source_lp[1] shared from jax_cpu dense via --stages-from). Incomplete: slam_base_lightfixed mass_total[1] on both seeds (running). Partial stage dirs left on disk under output/autolens/imaging/hst/slam/<variant>/hpc_a100_numba_cpu_sparse_fp64/seed_<n>/ (flat layout, autolens_inference#36)

    • +
  • 2026-10-19 — run —
    412541 failed — wall 47:04: hst-imaging-mass-samplers clean slate: hpc_a100_jax_cpu_sparse_fp64 (submit_hst_cleanslate_jax_cpu_sparse), --lens-light both, seeds 0-1, pin 2026.10.17.1 (PYAUTO_LIB_BASE=/mnt/ral/jnightin/inference/pins/2026.1…show less

    412541 failed — wall 47:04: hst-imaging-mass-samplers clean slate: hpc_a100_jax_cpu_sparse_fp64 (submit_hst_cleanslate_jax_cpu_sparse), --lens-light both, seeds 0-1, pin 2026.10.17.1 (PYAUTO_LIB_BASE=/mnt/ral/jnightin/inference/pins/2026.10.17.1), output root output/cleanslate_2026_10_17, results tag cleanslate_2026_10_17, prepared export output/prepared/cleanslate_2026_10_17/hpc_a100_jax_cpu_sparse_fp64/seed_<n>/<variant>/; human-authorised 2026-10-17 — CANCELLED by scancel ~15:51Z on the human's decision, not a run failure (sacct CANCELLED, elapsed 1-23:04:43). Complete at cancel (.completed markers): seed 0 slam_base light[1], source_lp[1], source_pix[1], source_pix[2], mass_total[1]; seed 1 source_lp[1], source_pix[1]. Incomplete: seed 0 slam_base_lightfixed mass_total[1] (running), seed 1 source_pix[2] (running) onward and its lightfixed leg. Partial stage dirs left on disk under output/autolens/imaging/hst/slam/<variant>/hpc_a100_jax_cpu_sparse_fp64/seed_<n>/ (flat layout, autolens_inference#36)

full log

active · both · autolens_inference#4

@@ -719,6 +711,29 @@

Runs (0)

+
+

cowls_ansofi

autolens_inference Delaunay-1250 numba CPU SLaM on COSJ100024+015334, F444W onlyno runs recorded · updated 2026-10-19 +
+
+
Registered:
2026-10-19
+
Local:
/mnt/c/Users/Jammy/Science/active/cowls_ansofi
+
RAL:
/mnt/ral/jnightin/active/cowls_ansofi
+
+

Current position

+

Born 2026-10-19. Repo PyAutoLabs/cowls_ansofi (private), M25 COSJ100024+015334 F444W dataset and scripts/slam.py ready; test mode passes all five searches (144 s, numba sparse on every stage). Next: hpc/sync push-submit cpu submit_slam (8 CPU, 64 GB, 5 days, partition ral).

+ +
+

Runs (0)

+

nothing on the cluster

+
Recent log — 1 of 1 entry +
    +
  • 2026-10-19 — note — ledger opened
    • +
+

full log

+

active · ral · no issue yet

+
+
+

euclid

The original Euclid DR1 grade-AB catalogue project, superseded by euclid_dr1dormant · no runs recorded · updated 2026-10-15
diff --git a/dashboard.md b/dashboard.md index 2e79ed8..b201679 100644 --- a/dashboard.md +++ b/dashboard.md @@ -1,18 +1,18 @@ # PyAutoCortex Dashboard - + This is the markdown version of the [PyAutoCortex Dashboard](https://pyautolabs.github.io/PyAutoCortex/), which puts the check-in and each project's resume prompt on your clipboard with a single tap of 📋. One ledger per science project: what is on the cluster right now, what the human said they would do next, and the last few things they wrote down. Nothing here is inferred from a result — the board hands you the check-in and the resume prompt; what a run meant is yours to log. -> **Last updated 2026-10-18.** +> **Last updated 2026-10-19.** | Where | Count | |-------|------:| -| [Running](#projects) | 20 | -| [Open](#projects) | 49 | -| [Projects](#projects) | 11 | +| [Running](#projects) | 16 | +| [Open](#projects) | 47 | +| [Projects](#projects) | 12 |
📋 check in since last time @@ -49,10 +49,11 @@ Last check-in: 2026-10-17T11:18Z. | ep_toy_gaussian | 0 | 0 | 2026-10-14 | | slope_hierarchy_scale | 0 | 0 | 2026-10-14 | | ic50_workspace | 0 | 0 | 2026-10-14 | -| autolens_inference | 4 | 2 | 2026-10-17 | +| autolens_inference | 0 | 0 | 2026-10-19 | | autofit_inference | 0 | 0 | 2026-10-16 | | euclid_detailed | 0 | 0 | 2026-10-15 | | cowls | 0 | 0 | 2026-10-15 | +| cowls_ansofi | 0 | 0 | 2026-10-19 | ## Projects @@ -456,7 +457,7 @@ active · ral · [PyAutoCortex#36](https://github.com/PyAutoLabs/PyAutoCortex/is
-
autolens_inference — Do the PyAutoLens inference backends agree on the HST SLaM posterior · 4 running · 2 open · updated 2026-10-17 +
autolens_inference — Do the PyAutoLens inference backends agree on the HST SLaM posterior · no runs recorded · updated 2026-10-19 **Registered:** not recorded @@ -466,7 +467,7 @@ active · ral · [PyAutoCortex#36](https://github.com/PyAutoLabs/PyAutoCortex/is **Current position** -Clean slate on pin 2026.10.17.1, campaign hst-imaging-mass-samplers, state read 2026-10-17T19:40Z. SLaM base arrays (seeds 0-1, --lens-light both): 412535 (jax_gpu dense) COMPLETED, rows and prepared problems on RAL, not yet pulled; 412537 (jax_gpu sparse), 412539 (jax_cpu dense) and 412541 (jax_cpu sparse) RUNNING since 16:46Z; 412563 (numba_cpu dense) and 412565 (numba_cpu sparse) on ral, 8 CPUs, submitted 18:58Z, waiting in-job on the shared source_lp[1] stage of hpc_a100_jax_cpu_dense_fp64 (rows mark it shared and exclude its wall from the numba chain wall). GPU sampler wave 1 (412543) attempt 1 failed on every arm in two classes: a row-writer TypeError on tuple-keyed parameter names after the nautilus and emcee fits completed, and A100 out-of-memory for NUTS (385.32GiB) and SMC (48.45GiB); pending arms 18-23 cancelled at 19:40Z, arms 9, 13, 17 still running then. Next: pull the A100 dense rows and read them; wave 1 attempt 2 (nautilus and emcee) after the writer fix in the filed Mind task, NUTS and SMC deferred with the OOM evidence; then the CPU wave once the CPU dense base rows land. No scientific acceptance implied. +Clean slate on pin 2026.10.17.1, campaign hst-imaging-mass-samplers, state read 2026-10-19 15:49:05Z. Nothing is queued: on the human's decision, 412541 (jax_cpu sparse) and 412565 (numba_cpu sparse) were cancelled ~15:51Z with partial stages left on disk (no CPU fixed-light row exists). 412535 and 412537 (A100) and 412539 (jax_cpu dense) completed; numba_cpu dense dropped. The project is on the flat layout (autolens_inference#36, c3ca4de; output/autolens/..., rows at results/slam/imaging/hst/<variant>/<config>/), and the wave-1 attempt-1 material is set aside. Next, each needing the human's go: confirm keeping prepared/; a Mind bug task and laptop repro for the jax_gpu sparse fixed-light mass_total[1] divergence; rerun the cancelled sparse remainder (jax_cpu sparse seed 1 from source_pix[2], the lightfixed mass_total[1] legs); GPU wave 1 attempt 2 (nautilus+emcee); commit the clean-slate rows in a project PR. No scientific acceptance implied.
📋 @@ -476,35 +477,17 @@ Use the cortex skill. — resume autolens_inference: read PyAutoCortex projects/
-**Runs** (6) - -| Job | State | Partition | Date | Description | -|---|---|---|---|---| -| `412537` | running | gpu | 2026-10-17 | hst-imaging-mass-samplers clean slate: hpc_a100_jax_gpu_sparse_fp64 (submit_hst_cleanslate_jax_gpu_sparse), --lens-light both, seeds 0-1, pin 2026.10.17.1 (PYA… | -| `412539` | running | ral | 2026-10-17 | hst-imaging-mass-samplers clean slate: hpc_a100_jax_cpu_dense_fp64 (submit_hst_cleanslate_jax_cpu_dense), --lens-light both, seeds 0-1, pin 2026.10.17.1 (PYAUT… | -| `412541` | running | ral | 2026-10-17 | hst-imaging-mass-samplers clean slate: hpc_a100_jax_cpu_sparse_fp64 (submit_hst_cleanslate_jax_cpu_sparse), --lens-light both, seeds 0-1, pin 2026.10.17.1 (PYA… | -| `412543` | running | gpu | 2026-10-17 | hst-imaging-mass-samplers wave 1 (cleanslate_2026_10_17_wave1, submit_hst_cleanslate_mass_samplers_jax_gpu_dense): 24-arm slice nautilus/emcee/nuts/smc x sampl… | -| `412563` | open | ral | 2026-10-17 | hst-imaging-mass-samplers clean slate: numba_cpu dense SLaM base (48gb, 8 CPUs), --lens-light both, seeds 0-1, from autolens_inference main b91ce9c (PR #32), p… | -| `412565` | open | ral | 2026-10-17 | hst-imaging-mass-samplers clean slate: numba_cpu sparse SLaM base (64gb, 8 CPUs), --lens-light both, seeds 0-1, from autolens_inference main b91ce9c (PR #32),… | - -
Full descriptions (6 shortened above) - -- `412537` — hst-imaging-mass-samplers clean slate: hpc_a100_jax_gpu_sparse_fp64 (submit_hst_cleanslate_jax_gpu_sparse), --lens-light both, seeds 0-1, pin 2026.10.17.1 (PYAUTO_LIB_BASE=/mnt/ral/jnightin/inference/pins/2026.10.17.1), output root output/cleanslate_2026_10_17, results tag cleanslate_2026_10_17, prepared export output/prepared/cleanslate_2026_10_17/hpc_a100_jax_gpu_sparse_fp64/seed_<n>/<variant>/; human-authorised 2026-10-17 -- `412539` — hst-imaging-mass-samplers clean slate: hpc_a100_jax_cpu_dense_fp64 (submit_hst_cleanslate_jax_cpu_dense), --lens-light both, seeds 0-1, pin 2026.10.17.1 (PYAUTO_LIB_BASE=/mnt/ral/jnightin/inference/pins/2026.10.17.1), output root output/cleanslate_2026_10_17, results tag cleanslate_2026_10_17, prepared export output/prepared/cleanslate_2026_10_17/hpc_a100_jax_cpu_dense_fp64/seed_<n>/<variant>/; human-authorised 2026-10-17 -- `412541` — hst-imaging-mass-samplers clean slate: hpc_a100_jax_cpu_sparse_fp64 (submit_hst_cleanslate_jax_cpu_sparse), --lens-light both, seeds 0-1, pin 2026.10.17.1 (PYAUTO_LIB_BASE=/mnt/ral/jnightin/inference/pins/2026.10.17.1), output root output/cleanslate_2026_10_17, results tag cleanslate_2026_10_17, prepared export output/prepared/cleanslate_2026_10_17/hpc_a100_jax_cpu_sparse_fp64/seed_<n>/<variant>/; human-authorised 2026-10-17 -- `412543` — hst-imaging-mass-samplers wave 1 (cleanslate_2026_10_17_wave1, submit_hst_cleanslate_mass_samplers_jax_gpu_dense): 24-arm slice nautilus/emcee/nuts/smc x sampler seeds 0-2 x both light modes on the base-seed-0 prepared problems of hpc_a100_jax_gpu_dense_fp64, array 0-23%4, 6 h containment, --dependency afterok:412535_0, pin 2026.10.17.1; human-authorised 2026-10-17 -- `412563` — hst-imaging-mass-samplers clean slate: numba_cpu dense SLaM base (48gb, 8 CPUs), --lens-light both, seeds 0-1, from autolens_inference main b91ce9c (PR #32), pin 2026.10.17.1, output root output/cleanslate_2026_10_17, results tag cleanslate_2026_10_17, prepared export; --stages-from hpc_a100_jax_cpu_dense_fp64:source_lp shares source_lp[1] from the jax_cpu dense chain with an in-job wait (hpc/wait_for_stage.sh); rows mark that stage shared and exclude its wall from the numba chain wall; submitted 2026-10-17T18:58Z by the architect; human decisions 2026-10-17 (8 CPUs, shared stage, submit now) -- `412565` — hst-imaging-mass-samplers clean slate: numba_cpu sparse SLaM base (64gb, 8 CPUs), --lens-light both, seeds 0-1, from autolens_inference main b91ce9c (PR #32), pin 2026.10.17.1, output root output/cleanslate_2026_10_17, results tag cleanslate_2026_10_17, prepared export; --stages-from hpc_a100_jax_cpu_dense_fp64:source_lp shares source_lp[1] from the jax_cpu dense chain with an in-job wait (hpc/wait_for_stage.sh); rows mark that stage shared and exclude its wall from the numba chain wall; submitted 2026-10-17T18:58Z by the architect; human decisions 2026-10-17 (8 CPUs, shared stage, submit now) +**Runs** (0) -
+_nothing on the cluster_ -
Recent log — 5 of 35 entries +
Recent log — 5 of 45 entries -- 2026-10-17 — *note* — 412543 wave 1 attempt 1 (read 19:40Z): every arm fails. NUTS arms 2, 6, 10, 14 FAILED with JaxRuntimeError RESOURCE_EXHAUSTED, out of memory allocating 385.32GiB [jit_log_l]; SMC arms 3, 7, 11, 15 FAILED allocating 48.45GiB [jit_scan]; their failed-status rows were written. Nautilus and emcee arms 0, 1, 4, 5, 8, 12, 16 completed their mass_total[1] fits (nautilus arm 0 about 13 min, emcee arm 1 about 37 min of job time) and then FAILED in the row writer, scripts/misc/experiments/run.py line 350, TypeError: keys must be str, int, float, bool or None, not tuple (tuple-keyed parameter names from the pinned PyAutoFit); no row written, output trees exist. The architect cancelled pending arms 18-23 at 19:40Z; arms 9, 13, 17 were still running. Fix task filed in Mind (draft/bug/autolens_inference/wave1_row_writer_tuple_keys_and_gpu_oom_reslice.md): fix the writer, re-run nautilus and emcee as attempt 2, defer NUTS and SMC with the OOM evidence. -- 2026-10-17 — *run* — 412565 submitted: hst-imaging-mass-samplers clean slate: numba_cpu sparse SLaM base (64gb, 8 CPUs), --lens-light both, seeds 0-1, from autolens_inference main b91ce9c (PR #32), pin 2026.10.17.1, output root output/cleanslate_2026_10_17, results tag cleanslate_2026_10_17, prepared export; --stages-from hpc_a100_jax_cpu_dense_fp64:source_lp shares source_lp[1] from the jax_cpu dense chain with an in-job wait (hpc/wait_for_stage.sh); rows mark that stage shared and exclude its wall from the numba chain wall; submitted 2026-10-17T18:58Z by the architect; human decisions 2026-10-17 (8 CPUs, shared stage, submit now) -- 2026-10-17 — *run* — 412563 submitted: hst-imaging-mass-samplers clean slate: numba_cpu dense SLaM base (48gb, 8 CPUs), --lens-light both, seeds 0-1, from autolens_inference main b91ce9c (PR #32), pin 2026.10.17.1, output root output/cleanslate_2026_10_17, results tag cleanslate_2026_10_17, prepared export; --stages-from hpc_a100_jax_cpu_dense_fp64:source_lp shares source_lp[1] from the jax_cpu dense chain with an in-job wait (hpc/wait_for_stage.sh); rows mark that stage shared and exclude its wall from the numba chain wall; submitted 2026-10-17T18:58Z by the architect; human decisions 2026-10-17 (8 CPUs, shared stage, submit now) -- 2026-10-17 — *run* — 412535 finished — wall 1:01: hst-imaging-mass-samplers clean slate: hpc_a100_jax_gpu_dense_fp64 (submit_hst_cleanslate_jax_gpu_dense), --lens-light both, seeds 0-1, pin 2026.10.17.1 (PYAUTO_LIB_BASE=/mnt/ral/jnightin/inference/pins/2026.10.17.1), output root output/cleanslate_2026_10_17, results tag cleanslate_2026_10_17, prepared export output/prepared/cleanslate_2026_10_17/hpc_a100_jax_gpu_dense_fp64/seed_<n>/<variant>/; human-authorised 2026-10-17 — COMPLETED 0:0 both seeds on gpu, ended ~17:46Z (elapsed 01:00:33 seed 0 / 00:58:35 seed 1); rows on RAL under results/slam/imaging/hst/slam_base/ and slam_base_lightfixed/hpc_a100_jax_gpu_dense_fp64__cleanslate_2026_10_17/stages_seed{0,1}.json; prepared problems exported to output/prepared/cleanslate_2026_10_17/hpc_a100_jax_gpu_dense_fp64/seed_0/{slam_base,slam_base_lightfixed}/; rows not yet pulled or committed, no walls read yet -- 2026-10-17 — *run* — 412543 submitted: hst-imaging-mass-samplers wave 1 (cleanslate_2026_10_17_wave1, submit_hst_cleanslate_mass_samplers_jax_gpu_dense): 24-arm slice nautilus/emcee/nuts/smc x sampler seeds 0-2 x both light modes on the base-seed-0 prepared problems of hpc_a100_jax_gpu_dense_fp64, array 0-23%4, 6 h containment, --dependency afterok:412535_0, pin 2026.10.17.1; human-authorised 2026-10-17 +- 2026-10-19 — *note* — 412539 (jax_cpu dense) elapsed from sacct, read 2026-10-19 ~17:25Z by the session architect: 412539_0 1-05:59:53 (ended 2026-10-18T23:46:01), 412539_1 1-02:57:03 (ended 2026-10-18T20:43:11). This supplements the 2026-10-19 finished entry, which carries no wall. Its rows have still not been read. +- 2026-10-19 — *run* — 412543 failed — wall 0:47: hst-imaging-mass-samplers wave 1 (cleanslate_2026_10_17_wave1, submit_hst_cleanslate_mass_samplers_jax_gpu_dense): 24-arm slice nautilus/emcee/nuts/smc x sampler seeds 0-2 x both light modes on the base-seed-0 prepared problems of hpc_a100_jax_gpu_dense_fp64, array 0-23%4, 6 h containment, --dependency afterok:412535_0, pin 2026.10.17.1; human-authorised 2026-10-17 — sacct read 2026-10-19 ~17:25Z by the session architect: tasks _0.._17 all FAILED on 2026-10-17 (ends 18:51:38-20:32:26Z, elapsed 00:04:15-00:47:10; wall given is the longest task); _18 CANCELLED after 00:04:00; _[19-23] CANCELLED unstarted 2026-10-17T20:01:03Z. Failure classes as in the 2026-10-17 note (NUTS/SMC A100 OOM; nautilus/emcee row-writer TypeError). Its output and the 8 failed NUTS/SMC rows were moved aside on 2026-10-19 to aside/autolens_inference_cleanslate_wave1_attempt1/ (RAL and laptop) +- 2026-10-19 — *note* — HUMAN DECISION 2026-10-19, verbatim: "I want us to check the on going inference campaign on RAL, but I am ready for us to update folder structures on RAL and locally and clean things up so cancel on going jobs and we'll work on the clean up". Scheduler read 15:49:05Z: only 412541 and 412565 running; both scancelled ~15:51Z, queue empty; 412539 had COMPLETED. On "yes, go", autolens_inference#36 (flatten, closes #35) merged 15:53:33Z as c3ca4de; the RAL checkout fast-forwarded b91ce9c -> c3ca4de and the laptop to c3ca4de, after a full hpc/sync pull. flatten_cleanslate.py --apply: RAL 28 moves, 55 rewrites; laptop mirror 28 moves, 38 rewrites; both EXIT 0. New layout: output/autolens/imaging/hst/slam/<variant>/<config>/seed_<n>/<stage>/, output/autolens/prepared/<config>/seed_<n>/<variant>/, rows at results/slam/imaging/hst/<variant>/<config>/ (no __cleanslate_2026_10_17 suffix). RAL and laptop each hold 1974 files with identical paths and sizes; hpc/sync status dry run transfers 0. Human, verbatim: "check some older chats, think we dont need prepared, experiments and experiment_sources to exist anymore? Can we archive them somewhere separate to makr things easier to follow. Then do the apply". The wave-1 attempt-1 material (output/experiments 341M, output/experiment_sources 76M, results/searches/experiments with 8 failed NUTS/SMC rows) moved to /mnt/ral/jnightin/inference/aside/autolens_inference_cleanslate_wave1_attempt1/ on RAL and /mnt/c/Users/Jammy/Science/inference/aside/autolens_inference_cleanslate_wave1_attempt1/ on the laptop. prepared/ KEPT as the agent's proposal, not yet confirmed by the human: wave.py needs its manifests, and so does the sparse fixed-light repro. Partial stage dirs of the cancelled legs remain; they are not failures. Execution completion only. No compute submitted. +- 2026-10-19 — *run* — 412565 failed — wall 44:52: hst-imaging-mass-samplers clean slate: numba_cpu sparse SLaM base (64gb, 8 CPUs), --lens-light both, seeds 0-1, from autolens_inference main b91ce9c (PR #32), pin 2026.10.17.1, output root output/cleanslate_2026_10_17, results tag cleanslate_2026_10_17, prepared export; --stages-from hpc_a100_jax_cpu_dense_fp64:source_lp shares source_lp[1] from the jax_cpu dense chain with an in-job wait (hpc/wait_for_stage.sh); rows mark that stage shared and exclude its wall from the numba chain wall; submitted 2026-10-17T18:58Z by the architect; human decisions 2026-10-17 (8 CPUs, shared stage, submit now) — CANCELLED by scancel ~15:51Z on the human's decision, not a run failure (sacct CANCELLED, elapsed 1-20:52:11). Complete at cancel (.completed markers): seeds 0 and 1 slam_base light[1], source_pix[1], source_pix[2], mass_total[1] (source_lp[1] shared from jax_cpu dense via --stages-from). Incomplete: slam_base_lightfixed mass_total[1] on both seeds (running). Partial stage dirs left on disk under output/autolens/imaging/hst/slam/<variant>/hpc_a100_numba_cpu_sparse_fp64/seed_<n>/ (flat layout, autolens_inference#36) +- 2026-10-19 — *run* — 412541 failed — wall 47:04: hst-imaging-mass-samplers clean slate: hpc_a100_jax_cpu_sparse_fp64 (submit_hst_cleanslate_jax_cpu_sparse), --lens-light both, seeds 0-1, pin 2026.10.17.1 (PYAUTO_LIB_BASE=/mnt/ral/jnightin/inference/pins/2026.10.17.1), output root output/cleanslate_2026_10_17, results tag cleanslate_2026_10_17, prepared export output/prepared/cleanslate_2026_10_17/hpc_a100_jax_cpu_sparse_fp64/seed_<n>/<variant>/; human-authorised 2026-10-17 — CANCELLED by scancel ~15:51Z on the human's decision, not a run failure (sacct CANCELLED, elapsed 1-23:04:43). Complete at cancel (.completed markers): seed 0 slam_base light[1], source_lp[1], source_pix[1], source_pix[2], mass_total[1]; seed 1 source_lp[1], source_pix[1]. Incomplete: seed 0 slam_base_lightfixed mass_total[1] (running), seed 1 source_pix[2] (running) onward and its lightfixed leg. Partial stage dirs left on disk under output/autolens/imaging/hst/slam/<variant>/hpc_a100_jax_cpu_sparse_fp64/seed_<n>/ (flat layout, autolens_inference#36) [full log](https://github.com/PyAutoLabs/PyAutoCortex/blob/main/projects/autolens_inference.md) @@ -626,6 +609,42 @@ active · ral · no issue yet
+
cowls_ansofi — autolens_inference Delaunay-1250 numba CPU SLaM on COSJ100024+015334, F444W only · no runs recorded · updated 2026-10-19 + +**Registered:** 2026-10-19 + +**Local:** `` /mnt/c/Users/Jammy/Science/active/cowls_ansofi `` + +**RAL:** `` /mnt/ral/jnightin/active/cowls_ansofi `` + +**Current position** + +Born 2026-10-19. Repo PyAutoLabs/cowls_ansofi (private), M25 COSJ100024+015334 F444W dataset and scripts/slam.py ready; test mode passes all five searches (144 s, numba sparse on every stage). Next: hpc/sync push-submit cpu submit_slam (8 CPU, 64 GB, 5 days, partition ral). + +
📋 + +``` +Use the cortex skill. — resume cowls_ansofi: read PyAutoCortex projects/cowls_ansofi.md (Now, Runs, Log) and then /mnt/c/Users/Jammy/Science/active/cowls_ansofi/wiki/project/state.md and the assistant autolens_assistant's AGENTS.md; tell me where I left off and what I said I would do next. Submit nothing and log nothing until I say. +``` + +
+ +**Runs** (0) + +_nothing on the cluster_ + +
Recent log — 1 of 1 entry + +- 2026-10-19 — *note* — ledger opened + +[full log](https://github.com/PyAutoLabs/PyAutoCortex/blob/main/projects/cowls_ansofi.md) + +active · ral · no issue yet + +
+ +
+
euclid — The original Euclid DR1 grade-AB catalogue project, superseded by euclid_dr1 · dormant · no runs recorded · updated 2026-10-15 **Registered:** not recorded diff --git a/projects/autolens_inference.md b/projects/autolens_inference.md index e1af752..3a15a0a 100644 --- a/projects/autolens_inference.md +++ b/projects/autolens_inference.md @@ -5,19 +5,23 @@ Issue: autolens_inference#4 ## Now -Clean slate on pin 2026.10.17.1, campaign hst-imaging-mass-samplers, state read 2026-10-17T19:40Z. SLaM base arrays (seeds 0-1, --lens-light both): 412535 (jax_gpu dense) COMPLETED, rows and prepared problems on RAL, not yet pulled; 412537 (jax_gpu sparse), 412539 (jax_cpu dense) and 412541 (jax_cpu sparse) RUNNING since 16:46Z; 412563 (numba_cpu dense) and 412565 (numba_cpu sparse) on ral, 8 CPUs, submitted 18:58Z, waiting in-job on the shared source_lp[1] stage of hpc_a100_jax_cpu_dense_fp64 (rows mark it shared and exclude its wall from the numba chain wall). GPU sampler wave 1 (412543) attempt 1 failed on every arm in two classes: a row-writer TypeError on tuple-keyed parameter names after the nautilus and emcee fits completed, and A100 out-of-memory for NUTS (385.32GiB) and SMC (48.45GiB); pending arms 18-23 cancelled at 19:40Z, arms 9, 13, 17 still running then. Next: pull the A100 dense rows and read them; wave 1 attempt 2 (nautilus and emcee) after the writer fix in the filed Mind task, NUTS and SMC deferred with the OOM evidence; then the CPU wave once the CPU dense base rows land. No scientific acceptance implied. +Clean slate on pin 2026.10.17.1, campaign hst-imaging-mass-samplers, state read 2026-10-19 15:49:05Z. Nothing is queued: on the human's decision, 412541 (jax_cpu sparse) and 412565 (numba_cpu sparse) were cancelled ~15:51Z with partial stages left on disk (no CPU fixed-light row exists). 412535 and 412537 (A100) and 412539 (jax_cpu dense) completed; numba_cpu dense dropped. The project is on the flat layout (autolens_inference#36, c3ca4de; output/autolens/..., rows at results/slam/imaging/hst///), and the wave-1 attempt-1 material is set aside. Next, each needing the human's go: confirm keeping prepared/; a Mind bug task and laptop repro for the jax_gpu sparse fixed-light mass_total[1] divergence; rerun the cancelled sparse remainder (jax_cpu sparse seed 1 from source_pix[2], the lightfixed mass_total[1] legs); GPU wave 1 attempt 2 (nautilus+emcee); commit the clean-slate rows in a project PR. No scientific acceptance implied. ## Runs -- 412537 — running — gpu — 2026-10-17 — hst-imaging-mass-samplers clean slate: hpc_a100_jax_gpu_sparse_fp64 (submit_hst_cleanslate_jax_gpu_sparse), --lens-light both, seeds 0-1, pin 2026.10.17.1 (PYAUTO_LIB_BASE=/mnt/ral/jnightin/inference/pins/2026.10.17.1), output root output/cleanslate_2026_10_17, results tag cleanslate_2026_10_17, prepared export output/prepared/cleanslate_2026_10_17/hpc_a100_jax_gpu_sparse_fp64/seed_//; human-authorised 2026-10-17 -- 412539 — running — ral — 2026-10-17 — hst-imaging-mass-samplers clean slate: hpc_a100_jax_cpu_dense_fp64 (submit_hst_cleanslate_jax_cpu_dense), --lens-light both, seeds 0-1, pin 2026.10.17.1 (PYAUTO_LIB_BASE=/mnt/ral/jnightin/inference/pins/2026.10.17.1), output root output/cleanslate_2026_10_17, results tag cleanslate_2026_10_17, prepared export output/prepared/cleanslate_2026_10_17/hpc_a100_jax_cpu_dense_fp64/seed_//; human-authorised 2026-10-17 -- 412541 — running — ral — 2026-10-17 — hst-imaging-mass-samplers clean slate: hpc_a100_jax_cpu_sparse_fp64 (submit_hst_cleanslate_jax_cpu_sparse), --lens-light both, seeds 0-1, pin 2026.10.17.1 (PYAUTO_LIB_BASE=/mnt/ral/jnightin/inference/pins/2026.10.17.1), output root output/cleanslate_2026_10_17, results tag cleanslate_2026_10_17, prepared export output/prepared/cleanslate_2026_10_17/hpc_a100_jax_cpu_sparse_fp64/seed_//; human-authorised 2026-10-17 -- 412543 — running — gpu — 2026-10-17 — hst-imaging-mass-samplers wave 1 (cleanslate_2026_10_17_wave1, submit_hst_cleanslate_mass_samplers_jax_gpu_dense): 24-arm slice nautilus/emcee/nuts/smc x sampler seeds 0-2 x both light modes on the base-seed-0 prepared problems of hpc_a100_jax_gpu_dense_fp64, array 0-23%4, 6 h containment, --dependency afterok:412535_0, pin 2026.10.17.1; human-authorised 2026-10-17 -- 412563 — open — ral — 2026-10-17 — hst-imaging-mass-samplers clean slate: numba_cpu dense SLaM base (48gb, 8 CPUs), --lens-light both, seeds 0-1, from autolens_inference main b91ce9c (PR #32), pin 2026.10.17.1, output root output/cleanslate_2026_10_17, results tag cleanslate_2026_10_17, prepared export; --stages-from hpc_a100_jax_cpu_dense_fp64:source_lp shares source_lp[1] from the jax_cpu dense chain with an in-job wait (hpc/wait_for_stage.sh); rows mark that stage shared and exclude its wall from the numba chain wall; submitted 2026-10-17T18:58Z by the architect; human decisions 2026-10-17 (8 CPUs, shared stage, submit now) -- 412565 — open — ral — 2026-10-17 — hst-imaging-mass-samplers clean slate: numba_cpu sparse SLaM base (64gb, 8 CPUs), --lens-light both, seeds 0-1, from autolens_inference main b91ce9c (PR #32), pin 2026.10.17.1, output root output/cleanslate_2026_10_17, results tag cleanslate_2026_10_17, prepared export; --stages-from hpc_a100_jax_cpu_dense_fp64:source_lp shares source_lp[1] from the jax_cpu dense chain with an in-job wait (hpc/wait_for_stage.sh); rows mark that stage shared and exclude its wall from the numba chain wall; submitted 2026-10-17T18:58Z by the architect; human decisions 2026-10-17 (8 CPUs, shared stage, submit now) ## Log +- 2026-10-19 — note — 412539 (jax_cpu dense) elapsed from sacct, read 2026-10-19 ~17:25Z by the session architect: 412539_0 1-05:59:53 (ended 2026-10-18T23:46:01), 412539_1 1-02:57:03 (ended 2026-10-18T20:43:11). This supplements the 2026-10-19 finished entry, which carries no wall. Its rows have still not been read. +- 2026-10-19 — run — 412543 failed — wall 0:47: hst-imaging-mass-samplers wave 1 (cleanslate_2026_10_17_wave1, submit_hst_cleanslate_mass_samplers_jax_gpu_dense): 24-arm slice nautilus/emcee/nuts/smc x sampler seeds 0-2 x both light modes on the base-seed-0 prepared problems of hpc_a100_jax_gpu_dense_fp64, array 0-23%4, 6 h containment, --dependency afterok:412535_0, pin 2026.10.17.1; human-authorised 2026-10-17 — sacct read 2026-10-19 ~17:25Z by the session architect: tasks _0.._17 all FAILED on 2026-10-17 (ends 18:51:38-20:32:26Z, elapsed 00:04:15-00:47:10; wall given is the longest task); _18 CANCELLED after 00:04:00; _[19-23] CANCELLED unstarted 2026-10-17T20:01:03Z. Failure classes as in the 2026-10-17 note (NUTS/SMC A100 OOM; nautilus/emcee row-writer TypeError). Its output and the 8 failed NUTS/SMC rows were moved aside on 2026-10-19 to aside/autolens_inference_cleanslate_wave1_attempt1/ (RAL and laptop) +- 2026-10-19 — note — HUMAN DECISION 2026-10-19, verbatim: "I want us to check the on going inference campaign on RAL, but I am ready for us to update folder structures on RAL and locally and clean things up so cancel on going jobs and we'll work on the clean up". Scheduler read 15:49:05Z: only 412541 and 412565 running; both scancelled ~15:51Z, queue empty; 412539 had COMPLETED. On "yes, go", autolens_inference#36 (flatten, closes #35) merged 15:53:33Z as c3ca4de; the RAL checkout fast-forwarded b91ce9c -> c3ca4de and the laptop to c3ca4de, after a full hpc/sync pull. flatten_cleanslate.py --apply: RAL 28 moves, 55 rewrites; laptop mirror 28 moves, 38 rewrites; both EXIT 0. New layout: output/autolens/imaging/hst/slam///seed_//, output/autolens/prepared//seed_//, rows at results/slam/imaging/hst/// (no __cleanslate_2026_10_17 suffix). RAL and laptop each hold 1974 files with identical paths and sizes; hpc/sync status dry run transfers 0. Human, verbatim: "check some older chats, think we dont need prepared, experiments and experiment_sources to exist anymore? Can we archive them somewhere separate to makr things easier to follow. Then do the apply". The wave-1 attempt-1 material (output/experiments 341M, output/experiment_sources 76M, results/searches/experiments with 8 failed NUTS/SMC rows) moved to /mnt/ral/jnightin/inference/aside/autolens_inference_cleanslate_wave1_attempt1/ on RAL and /mnt/c/Users/Jammy/Science/inference/aside/autolens_inference_cleanslate_wave1_attempt1/ on the laptop. prepared/ KEPT as the agent's proposal, not yet confirmed by the human: wave.py needs its manifests, and so does the sparse fixed-light repro. Partial stage dirs of the cancelled legs remain; they are not failures. Execution completion only. No compute submitted. +- 2026-10-19 — run — 412565 failed — wall 44:52: hst-imaging-mass-samplers clean slate: numba_cpu sparse SLaM base (64gb, 8 CPUs), --lens-light both, seeds 0-1, from autolens_inference main b91ce9c (PR #32), pin 2026.10.17.1, output root output/cleanslate_2026_10_17, results tag cleanslate_2026_10_17, prepared export; --stages-from hpc_a100_jax_cpu_dense_fp64:source_lp shares source_lp[1] from the jax_cpu dense chain with an in-job wait (hpc/wait_for_stage.sh); rows mark that stage shared and exclude its wall from the numba chain wall; submitted 2026-10-17T18:58Z by the architect; human decisions 2026-10-17 (8 CPUs, shared stage, submit now) — CANCELLED by scancel ~15:51Z on the human's decision, not a run failure (sacct CANCELLED, elapsed 1-20:52:11). Complete at cancel (.completed markers): seeds 0 and 1 slam_base light[1], source_pix[1], source_pix[2], mass_total[1] (source_lp[1] shared from jax_cpu dense via --stages-from). Incomplete: slam_base_lightfixed mass_total[1] on both seeds (running). Partial stage dirs left on disk under output/autolens/imaging/hst/slam//hpc_a100_numba_cpu_sparse_fp64/seed_/ (flat layout, autolens_inference#36) +- 2026-10-19 — run — 412541 failed — wall 47:04: hst-imaging-mass-samplers clean slate: hpc_a100_jax_cpu_sparse_fp64 (submit_hst_cleanslate_jax_cpu_sparse), --lens-light both, seeds 0-1, pin 2026.10.17.1 (PYAUTO_LIB_BASE=/mnt/ral/jnightin/inference/pins/2026.10.17.1), output root output/cleanslate_2026_10_17, results tag cleanslate_2026_10_17, prepared export output/prepared/cleanslate_2026_10_17/hpc_a100_jax_cpu_sparse_fp64/seed_//; human-authorised 2026-10-17 — CANCELLED by scancel ~15:51Z on the human's decision, not a run failure (sacct CANCELLED, elapsed 1-23:04:43). Complete at cancel (.completed markers): seed 0 slam_base light[1], source_lp[1], source_pix[1], source_pix[2], mass_total[1]; seed 1 source_lp[1], source_pix[1]. Incomplete: seed 0 slam_base_lightfixed mass_total[1] (running), seed 1 source_pix[2] (running) onward and its lightfixed leg. Partial stage dirs left on disk under output/autolens/imaging/hst/slam//hpc_a100_jax_cpu_sparse_fp64/seed_/ (flat layout, autolens_inference#36) +- 2026-10-19 — run — 412539 finished: hst-imaging-mass-samplers clean slate: hpc_a100_jax_cpu_dense_fp64 (submit_hst_cleanslate_jax_cpu_dense), --lens-light both, seeds 0-1, pin 2026.10.17.1 (PYAUTO_LIB_BASE=/mnt/ral/jnightin/inference/pins/2026.10.17.1), output root output/cleanslate_2026_10_17, results tag cleanslate_2026_10_17, prepared export output/prepared/cleanslate_2026_10_17/hpc_a100_jax_cpu_dense_fp64/seed_//; human-authorised 2026-10-17 — COMPLETED both seeds (sacct read 2026-10-19 15:49:05Z): 412539_0 ended 2026-10-18T23:46:01, 412539_1 ended 2026-10-18T20:43:11; rows not re-read in this record; on 2026-10-19 its output and rows moved to the flat layout (autolens_inference#36): output/autolens/imaging/hst/slam//hpc_a100_jax_cpu_dense_fp64/seed_/, output/autolens/prepared/hpc_a100_jax_cpu_dense_fp64/, results/slam/imaging/hst//hpc_a100_jax_cpu_dense_fp64/ +- 2026-10-18 — note — HUMAN DECISION 2026-10-18, verbatim: "lets just drop dense numba we would never use it." The numba_cpu dense leg (RAL 412563, failed with pool-worker SIGSEGV) is dropped from hst-imaging-mass-samplers with no rerun; the campaign's CPU base is jax_cpu dense, jax_cpu sparse and numba_cpu sparse. Canonical record: https://github.com/PyAutoLabs/PyAutoInsight/blob/main/decisions/hst-cleanslate-drop-numba-dense.md (Insight commit e289c04 on PR #36). +- 2026-10-18 — note — OBSERVATION awaiting the human's reading, not a conclusion (architect, 2026-10-18 ~16:55Z, rows read on RAL, pin 2026.10.17.1): the slam_base_lightfixed mass_total[1] stage on the jax_gpu SPARSE inversion (412537) lands on a wrong, sharply peaked solution on both seeds: max logL about -43 500 against about +31 649 for the linear mode; log Z -43567.1 / -43201.6; posterior pinned at slope 2.964+-0.0004 / 2.963+-0.0002, einstein_radius 2.493 / 2.427, gamma_1 0.0994 / 0.0998, ell_comps_0 0.59 / 0.57; truth_delta_sigma slope 2250 / 6275 (seed 0 einstein_radius 114, gamma_1 289); 48 700 evaluations at seed 0. The same stage on the jax_gpu DENSE inversion (412535) is the best-recovered row of the campaign (worst abs delta-sigma 0.24 / 0.34). Stages 1-4 are shared between the two light modes, so the difference is isolated to the fixed-light mass_total[1] under the sparse path. Possibly a library or driver fault in the sparse (w_tilde) inversion with ordinary (non-linear) lens light profiles rather than a sampler effect; a laptop reproduction on the exported prepared problem is proposed before any CPU or sampler wave uses the sparse fixed-light problem. Human reading pending. +- 2026-10-18 — run — 412563 failed — wall 0:01: hst-imaging-mass-samplers clean slate: numba_cpu dense SLaM base (48gb, 8 CPUs), --lens-light both, seeds 0-1, from autolens_inference main b91ce9c (PR #32), pin 2026.10.17.1, output root output/cleanslate_2026_10_17, results tag cleanslate_2026_10_17, prepared export; --stages-from hpc_a100_jax_cpu_dense_fp64:source_lp shares source_lp[1] from the jax_cpu dense chain with an in-job wait (hpc/wait_for_stage.sh); rows mark that stage shared and exclude its wall from the numba chain wall; submitted 2026-10-17T18:58Z by the architect; human decisions 2026-10-17 (8 CPUs, shared stage, submit now) — FAILED 1:0 both seeds after 00:00:57 (sacct 2026-10-18 ~16:55Z); the in-job wait for the shared jax_cpu dense source_lp[1] passed (completed 18:58Z); source_pix[1] started 19:59Z and the first likelihood batch killed two Nautilus pool workers with SIGSEGV (exit -11), MaxRSS 356 MB so not an OOM kill; PyAutoFit _check_workers_alive raised SearchException; rows written as failed carrying only the shared source_lp[1] stage. The sibling numba sparse leg 412565 runs the same pool without crashing. Rerun options (--cores 1, or NUMBA_THREADING_LAYER=workqueue) await a human decision +- 2026-10-18 — run — 412537 finished — wall 1:44: hst-imaging-mass-samplers clean slate: hpc_a100_jax_gpu_sparse_fp64 (submit_hst_cleanslate_jax_gpu_sparse), --lens-light both, seeds 0-1, pin 2026.10.17.1 (PYAUTO_LIB_BASE=/mnt/ral/jnightin/inference/pins/2026.10.17.1), output root output/cleanslate_2026_10_17, results tag cleanslate_2026_10_17, prepared export output/prepared/cleanslate_2026_10_17/hpc_a100_jax_gpu_sparse_fp64/seed_//; human-authorised 2026-10-17 — COMPLETED 0:0 both seeds on gpu (sacct 2026-10-18 ~16:55Z; elapsed 01:44:19 / 01:36:15, MaxRSS 7.5 GB); rows for both light modes and both seeds read on RAL, status complete, release_pin 2026.10.17.1; slam_base_lightfixed mass_total[1] rows have log Z -43567.1 / -43201.6 (see the 2026-10-18 observation note); rows not yet pulled to the laptop or committed - 2026-10-17 — note — 412543 wave 1 attempt 1 (read 19:40Z): every arm fails. NUTS arms 2, 6, 10, 14 FAILED with JaxRuntimeError RESOURCE_EXHAUSTED, out of memory allocating 385.32GiB [jit_log_l]; SMC arms 3, 7, 11, 15 FAILED allocating 48.45GiB [jit_scan]; their failed-status rows were written. Nautilus and emcee arms 0, 1, 4, 5, 8, 12, 16 completed their mass_total[1] fits (nautilus arm 0 about 13 min, emcee arm 1 about 37 min of job time) and then FAILED in the row writer, scripts/misc/experiments/run.py line 350, TypeError: keys must be str, int, float, bool or None, not tuple (tuple-keyed parameter names from the pinned PyAutoFit); no row written, output trees exist. The architect cancelled pending arms 18-23 at 19:40Z; arms 9, 13, 17 were still running. Fix task filed in Mind (draft/bug/autolens_inference/wave1_row_writer_tuple_keys_and_gpu_oom_reslice.md): fix the writer, re-run nautilus and emcee as attempt 2, defer NUTS and SMC with the OOM evidence. - 2026-10-17 — run — 412565 submitted: hst-imaging-mass-samplers clean slate: numba_cpu sparse SLaM base (64gb, 8 CPUs), --lens-light both, seeds 0-1, from autolens_inference main b91ce9c (PR #32), pin 2026.10.17.1, output root output/cleanslate_2026_10_17, results tag cleanslate_2026_10_17, prepared export; --stages-from hpc_a100_jax_cpu_dense_fp64:source_lp shares source_lp[1] from the jax_cpu dense chain with an in-job wait (hpc/wait_for_stage.sh); rows mark that stage shared and exclude its wall from the numba chain wall; submitted 2026-10-17T18:58Z by the architect; human decisions 2026-10-17 (8 CPUs, shared stage, submit now) - 2026-10-17 — run — 412563 submitted: hst-imaging-mass-samplers clean slate: numba_cpu dense SLaM base (48gb, 8 CPUs), --lens-light both, seeds 0-1, from autolens_inference main b91ce9c (PR #32), pin 2026.10.17.1, output root output/cleanslate_2026_10_17, results tag cleanslate_2026_10_17, prepared export; --stages-from hpc_a100_jax_cpu_dense_fp64:source_lp shares source_lp[1] from the jax_cpu dense chain with an in-job wait (hpc/wait_for_stage.sh); rows mark that stage shared and exclude its wall from the numba chain wall; submitted 2026-10-17T18:58Z by the architect; human decisions 2026-10-17 (8 CPUs, shared stage, submit now) diff --git a/state.json b/state.json index c5b24e9..c4a389d 100644 --- a/state.json +++ b/state.json @@ -3,8 +3,8 @@ "organ": "cortex", "repo": "PyAutoCortex", "status": "yellow", - "headline": "YELLOW \u2014 11 active \u00b7 20 running", - "updated": "2026-10-18T08:17:23Z", + "headline": "YELLOW \u2014 12 active \u00b7 16 running", + "updated": "2026-10-19T19:05:59Z", "pages_url": "https://pyautolabs.github.io/PyAutoCortex/", "items": [ { @@ -108,24 +108,6 @@ "text": "euclid_dr1: 402722 running (ral)", "url": null, "prompt": "Use the cortex skill. \u2014 resume euclid_dr1: read PyAutoCortex projects/euclid_dr1.md (Now, Runs, Log) and then /mnt/c/Users/Jammy/Science/euclid_dr1/wiki/project/state.md and the assistant autolens_assistant's AGENTS.md; tell me where I left off and what I said I would do next. Submit nothing and log nothing until I say." - }, - { - "severity": "info", - "text": "autolens_inference: 412537 running (gpu)", - "url": "https://github.com/PyAutoLabs/autolens_inference/issues/4", - "prompt": "Use the cortex skill. \u2014 resume autolens_inference: read PyAutoCortex projects/autolens_inference.md (Now, Runs, Log) and then /home/jammy/Code/PyAutoLabs/lens/autolens_inference/wiki/project/state.md and the assistant autolens_assistant's AGENTS.md; tell me where I left off and what I said I would do next. Submit nothing and log nothing until I say." - }, - { - "severity": "info", - "text": "autolens_inference: 412539 running (ral)", - "url": "https://github.com/PyAutoLabs/autolens_inference/issues/4", - "prompt": "Use the cortex skill. \u2014 resume autolens_inference: read PyAutoCortex projects/autolens_inference.md (Now, Runs, Log) and then /home/jammy/Code/PyAutoLabs/lens/autolens_inference/wiki/project/state.md and the assistant autolens_assistant's AGENTS.md; tell me where I left off and what I said I would do next. Submit nothing and log nothing until I say." - }, - { - "severity": "info", - "text": "autolens_inference: 412541 running (ral)", - "url": "https://github.com/PyAutoLabs/autolens_inference/issues/4", - "prompt": "Use the cortex skill. \u2014 resume autolens_inference: read PyAutoCortex projects/autolens_inference.md (Now, Runs, Log) and then /home/jammy/Code/PyAutoLabs/lens/autolens_inference/wiki/project/state.md and the assistant autolens_assistant's AGENTS.md; tell me where I left off and what I said I would do next. Submit nothing and log nothing until I say." } ] }