From 6c0a204038adf043ac5f0665f3d68228ae5ba789 Mon Sep 17 00:00:00 2001 From: Jammy2211 Date: Sat, 17 Oct 2026 16:13:53 +0100 Subject: [PATCH] docs: persist evening profiling campaign check-in 2026-10-17 - interferometer: parked at knowledge-complete (human decisions 2026-10-17: CPU lane closed, preload parked, GPU levers filed as research). Aris L2 at PR-open (PyAutoArray#636 + autolens_profiling#425); Knowledge (f)/(g)/(h) updated; FFT levers (#433), streaming + memo (#435) tasks parked. - decisions: add direct-conv-symmetric-gate (human decision 2026-10-16 on PyAutoArray#635) and its decisions.yaml entry. - point-source-source: closed as research (#29, #432); runtime warm-up task complete with fresh medians. - point-source-image-gpu: parked with the A100 reference measurement (#434); phase-2 inference fit dropped. - imaging: production over-sampling phase 1 at PR-open (#431); phase 2 needs-decision. - new parked campaign delaunay-nn-cpu + task (PyAutoArray#637). - checkins/2026-10-17.md with merge order and RAL cleanup list; last_checkin bumped; dashboard regenerated. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01QBeyBLN7ntDChx9v1fmt6M --- campaigns.yaml | 186 ++++++++++-------- checkins/2026-10-17.md | 90 +++++++++ dashboard.html | 2 +- dashboard.md | 58 +++--- decisions.yaml | 4 + decisions/direct-conv-symmetric-gate.md | 85 ++++++++ tasks/delaunay_nn_cpu_imaging.md | 60 ++++++ tasks/imaging_production_over_sampling.md | 10 + ...erometer_fixed_mapper_curvature_preload.md | 6 + tasks/interferometer_large_mask_aris.md | 59 +++++- ...ometer_nnls_memo_scattered_stream_guard.md | 12 ++ tasks/interferometer_streaming_scaling.md | 12 ++ .../interferometer_w_tilde_fft_size_levers.md | 9 + .../point_source_image_plane_gpu_breakdown.md | 12 ++ ...t_source_source_plane_chi_squared_speed.md | 11 ++ tasks/runtime_cell_single_jit_gpu_warmup.md | 7 + 16 files changed, 510 insertions(+), 113 deletions(-) create mode 100644 checkins/2026-10-17.md create mode 100644 decisions/direct-conv-symmetric-gate.md create mode 100644 tasks/delaunay_nn_cpu_imaging.md diff --git a/campaigns.yaml b/campaigns.yaml index 99c8e24..acdac10 100644 --- a/campaigns.yaml +++ b/campaigns.yaml @@ -1,5 +1,5 @@ version: 1 -last_checkin: '2026-10-17T11:09:26Z' +last_checkin: '2026-10-17T15:12:59Z' campaigns: - id: setup-baseline title: Setup baseline collection and scientific acceptance @@ -25,24 +25,24 @@ campaigns: evidence: https://github.com/PyAutoLabs/autolens_profiling/blob/4db4d988e520a82219152743748c3188d485a4ba/results/notes/point_source_cpu_campaign.md#campaign-completion-evidence-2026-10-04 - id: point-source-image-gpu title: Point-source image plane · A100 - status: needs-decision + status: parked updated: '2026-10-17' - next: 'No new image-plane fit measurement found. Correctness fix #768 remains shipped; current libraries now have - release 2026.10.12.1, not a measured performance baseline. Image-plane admission measurement still needs - authorization. #411 merged 2026-10-13, recording the paired-round MDI reinterpretation; no new measurement. - Check-in 2026-10-17: no project wiki/results/issue change since 2026-10-13 beyond - autolens_profiling#420/#422/#424 (interferometer only); releases 2026.10.16.1/.2 and 2026.10.17.1 supply no - new measurement.' + next: 'Parked as research, human decision 2026-10-17 (chat) ("just research to file"). Reference measurement autolens_profiling#434 + (issue #427, OPEN): job 412519 euclid-ral-gpu-2 exclusive A100, jax 0.10.2, fp64, 2026.10.17.1 private clone: + single call 0.934 ms (pre-#768 0.910), vmap-256 0.0125 ms (0.0120), compile 6.54 s; forward gradient finite 16/16, + equal to reverse to 1.6e-11; logL bit-exact; +2.7% not attributed (different node/date). A100 per-step breakdown + rows unusable. Phase-2 inference fit measurement dropped; no baseline admitted.' evidence: https://github.com/PyAutoLabs/autolens_profiling/blob/4db4d988e520a82219152743748c3188d485a4ba/wiki/campaigns/point_source_gpu_breakdown.md - id: point-source-source title: Point-source source plane - status: parked + status: complete updated: '2026-10-17' - next: 'Core remains parked; autolens_inference#21 reorganized setup evidence but added no point-source gradient - admission measurement. #408 merged median fields into 11 runtime cells; #412 merged 2026-10-13, completing - headline/repeat support. No new runtime sweep; keep Blackjax parked. Check-in 2026-10-17: no project - wiki/results/issue change since 2026-10-13 beyond autolens_profiling#420/#422/#424 (interferometer only); - releases 2026.10.16.1/.2 and 2026.10.17.1 supply no new measurement.' + next: 'Closed as research, human decision 2026-10-17 (chat): source-plane speed-ups "not worth pursuing again". + Evidence: autolens_inference#29 (issue #25) + autolens_profiling#432 (issue #428), both OPEN at PR-open: MultiStartAdam + forward gradients, 48 starts, 5 seeds, 2026.10.17.1 wheels (jax 0.11.2), fp64; likelihood+gradient ≤3.7% of fit + wall, A100 slower end to end than 8 CPU cores; default 300 steps stops 4/5 seeds on the limit, 3000 converges. + Runtime cell medians A100 0.233 ms / RAL CPU 0.147 ms. Blackjax stays parked; Adam step limit recorded as a PyAutoFit + search-settings note.' evidence: https://github.com/PyAutoLabs/autolens_profiling/blob/4db4d988e520a82219152743748c3188d485a4ba/wiki/campaigns/point_source_source_plane.md - id: cluster-pointsolver title: Cluster PointSolver robustness and performance @@ -56,20 +56,18 @@ campaigns: evidence: https://github.com/PyAutoLabs/autolens_profiling/blob/4db4d988e520a82219152743748c3188d485a4ba/wiki/campaigns/cluster_pointsolver.md - id: interferometer title: Interferometer likelihood and streaming - status: active + status: parked updated: '2026-10-17' - next: 'Aris large-mask (PyAutoPulse#62): library PRs PyAutoArray#630/#631/#632/#634 RELEASED in 2026.10.16.1 - (2026-10-16; 2026.10.16.2 and 2026.10.17.1 followed with no further PyAutoArray change). Phase-2 route-gate - record autolens_profiling#420 merged 2026-10-15; #422 (gate-60 wording historical) and #424 (lint re-pin - after #634) merged 2026-10-16; project main fd29902. L1 Cholesky reuse (#634) measured on the NumPy path: - solve+log-det 144.5→50.9 ms at 1,405 vertices, 1,268→373 ms at 2,947 (full call −22–24% / −51%). L2 - direct_conv symmetry (PyAutoArray#635) PAUSED by the human 2026-10-16, uncommitted on worktree - feature/direct-conv-symmetric (base e59d5fa9), no PR: synthetic Aris-512 serial RAL witness (job 411835) - 1.81×/1.85×/1.90× at S 1,428/2,540/5,712, computed entries bit-identical; real Aris data on the laptop 1.32× - (centroid order) / 1.46× (ymin order), work fraction 0.59/0.51; human decision 2026-10-16: rank by ascending - ymin, default gate 0.5→0.7; v3 (ymin + 0.7) present in the worktree diff, its test run, the RAL real-data - witness (≥1.3× curvature step), ship and write-up are owed. Streaming 1e8 CPU/A100 rows, the #332 memo - after-measurement, curvature preload (fixed-mapper science decision) and FFT levers unchanged.' + next: 'Evening 2026-10-17: human decision 2026-10-17 (chat): interferometer CPU work is exhaustive and closed; curvature + preload parked ("we will come back to it soon"); GPU levers "file wiki research but not make source code changes + for now", so no library prompt for JAX FFT pruning. Parked at knowledge-complete. Open PRs (human''s to merge + or close): L2 symmetric direct_conv PyAutoArray#636 (pending-release; real Aris RAL 1.49×/1.53×/1.58× at 1,459/2,577/5,746 + vertices, logL to round-off; decision record direct-conv-symmetric-gate) + record autolens_profiling#425; A100 + FFT levers #433 (pruned FFT 1.08×/1.18×/1.41× whole likelihood at alma/alma_high/jvla; pixel-scale coarsening + fails the 0.5-nat bar); streaming 1e8 CPU/A100 + memo after-measurement #435. Route-constant refit (RAL FFT/full + direct_conv 0.68 vs 0.89 predicted) deferred. Merge #425 after #636. Six open autolens_profiling PRs (#425, #431, + #432, #433, #434, #435) all regenerate the shared catalogue/dashboard: each merge after the first needs a rebase + + re-render; suggested order #425, #432, #431, #433, #434, #435. #425 after PyAutoArray#636.' evidence: https://github.com/PyAutoLabs/autolens_profiling/blob/fd29902bd09f/wiki/campaigns/interferometer_likelihood.md - id: linear-solver title: Linear-solver accuracy and cost @@ -86,25 +84,33 @@ campaigns: title: Production imaging and likelihood breakdown status: active updated: '2026-10-17' - next: 'No new production-over-sampling or post-certified measurements. #408 now adds median headline fields to - runtime cells; #412 merged 2026-10-13 and extends remaining headline surfaces. Production presets, - memo-blind harness, cumulative MGE steps and quick-update costs remain separate pending work. Scope - production-preset phase 1 (CPU smoke only); A100 measurement separately authorized. New related library - evidence: PyAutoArray#624/#625 (numba Sibson kernel; DelaunayNN within 2× Delaunay on Aris 1,405) released - 2026.10.16.1; follow-up PyAutoArray#629 (imaging DelaunayNN still ~2.7× Delaunay at sub_size 4: - unique_mappings 0.30 vs 0.10 s, adaptive_pixel_signals 0.16 vs 0.01 s; laptop, threads=1, load 3–5, not a - controlled measurement) is open and its Mind draft expects a Pulse campaign delaunay-nn-cpu that does not - yet exist — human decision whether to open one.' + next: 'Production over-sampling phase 1 at PR-open: autolens_profiling#431 (issue #430, OPEN, CPU smoke 8/8): --over-sampling + {flat1,production} on the 4 JAX Delaunay/DelaunayNN cells, euclid/hst presets. Findings: HST flat-1 adapt_split + pin fails on unmodified main (29155.000 vs 29140.296); numba cells'' production threshold never reaches sub 4 + for HST; rectangular JAX cells uncovered. Phase 2 (A100 production re-measure) needs-decision (human decision + 2026-10-17 (chat): undecided). DelaunayNN imaging moved to the parked delaunay-nn-cpu campaign. Post-certified, + quick-update plotting and other tasks unchanged.' evidence: https://github.com/PyAutoLabs/autolens_profiling/blob/4db4d988e520a82219152743748c3188d485a4ba/wiki/campaigns/post_certified_breakdown.md +- id: delaunay-nn-cpu + title: DelaunayNN (Sibson) mesh at Delaunay cost on the numba/NumPy CPU path + status: parked + updated: '2026-10-17' + next: 'Created parked, human decision 2026-10-17 (chat): "file in research to come back to in the future". PyAutoArray#637 + (issue #629, OPEN, pending-release, CI 3 legs green): RAL compute-10-4 exclusive 1 thread A-B-B-A ×2, 1,030 vertices + / 15,380 px: NN sub4 0.569→0.392 s (2.20× Delaunay, was 3.00×), sub2 0.235→0.189 s (1.75×); logL bit-identical; + fixes the size-0 Sibson NaN case. ~1.5× target not met; remaining cost is Sibson (~0.16 s) + wider curvature rows. + Target and two hazards recorded as future work, not filed.' + evidence: https://github.com/PyAutoLabs/PyAutoArray/pull/637 - id: measurement-tools title: Measurement reliability and profiling tools status: active updated: '2026-10-17' next: 'Timing audit #410–#412 and raw HST/MGE pilot producer #416 merged; implementation issues #362/#415 closed. - No fresh production measurements or scientific acceptance. C1/C11 repeat counts, C5 estimator policy, memo - interpretation, fresh runtime repeats and other tools remain pending. Check-in 2026-10-17: - autolens_profiling#418 (pilot qualification design) open since 2026-10-13 with CI green and no review; #417 - open; no new measurements.' + No fresh production measurements or scientific acceptance. C1/C11 repeat counts, C5 estimator policy, memo interpretation, + fresh runtime repeats and other tools remain pending. #418 and #417 open. Evening 2026-10-17 merge-order note: + Six open autolens_profiling PRs (#425, #431, #432, #433, #434, #435) all regenerate the shared catalogue/dashboard: + each merge after the first needs a rebase + re-render; suggested order #425, #432, #431, #433, #434, #435. #425 + after PyAutoArray#636.' evidence: https://github.com/PyAutoLabs/autolens_profiling/pull/416 - id: autofit title: PyAutoFit profiling bootstrap @@ -166,13 +172,12 @@ tasks: campaign: point-source-source title: Runtime cells' A100 `single_jit` includes the post-compile warm-up — source-plane 0.642 ms vs a steady 0.267 ms on the same node - status: active + status: complete priority: low path: tasks/runtime_cell_single_jit_gpu_warmup.md - updated: '2026-10-13' - next: '#408 merged 2026-10-09: median headline fields now wired into 11 runtime cells, superseding the old - imaging-not-wired note. #412 merged 2026-10-13 with remaining headline/repeat support. No new runtime sweep; - fresh median evidence still missing.' + updated: '2026-10-17' + next: 'Fresh medians at PR-open, autolens_profiling#432 (OPEN), 2026.10.17.1 wheels: A100 0.233 ms (single_jit 0.347), + RAL CPU 0.147 ms (0.177); pins pass. Remaining release-sweep cells gain the field at the next full sweep.' issue: https://github.com/PyAutoLabs/autolens_profiling/issues/371 evidence: https://github.com/PyAutoLabs/autolens_profiling/blob/719459fefb40c7e5be157f652818dbf923830557/scripts/point_source_source/source_plane/likelihood_runtime_solved.py - id: timing_noise_audit @@ -289,12 +294,14 @@ tasks: - id: imaging_production_over_sampling campaign: imaging title: Profile imaging pixelizations at production over-sampling - status: ready + status: needs-decision priority: normal path: tasks/imaging_production_over_sampling.md - updated: '2026-10-13' - next: 'Phase 1 (workspace-only): --instrument euclid|hst presets for the JAX Delaunay runtime/breakdown cells, - keeping flat-1 selectable; CPU smoke only. A100 re-measure is a separate compute-authorized phase.' + updated: '2026-10-17' + next: 'Phase 1 at PR-open: autolens_profiling#431 (issue #430, OPEN, CPU smoke 8/8). Phase 2 A100 production re-measure + needs-decision (human decision 2026-10-17 (chat): undecided); requires #431 merged and compute authorization. + Findings: HST flat-1 adapt_split pin fails on main; numba HST threshold never reaches sub 4; rectangular JAX cells + uncovered.' evidence: https://github.com/PyAutoLabs/PyAutoArray/pull/606 - id: interferometer_decision_matrix_last_cell campaign: interferometer @@ -312,35 +319,38 @@ tasks: - id: interferometer_fixed_mapper_curvature_preload campaign: interferometer title: 'Interferometer fixed-mapper searches: reuse the W~ curvature matrix across likelihood calls via `preloads.curvature_matrix`' - status: ready + status: parked priority: normal path: tasks/interferometer_fixed_mapper_curvature_preload.md - updated: '2026-10-13' - next: Phase 1 Mind draft (2026-10-04) still unstarted; run it through start_dev. Human science call first - - no production SLaM stage holds the mapper fixed today. + updated: '2026-10-17' + next: 'Parked, human decision 2026-10-17 (chat): interferometer CPU work closed; "we will come back to it soon". + Phase-1 Mind draft unstarted.' evidence: https://github.com/PyAutoLabs/PyAutoMind/blob/55a453cf/draft/feature/autoarray/interferometer_curvature_preload_phase1.md - id: interferometer_nnls_memo_scattered_stream_guard campaign: interferometer title: 'fnnls warm-start memo: stop it slowing scattered evaluation streams (interferometer CPU, autolens_profiling#332)' - status: active + status: parked priority: medium path: tasks/interferometer_nnls_memo_scattered_stream_guard.md - updated: '2026-10-13' - next: 'Library half shipped: PyAutoArray#615 (closes #613) merged 2026-10-06, in 2026.10.7.1. Local solver-only - witness (n=576): iid memo on/off 1.49x → 1.17x; walk 0.18x unchanged. Owed: re-run the autolens_profiling#332 - ALMA Delaunay 2.17x after-measurement (needs compute authorization).' + updated: '2026-10-17' + next: 'After-measurement at PR-open autolens_profiling#435 (issue #426, OPEN; job 412526, ral compute-10-2, load + ~10, 1 thread, 6 rounds, paired bootstrap 90% CI): Delaunay iid #332 flags 1.918 [1.887,1.925] (2.17 in #332); + 15 timed solves 1.018 [1.013,1.023]; Delaunay walk 0.46; rect iid 0.887/0.987; rect walk 0.15; FoM identical. + Not evidence #613 is fixed. Research owed: Nautilus-replay witness, varied-seed rounds. Parked, human decision + 2026-10-17 (chat).' issue: https://github.com/PyAutoLabs/PyAutoArray/issues/613 evidence: https://github.com/PyAutoLabs/PyAutoArray/pull/615 - id: interferometer_streaming_scaling campaign: interferometer title: 'Campaign: interferometer streaming (array-free) vs in-memory — memory and time scaling to 2e8 visibilities' - status: active + status: parked priority: high path: tasks/interferometer_streaming_scaling.md - updated: '2026-10-13' - next: 'Phase 1 CPU cells + results/streaming_scaling rows merged (autolens_profiling#375, 2026-10-04); wiki/campaigns/interferometer_streaming.md. - Optional: 1e8 CPU row and A100 rows (compute authorization). Library candidate unfiled: chunk transformer.image_from - in apply_sparse_operator (memory).' + updated: '2026-10-17' + next: 'Phase 2 at PR-open autolens_profiling#435 (issue #368 reopened, OPEN): from_stream chunk 65536 fp64, 1e8 + vis RAL CPU 707.5 s / 1.78 GB RSS, A100 157.3 s / 1.48 GB device; compile ~3 s constant. In-memory fails at 2.5e5 + (RAL 10 GB cap) and 4e6 (A100); uncapped CPU ~7.5 kB/vis. Optional 2e8 rows are research; library candidate (chunk + transformer.image_from) unfiled per human decision 2026-10-17 (chat). Parked.' issue: https://github.com/PyAutoLabs/autolens_profiling/issues/368 evidence: https://github.com/PyAutoLabs/autolens_profiling/blob/719459fefb40c7e5be157f652818dbf923830557/wiki/campaigns/interferometer_streaming.md - id: interferometer_large_mask_aris @@ -350,24 +360,24 @@ tasks: priority: high path: tasks/interferometer_large_mask_aris.md updated: '2026-10-17' - next: 'Released: #630/#631/#632/#634 in 2026.10.16.1 (2026-10-16). Phase-2 record #420 merged 2026-10-15 (table - re-pinned at fd29902); #422/#424 merged 2026-10-16. L2 (PyAutoArray#635) paused 2026-10-16 with v3 (ymin - order, gate 0.7) uncommitted in worktree feature/direct-conv-symmetric; synthetic witness PASS 1.81–1.90× - (RAL 411835), real-data laptop 1.46× at ymin order; owed on resume: verify v3 + full suite, RAL real-data - witness ≥1.3× (data and private library clones staged under scratch/aris_l2_20261016), Heart check, - ship_library PR, write-up (results note, Knowledge section, decision record direct-conv-symmetric-gate). - κ=1.32 means the #632 cost-model constant for direct_conv is effectively refit when L2 ships.' + next: 'L2 shipped to PR: PyAutoArray#636 (head 62a1e3ac, pending-release) + autolens_profiling#425 (results note), + both OPEN, CI green. RAL real-data witness 1.49×/1.53×/1.58× at 1,459/2,577/5,746 vertices (pred 1.48×/1.50×/1.53×), + logL to round-off; peak RSS 43 GB without NUFFT chunking. Decision record direct-conv-symmetric-gate added. Route-constant + refit (0.68 measured vs 0.89 predicted) deferred: CPU lane closed, human decision 2026-10-17 (chat). Remaining: + human merge of #636 then #425, and release.' issue: https://github.com/PyAutoLabs/PyAutoPulse/issues/62 evidence: https://github.com/PyAutoLabs/PyAutoArray/issues/626 - id: interferometer_w_tilde_fft_size_levers campaign: interferometer title: 'Interferometer W~ curvature matrix is FFT-bound on the mask extent: pruned padded FFT and real-space pixel scale on the A100' - status: ready + status: parked priority: normal path: tasks/interferometer_w_tilde_fft_size_levers.md - updated: '2026-10-13' - next: Unstarted (no PR/issue/branch/results, 2026-10-04). Select one bounded step when prioritised. + updated: '2026-10-17' + next: 'A100 research filed at PR-open autolens_profiling#433 (issue #429, OPEN; job 412515): pruned padded FFT 1.08×/1.18×/1.41× + whole likelihood at alma/alma_high/jvla, F exact to 1.7e-15; pixel-scale coarsening shifts log evidence by +1,310 + to +2,757 nats (fails). No library prompt, human decision 2026-10-17 (chat). Parked.' - id: point_solver_profiling_cells campaign: cluster-pointsolver title: 'PointSolver profiling cells: lensed quasar → cluster runtime tier → single/multi-source → multiplane' @@ -380,24 +390,24 @@ tasks: - id: point_source_image_plane_gpu_breakdown campaign: point-source-image-gpu title: 'Point-source A100 speed-up campaign: profile and optimize with the shared breakdown' - status: needs-decision + status: parked priority: high path: tasks/point_source_image_plane_gpu_breakdown.md - updated: '2026-10-13' - next: 'No new image-plane fit measurement found. Correctness fix #768 remains shipped; current libraries - now have release 2026.10.12.1, not a measured performance baseline. Image-plane admission measurement still - needs authorization. #411 merged 2026-10-13, recording the paired-round MDI reinterpretation; no new measurement.' + updated: '2026-10-17' + next: 'Parked, human decision 2026-10-17 (chat). Reference A100 measurement autolens_profiling#434 (OPEN, job 412519): + 0.934 ms single call, 0.0125 ms at vmap-256, forward gradient fixed; phase-2 inference fit measurement dropped.' issue: https://github.com/PyAutoLabs/PyAutoLens/issues/767 evidence: https://github.com/PyAutoLabs/PyAutoLens/pull/768 - id: point_source_source_plane_chi_squared_speed campaign: point-source-source title: Point-source source-plane chi-squared speed-up campaign — remaining candidates (phases 1–2e shipped) - status: blocked + status: complete priority: low path: tasks/point_source_source_plane_chi_squared_speed.md - updated: '2026-10-13' - next: 'Still blocked on a point-source gradient-sampler admission measurement. Inference #21 is setup organization, - not that measurement. Keep parked.' + updated: '2026-10-17' + next: 'Complete as research, human decision 2026-10-17 (chat). Admission evidence autolens_inference#29 + autolens_profiling#432 + (OPEN): likelihood+gradient ≤3.7% of MultiStartAdam fit wall, so a likelihood speed-up caps at a few percent. + Blackjax stays parked.' evidence: https://github.com/PyAutoLabs/autolens_inference/pull/17 - id: pointsolver_cpu_speed_campaign_remainder campaign: point-source-image-cpu @@ -459,3 +469,15 @@ tasks: next: Unchanged; targets PyAutoLens/PyAutoGalaxy (PyAutoLens#680 closed 2026-07-31, no follow-up issue). Select one bounded step when prioritised. evidence: https://github.com/PyAutoLabs/PyAutoLens/issues/680 +- id: delaunay_nn_cpu_imaging + campaign: delaunay-nn-cpu + title: DelaunayNN (Sibson) imaging likelihood at close to Delaunay cost on the numba/NumPy CPU path + status: parked + priority: normal + path: tasks/delaunay_nn_cpu_imaging.md + updated: '2026-10-17' + next: 'Parked, human decision 2026-10-17 (chat). PyAutoArray#637 (OPEN, pending-release) at PR-open: sub4 2.20× + Delaunay (was 3.00×), sub2 1.75×; logL bit-identical. Future (unfiled): ~1.5× target; hazards sub_slim_indexes_for_pix_index + padding and Sibson rows with mid-row gaps.' + issue: https://github.com/PyAutoLabs/PyAutoArray/issues/629 + evidence: https://github.com/PyAutoLabs/PyAutoArray/pull/637 diff --git a/checkins/2026-10-17.md b/checkins/2026-10-17.md new file mode 100644 index 0000000..e373f58 --- /dev/null +++ b/checkins/2026-10-17.md @@ -0,0 +1,90 @@ +# Profiling check-in — 2026-10-17 + +Reviewed at 2026-10-17T15:12:59Z; morning check-in 2026-10-17T11:09:26Z (Pulse PR #66, unmerged at this +check-in: the human's `!` merge commands did not take effect, so this update is stacked on it). +Review time and board generation are not measurement times. This check-in records verified +outcomes and the human's decisions of 2026-10-17 (chat). It authorizes no compute, default or +baseline change, campaign freeze, scientific acceptance, release or merge. + +## Human decisions 2026-10-17 (chat) + +These are recorded in the ledger `next:` texts and task check-in sections. They are not new +decision records unless stated. + +1. Interferometer CPU work is exhaustive and closed. Curvature preload parked: "we will come back to it soon". +2. Interferometer GPU levers: "file wiki research but not make source code changes for now". No library prompt for JAX FFT pruning; campaign parked at knowledge-complete. +3. DelaunayNN: "file in research to come back to in the future". New campaign `delaunay-nn-cpu`, parked; the ~1.5× target and two hazards recorded as future work, not filed. +4. Point-source source-plane speed-ups "not worth pursuing again, file as research". Campaign closed as research. +5. Point-source image plane: "just research to file". Campaign parked; A100 numbers recorded as a reference measurement; phase-2 inference fit measurement dropped. +6. Imaging phase 2 (A100 production over-sampling re-measure): undecided → needs-decision. + +New decision record: [`direct-conv-symmetric-gate`](../decisions/direct-conv-symmetric-gate.md), +capturing the human decision of 2026-10-16 on [PyAutoArray#635](https://github.com/PyAutoLabs/PyAutoArray/issues/635) +(ymin column ordering, default gate 0.5 → 0.7). + +## Verified availability + +- All PRs below are OPEN, CI green and unmerged. Every Heart read was STALE ("release validation incomplete: no rehearsal for current source"); no override was used anywhere. +- Mind: all related tasks are awaiting-merge. Cortex run log pushed for RAL jobs 412495/412498. +- Measurements were made by the project workers today; this check-in did not re-run them. + +## Campaign review + +| Campaign | State | Change / next bounded step | +|---|---|---| +| Interferometer likelihood and streaming | parked | Decisions 1–2. L2 symmetric `direct_conv` PR-open (PyAutoArray#636 + record #425): real Aris RAL 1.49× / 1.53× / 1.58× at 1,459 / 2,577 / 5,746 vertices (pred 1.48× / 1.50× / 1.53×), logL to round-off, peak RSS 43 GB without NUFFT chunking; gain over the old default route at 1,459 only ~1.02× (FFT 15.32 s). RAL FFT/full direct_conv 0.68 measured vs 0.89 predicted: constants refit deferred. A100 FFT levers (#433): pruned FFT 1.08× / 1.18× / 1.41× whole likelihood at alma / alma_high / jvla, F exact to 1.7e-15; pixel-scale coarsening shifts log evidence +1,310 / +1,807 / +2,757 nats (fails 0.5-nat bar). Streaming (#435): 1e8 vis CPU 707.5 s / 1.78 GB RSS, A100 157.3 s / 1.48 GB device, compile ~3 s; in-memory fails at 2.5e5 (RAL 10 GB cap) and 4e6 (A100). Memo after-measurement (#435, job 412526): Delaunay iid 1.918 [1.887, 1.925] vs 2.17 in #332; 15 solves 1.018 [1.013, 1.023]; walk 0.46; rect iid 0.887 / 0.987; rect walk 0.15; FoM identical — an after-measurement, not a fix. | +| Point-source source plane | complete | Decision 4. autolens_inference#29 + autolens_profiling#432: MultiStartAdam forward gradients, 48 starts, 5 seeds, 2026.10.17.1 wheels (jax 0.11.2), fp64. Default 300 steps: 4/5 seeds hit the limit, two 1.4σ off; 3000 steps: all converge into the Nautilus logL range, ≤ 0.8σ. Likelihood+gradient ≤ 3.7 % of fit wall; per step 1.7 ms CPU / 4.4 ms A100 (0.24 / 0.35 ms likelihood+grad); A100 slower end to end than 8 CPU cores. Runtime cell medians A100 0.233 ms (single_jit 0.347), RAL CPU 0.147 ms (0.177), pins pass. Blackjax stays parked; Adam step-limit is a PyAutoFit search-settings note, not a task. | +| Point-source image plane · A100 | parked | Decision 5. #434, job 412519 euclid-ral-gpu-2 exclusive, A100 80GB, jax 0.10.2, fp64: single call 0.934 ms (pre-#768 0.910), vmap-256 0.0125 ms (0.0120), compile 6.54 s; forward gradient finite 16/16, equal to reverse to 1.6e-11 (was NaN); reverse/likelihood 1.889; FD 4/6 (same points as before); logL bit-exact; within-run MDI 1.50 %; +2.7 % not attributed (different node/date). A100 per-step breakdown rows unusable (11/43 negative). | +| Production imaging and likelihood breakdown | active | Production over-sampling phase 1 PR-open (#431, smoke 8/8): `--over-sampling {flat1,production}` on the 4 JAX Delaunay/DelaunayNN cells; euclid [4,4,2]@[0.1,0.3] and hst [4,2,2]@[0.1,0.3], sub 4 where source S/N > 3 else 2; HST 2,022 / 15,361 px at sub 4, Euclid 1,057 / 3,841. Findings: HST flat-1 adapt_split pin fails on unmodified main (29155.000 vs 29140.296, 5e-4); numba HST threshold never reaches sub 4; rectangular JAX cells uncovered. Phase 2 A100: needs-decision (decision 6). | +| DelaunayNN mesh on the CPU path (new) | parked | Decision 3. PyAutoArray#637 (issue #629, pending-release, CI 3 legs green): RAL compute-10-4 exclusive, 1 thread, A-B-B-A ×2, 1,030 vertices / 15,380 px: NN sub4 0.569 → 0.392 s (2.20× Delaunay, was 3.00×); NN sub2 0.235 → 0.189 s (1.75×, was 2.10×); Delaunay sub4 0.189 → 0.178 s; logL bit-identical; adaptive signals 158 → 6.5 ms; unique_mappings 22.5 → 17.0 ms; Sibson ~0.16 s (~41 %). ~1.5× target not met; fixes the size-0 Sibson NaN edge case. | +| Measurement reliability and profiling tools | active | Merge-order note added (below). #417/#418 open; no new measurements. | +| Setup baseline, Point-source image CPU, Cluster PointSolver, Linear-solver, PyAutoFit, Critical curves, Certified solver | unchanged | Carried from the morning check-in; no new evidence this evening. | + +## Task changes + +| Task | State | Note | +|---|---|---| +| [interferometer_large_mask_aris](../tasks/interferometer_large_mask_aris.md) | active | L2 at PR-open; Knowledge (f) L2 row, (g) bullet and (h) added. Open only until #636/#425 merge and release. | +| [interferometer_w_tilde_fft_size_levers](../tasks/interferometer_w_tilde_fft_size_levers.md) | parked | A100 research filed (#433); no library prompt. | +| [interferometer_streaming_scaling](../tasks/interferometer_streaming_scaling.md) | parked | 1e8 CPU/A100 rows at PR-open (#435). | +| [interferometer_nnls_memo_scattered_stream_guard](../tasks/interferometer_nnls_memo_scattered_stream_guard.md) | parked | After-measurement at PR-open (#435); Nautilus-replay witness and varied-seed rounds owed as research. | +| [interferometer_fixed_mapper_curvature_preload](../tasks/interferometer_fixed_mapper_curvature_preload.md) | parked | Decision 1. | +| [point_source_source_plane_chi_squared_speed](../tasks/point_source_source_plane_chi_squared_speed.md) | complete | Closed as research. | +| [runtime_cell_single_jit_gpu_warmup](../tasks/runtime_cell_single_jit_gpu_warmup.md) | complete | Fresh medians (#432). | +| [point_source_image_plane_gpu_breakdown](../tasks/point_source_image_plane_gpu_breakdown.md) | parked | Reference measurement (#434). | +| [imaging_production_over_sampling](../tasks/imaging_production_over_sampling.md) | needs-decision | Phase 1 PR-open (#431); phase 2 undecided. | +| [delaunay_nn_cpu_imaging](../tasks/delaunay_nn_cpu_imaging.md) (new) | parked | #637 at PR-open. | + +## Open PRs (the human's to merge or close) + +| PR | Content | +|---|---| +| [PyAutoArray#636](https://github.com/PyAutoLabs/PyAutoArray/pull/636) | L2 symmetric direct_conv (head 62a1e3ac, pending-release) | +| [PyAutoArray#637](https://github.com/PyAutoLabs/PyAutoArray/pull/637) | DelaunayNN padded-row and size-0 Sibson fixes (pending-release) | +| [autolens_profiling#425](https://github.com/PyAutoLabs/autolens_profiling/pull/425) | L2 results note | +| [autolens_profiling#431](https://github.com/PyAutoLabs/autolens_profiling/pull/431) | Imaging production over-sampling phase 1 | +| [autolens_profiling#432](https://github.com/PyAutoLabs/autolens_profiling/pull/432) | Point-source source-plane gradient fit share + runtime medians | +| [autolens_profiling#433](https://github.com/PyAutoLabs/autolens_profiling/pull/433) | A100 FFT size levers (titles still name the dropped CPU constants) | +| [autolens_profiling#434](https://github.com/PyAutoLabs/autolens_profiling/pull/434) | Point-source image-plane A100 reference measurement | +| [autolens_profiling#435](https://github.com/PyAutoLabs/autolens_profiling/pull/435) | Interferometer streaming 1e8 rows + memo after-measurement | +| [autolens_inference#29](https://github.com/PyAutoLabs/autolens_inference/pull/29) | MultiStartAdam point-source source-plane fits | +| [PyAutoPulse#66](https://github.com/PyAutoLabs/PyAutoPulse/pull/66) | Morning check-in (this update is stacked on it) | + +**Merge order.** The six autolens_profiling PRs (#425, #431, #432, #433, #434, #435) all regenerate +the shared catalogue/dashboard, so each merge after the first needs a rebase and re-render before the +next. Suggested order: #425, #432, #431, #433, #434, #435. Library PRs #636 and #637 are file-disjoint +and independent; merge #425 after #636. + +## Unavailable or unverified + +- No release contains #636 or #637; all "speed-up" figures are PR-open measurements, not released behaviour or scientific acceptance. +- RAL jax 0.10.2 (used for the A100 image-plane and FFT-lever jobs) is outside 2026.10.17.1's declared jax range; source SHAs are the provenance. +- Owed research, not tasks: memo Nautilus-replay witness, varied-seed rounds, optional 2e8 streaming rows, RAL route-constant refit. + +## RAL cleanup owed (recorded, not deleted) + +- `/mnt/ral/jnightin/scratch/fft_levers_20261017/` (abandoned partial clones) +- `/mnt/ral/jnightin/scratch/fft_levers_a100_20261017/` (partial extract) +- `/mnt/ral/jnightin/scratch/fft_levers_stack.tgz`, `fft_levers_jobs/`, `fft_levers_results/` (results JSON is committed in #433) +- `/mnt/ral/jnightin/profiling/autolens_profiling_wt/ps-image-admission_stale_*` +- Worker private clones under `profiling/` and `inference/` are kept for reuse. diff --git a/dashboard.html b/dashboard.html index 56a21d9..d848496 100644 --- a/dashboard.html +++ b/dashboard.html @@ -646,7 +646,7 @@ - PyAutoPulse: https://github.com/PyAutoLabs/PyAutoPulse - autolens_profiling: https://github.com/PyAutoLabs/autolens_profiling - autofit_profiling: https://github.com/PyAutoLabs/autofit_profiling
Last updated 2026-10-17 08:07:26 UTC
↻ Update

-

Active campaigns

CampaignStatusLinks
Setup baseline collection and scientific acceptance
2 tasks · next step

Review the Pulse collection choices (candidate input hashes and 31-worker resource plan verified without execution) alongside project correctness audit #417. Resolve data origin/suitability, named host/stack/full settings, witness/tolerances and background-load method. No freeze, compute or acceptance authorized. Check-in 2026-10-17: no project wiki/results/issue change since 2026-10-13 beyond autolens_profiling#420/#422/#424 (interferometer only); releases 2026.10.16.1/.2 and 2026.10.17.1 supply no new measurement.

Reviewed 2026-10-17

Active tasks

Freeze the setup baseline specification, then collect and review evidence↗

needs-decision · high

HST/MGE raw fixed-parameter CPU runner merged in #416; Pulse handoff #55 merged and Mind implementation closed. Resolve exact production host/stack/data, full settings, independent correctness witness/tolerances and background-load method. No collection, campaign freeze or acceptance authorized; global compile dispatch remains unavailable.

Prepare and review the HST/MGE fp64 CPU pilot↗

needs-decision · high

Review the Pulse collection choices (candidate input hashes and 31-worker resource plan verified without execution) alongside project correctness audit #417. Resolve data origin/suitability, named host/stack/full settings, witness/tolerances and background-load method. No freeze, compute or acceptance authorized.

needs-decision
↗☷
Point-source image plane · CPU
1 task · next step

No new committed campaign measurements. Phases 1–4 remain shipped; epic closure is a human decision. Carried leftovers remain pending. RAL queue empty at review, but transfer preview timed out: cleanup folders and unpulled results unverified. Select one owed cleanup/measurement phase only after review. Check-in 2026-10-17: no project wiki/results/issue change since 2026-10-13 beyond autolens_profiling#420/#422/#424 (interferometer only); releases 2026.10.16.1/.2 and 2026.10.17.1 supply no new measurement. RAL queue holds only euclid_dr1 jobs; cleanup folders and unpulled results still unverified.

Reviewed 2026-10-17

Active tasks

active
↗☷
Point-source image plane · A100
1 task · next step

No new image-plane fit measurement found. Correctness fix #768 remains shipped; current libraries now have release 2026.10.12.1, not a measured performance baseline. Image-plane admission measurement still needs authorization. #411 merged 2026-10-13, recording the paired-round MDI reinterpretation; no new measurement. Check-in 2026-10-17: no project wiki/results/issue change since 2026-10-13 beyond autolens_profiling#420/#422/#424 (interferometer only); releases 2026.10.16.1/.2 and 2026.10.17.1 supply no new measurement.

Reviewed 2026-10-17

Active tasks

needs-decision
↗☷
Point-source source plane
2 tasks · next step

Core remains parked; autolens_inference#21 reorganized setup evidence but added no point-source gradient admission measurement. #408 merged median fields into 11 runtime cells; #412 merged 2026-10-13, completing headline/repeat support. No new runtime sweep; keep Blackjax parked. Check-in 2026-10-17: no project wiki/results/issue change since 2026-10-13 beyond autolens_profiling#420/#422/#424 (interferometer only); releases 2026.10.16.1/.2 and 2026.10.17.1 supply no new measurement.

Reviewed 2026-10-17

Active tasks

parked
↗☷
Cluster PointSolver robustness and performance
2 tasks · next step

No campaign wiki/results change or new matching implementation claim found. Representative cluster baseline and observable-overflow policy remain unstarted. Choose one bounded phase; inherited single-source results do not qualify a representative cluster baseline. Check-in 2026-10-17: no project wiki/results/issue change since 2026-10-13 beyond autolens_profiling#420/#422/#424 (interferometer only); releases 2026.10.16.1/.2 and 2026.10.17.1 supply no new measurement.

Reviewed 2026-10-17

Active tasks

active
↗☷
Interferometer likelihood and streaming
5 tasks · next step

Aris large-mask (PyAutoPulse#62): library PRs PyAutoArray#630/#631/#632/#634 RELEASED in 2026.10.16.1 (2026-10-16; 2026.10.16.2 and 2026.10.17.1 followed with no further PyAutoArray change). Phase-2 route-gate record autolens_profiling#420 merged 2026-10-15; #422 (gate-60 wording historical) and #424 (lint re-pin after #634) merged 2026-10-16; project main fd29902. L1 Cholesky reuse (#634) measured on the NumPy path: solve+log-det 144.5→50.9 ms at 1,405 vertices, 1,268→373 ms at 2,947 (full call −22–24% / −51%). L2 direct_conv symmetry (PyAutoArray#635) PAUSED by the human 2026-10-16, uncommitted on worktree feature/direct-conv-symmetric (base e59d5fa9), no PR: synthetic Aris-512 serial RAL witness (job 411835) 1.81×/1.85×/1.90× at S 1,428/2,540/5,712, computed entries bit-identical; real Aris data on the laptop 1.32× (centroid order) / 1.46× (ymin order), work fraction 0.59/0.51; human decision 2026-10-16: rank by ascending ymin, default gate 0.5→0.7; v3 (ymin + 0.7) present in the worktree diff, its test run, the RAL real-data witness (≥1.3× curvature step), ship and write-up are owed. Streaming 1e8 CPU/A100 rows, the #332 memo after-measurement, curvature preload (fixed-mapper science decision) and FFT levers unchanged.

Reviewed 2026-10-17

Active tasks

Interferometer large-mask likelihood (Aris 512×512): sparse data-vector memory, AdaptSplit log-det backend parity, numba route-gate cost model↗

active · high

Released: #630/#631/#632/#634 in 2026.10.16.1 (2026-10-16). Phase-2 record #420 merged 2026-10-15 (table re-pinned at fd29902); #422/#424 merged 2026-10-16. L2 (PyAutoArray#635) paused 2026-10-16 with v3 (ymin order, gate 0.7) uncommitted in worktree feature/direct-conv-symmetric; synthetic witness PASS 1.81–1.90× (RAL 411835), real-data laptop 1.46× at ymin order; owed on resume: verify v3 + full suite, RAL real-data witness ≥1.3× (data and private library clones staged under scratch/aris_l2_20261016), Heart check, ship_library PR, write-up (results note, Knowledge section, decision record direct-conv-symmetric-gate). κ=1.32 means the #632 cost-model constant for direct_conv is effectively refit when L2 ships.

active
↗☷
Linear-solver accuracy and cost
0 tasks · next step

Retain the 2026-10-08 human decision: Jacobi for Mapper, raw+polish for MGE-only, no default change. Project wiki still says decision pending (stale). No new solver measurements after the recorded phase-5 results; release 2026.10.12.1 does not supply a rerun. Wide-prior/interferometer gaps remain evidence gaps, not authorized tasks. Check-in 2026-10-17: no project wiki/results/issue change since 2026-10-13 beyond autolens_profiling#420/#422/#424 (interferometer only); releases 2026.10.16.1/.2 and 2026.10.17.1 supply no new measurement.

Reviewed 2026-10-17

Active tasks

No open tasks.

active
↗☷
Production imaging and likelihood breakdown
3 tasks · next step

No new production-over-sampling or post-certified measurements. #408 now adds median headline fields to runtime cells; #412 merged 2026-10-13 and extends remaining headline surfaces. Production presets, memo-blind harness, cumulative MGE steps and quick-update costs remain separate pending work. Scope production-preset phase 1 (CPU smoke only); A100 measurement separately authorized. New related library evidence: PyAutoArray#624/#625 (numba Sibson kernel; DelaunayNN within 2× Delaunay on Aris 1,405) released 2026.10.16.1; follow-up PyAutoArray#629 (imaging DelaunayNN still ~2.7× Delaunay at sub_size 4: unique_mappings 0.30 vs 0.10 s, adaptive_pixel_signals 0.16 vs 0.01 s; laptop, threads=1, load 3–5, not a controlled measurement) is open and its Mind draft expects a Pulse campaign delaunay-nn-cpu that does not yet exist — human decision whether to open one.

Reviewed 2026-10-17

Active tasks

active
↗☷
Measurement reliability and profiling tools
8 tasks · next step

Timing audit #410–#412 and raw HST/MGE pilot producer #416 merged; implementation issues #362/#415 closed. No fresh production measurements or scientific acceptance. C1/C11 repeat counts, C5 estimator policy, memo interpretation, fresh runtime repeats and other tools remain pending. Check-in 2026-10-17: autolens_profiling#418 (pilot qualification design) open since 2026-10-13 with CI green and no review; #417 open; no new measurements.

Reviewed 2026-10-17

Active tasks

jax_compile/probe.py lost its cell builder with the searches tier — give profiling its own

active · high

Manual HST/MGE fp64 jit builder #414 and separate raw pilot runner #416 merged. Bounded Mind implementations complete; wider builder coverage/global compile dispatch still unavailable. Agree a next capability or separately authorize a qualified pilot; no production compute authorized. Check-in 2026-10-17: autolens_profiling#418 (pilot qualification design) open since 2026-10-13 with CI green and no review; #417 open; no new measurements.

active
↗☷
PyAutoFit profiling bootstrap
1 task · next step

B4a (autofit_profiling#3) and B4b (autofit_profiling#5, issue #4 closed, merged 2026-10-15) shipped: search.fit breakdown cell, profiling-summary@2 exporter, EP/graphical N3/N10/N30 baselines and analytic parity cell; #6 repointed RAL paths to the category layout (2026-10-15). Pulse fit fixture/campaign merged (#60, #61); fit receipt ok at autofit_profiling 58e56b9 (31 records, fetched 2026-10-15T22:58Z), so the post-deploy refresh is done. Epic 2 not opened: next bounded step is a human-scoped epic-2 campaign row; B5c inference work belongs to Insight, not Pulse.

Reviewed 2026-10-17

Active tasks

autofit_profiling: bootstrap the repo + general PyAutoFit profiling epic

active · normal

B4a (autofit_profiling#3) and B4b (autofit_profiling#5, issue #4 closed, merged 2026-10-15) shipped: search.fit breakdown cell, profiling-summary@2 exporter, EP/graphical N3/N10/N30 baselines and analytic parity cell; #6 repointed RAL paths to the category layout (2026-10-15). Pulse fit fixture/campaign merged (#60, #61); fit receipt ok at autofit_profiling 58e56b9 (31 records, fetched 2026-10-15T22:58Z), so the post-deploy refresh is done. Epic 2 not opened: next bounded step is a human-scoped epic-2 campaign row; B5c inference work belongs to Insight, not Pulse.

active
↗☷
Critical curves and evaluation grids
0 tasks · next step

No new campaign measurement. Phase 3b cap fix remains shipped; project wiki still incorrectly says #646 awaits merge/release. Seed coverage and path completion remain unresolved; retain marching-squares decision and ask the human whether to scope the next phase. Check-in 2026-10-17: no project wiki/results/issue change since 2026-10-13 beyond autolens_profiling#420/#422/#424 (interferometer only); releases 2026.10.16.1/.2 and 2026.10.17.1 supply no new measurement.

Reviewed 2026-10-17

Active tasks

No open tasks.

needs-decision
↗☷
Certified positive solver
0 tasks · next step

Parked unchanged. No new campaign evidence; existing C2/library/workspace drafts remain pending. Do not infer admission from solver timing, and do not change default or restart the campaign. Check-in 2026-10-17: no project wiki/results/issue change since 2026-10-13 beyond autolens_profiling#420/#422/#424 (interferometer only); releases 2026.10.16.1/.2 and 2026.10.17.1 supply no new measurement.

Reviewed 2026-10-17

Active tasks

No open tasks.

parked
↗☷

Open means tracked, not necessarily running. Full ledger ↗

Profiling Results

Decision History

Route interferometer curvature by a calibrated two-constant cost modelExclude zero-weight mappings from AdaptSplit pixel signals on Delaunay meshes onlyKeep Jacobi PDIP as the Mapper default and raw+polish as the MGE-only default

Decision History

Gate the symmetric direct_conv kernel at work fraction 0.7, with ymin column orderingRoute interferometer curvature by a calibrated two-constant cost modelExclude zero-weight mappings from AdaptSplit pixel signals on Delaunay meshes onlyKeep Jacobi PDIP as the Mapper default and raw+polish as the MGE-only default