add-process-mep-extension
- - + +Context
See proposal.md for motivation and the two spec deltas for requirements. PythonExtension already implements native Rust frontend/backend traits. The baseline published SDK 0.3.0 exposed NativeExtension::frontend_backend, open_protocol, model methods and admission helpers. Its wire request type used a numeric ID, while the canonical contract admits numeric and string IDs; adopting helpers alone is not full draft conformance. The baseline Python0.4.2 implementation declared legacy MEP0.1 only. Candidate0.5.0 uses publishedSDK 0.4.0 and opts into the exact JSON model draft after native/WASM executable qualification. finos/morphir c552dac801f23f2b2b566d29e08780723498ce36 defines exact MEP 0.2.0-draft.1 and model envelope 0.1.0-draft.1. These versions are independent of SDK 0.3.0, compiler 0.4.2, claims format and IR version.
Goals and non-goals
Add a native Rust process guest and current-draft model adapters while retaining WASM and legacy behavior. Keep language semantics shared. Initial model tuples use morphir-ir/morphir-json and supported Library IR 3.0.0 and draft 4.0.0. Rich Ion profiles, unsupported distribution kinds, additional Python syntax, cancellation/progress and performance claims are not introduced. A Python CLI stdio wrapper can follow separately; the selected CLI installation remains mise/pipx.
Decisions
RNT-1. Add a thin native Rust guest
Add a process guest crate/entry point, depending on the maintained compiler and published SDK, with no PyO3 or CPython requirement. The proposed command is:
morphir-python-extension --stdioNo-argument launches also select stdio, matching the canonical native bundler and installed process launcher. Reuse the SDK's session endpoint and dispatch. Keep model adaptation in the shared PythonExtension so native, WASM and process paths agree. Put stdio effects in the guest layer. A Python CLI wrapper would reuse the engine but retain an interpreter/launcher dependency; a frozen Python CLI would bundle an unnecessary runtime for this guest.
| Boundary | Current release | Proposed change |
|---|---|---|
| MEP | Legacy 0.1 | Retain 0.1 and add exact 0.2.0-draft.1 |
| Model envelope | Not advertised | 0.1.0-draft.1, admitted JSON profile |
| Extension runtime | WASM | WASM and native process |
| SDK | 0.3.0, numeric-only wire IDs | Qualified published pin covering the canonical wire contract |
| Semantic IR | Supported v3 and draft v4 Library models | Same semantic scope, explicit admitted tuples |
SDK-1. Resolve concrete SDK wire gaps before claiming conformance
Pin a published SDK that supports canonical numeric and string request IDs and the required draft behavior. SDK 0.3.0 currently has numeric-only ExtensionRequest.id, so link and qualify an upstream SDK update if no suitable release exists. Do not vendor a sibling checkout or silently coerce wire IDs. Both WASM and process must pass the same ID-preservation vectors before the new draft is advertised. The concrete prerequisite is request ID support, issue 316, tracked as morphir-4wrx. SDK 0.4.0 is merged in PR 317 as d51daa8aea7ae09ae295ff254d72b27770c64104. All CI and the committed-source publication dry run pass. The maintainer approved publication at that exact merge. Tag crates/morphir-extension-sdk/v0.4.0 triggered successful workflow40695514380; Cargo independently downloaded the published0.4.0 crate. The compiler now pins registry dependency =0.4.0, without a sibling or Git SDK dependency. Related upstream work is MEP protocol type generation, issue 241; its existence is not evidence that the wire gap is fixed.
WIN-1. Keep the standalone Windows runtime independent of Python
Binary inspection of the b38e00e development executable found VCRUNTIME140.dll. Build only the standalone guest with -C target-feature=+crt-static; preserve the Python wheel's existing ABI configuration. The Windows producer uses MSVC dumpbin /DEPENDENTS, records exact artifact/tool/checker identities, and refuses VC or Python runtime DLL imports. Assembly requires that source-bound proof. This CRT configuration needs separate code generation from the Python wheel; it is not a compatible cache consumer.
MEP-1. Negotiate the new draft explicitly
Retain legacy 0.1 operations and exact 0.2.0-draft.1 offers. Opt in only after supported model methods and executable conformance pass. Reuse SDK model admission and session rules; method constants alone are not support. Never enable model methods under legacy negotiation. Keep optional flags false until implemented.
MOD-1. Start with the explicit JSON model profile
Implement compile_model/generate_model over the existing compile/generate core. Admit complete operation/profile/codec/envelope/IR tuples and their bilateral limits. The initial profile is morphir-ir/morphir-json with envelope 0.1.0-draft.1. Use the canonical base64 payload, exact byte length/hash and critical-field rules. After SDK admission, materialize JSON through published morphir-json 0.2.0 with finite depth/node/string budgets, preserving numeric lexemes and literal private-looking keys. Bound outgoing JSON before extending payload buffers. Reject unadvertised Ion profiles, guarantees and unsupported semantic forms. This achieves bounded new-draft support without implying rich Ion preservation or silently converting rich input.
IO-1. Treat framing and lifetime as a bounded server boundary
Use Content-Length JSON-RPC framing, byte-based lengths, an 8 KiB header ceiling and an initial 16 MiB complete-body ceiling. Advertised decoded/encoded model limits must fit those outer bounds, including envelope overhead. Enforce local bounds even if the peer advertises more. Check partial input, duplicate lengths, integer overflow, invalid UTF-8/JSON and EOF before unbounded allocation. Reserve stdout for frames and flush responses. Follow initialization, capability locks, shutdown and exit; do not respond to notifications. Host supervision owns deadlines and descendant cleanup.
RLS-1. Qualify dual-runtime distribution
Retain the portable WASM guest and publish native process executables for the advertised targets, with canonical empty arguments selecting stdio by default. Use a canonical host-readable descriptor with runtime/platform/mode/digest/claims bound per artifact. Validate the staged descriptor with pinned host tooling; do not assume that either the existing WASM-only packager or native-only bundle command can package a mixed release unchanged. Preserve historical assets. Release exact native/CLI package pairs if shared compiler changes affect them.
HST-1. Qualify hosts independently
The Rust host provides ProcessLaunch/ProcessChannel and verified-byte staging. It clears the child environment, favoring a directly executable Rust guest. Qualify a pinned conforming Rust host and test each documented MoonBit route independently. Use a pinned source build when a released host lacks the new draft, and state that prerequisite. If a route needs upstream work, record that dependency and label it unsupported until it passes. Existing MoonBit process wrappers demonstrate framing, not Python extension activation.
add-process-mep-extension
The b38e00e development candidate passed all five platform producers and process-only assembly in run 40676766327. Cached-tool identity, IR and generated-artifact digests agreed with WASM/API references. The verified CLI recipe selects separate named repositories, with uninstall before changing the active runtime. A real macOS check switched WASM/process/WASM, rejected a foreign-only descriptor and refused changed installed bytes. Integrity checks now run inside every platform producer with fresh project caches and exact expected/observed hashes; atomic replacement avoids modifying hardlinked inputs. These checks remain distinct from the pending new-draft MCK prerequisite. Replacement source 820c9d0 passed all five producers and assembly in run 40685569455. Its Windows dependency proof admits only OS DLLs, with no VC or Python runtime imports. All eleven combined release files, the five producer receipts and the Windows checker proof were independently matched to the collected assembly hashes. Runtime switching, installed-byte tampering and foreign-platform refusal passed on the assembled Linux candidate.
The current candidate supplies the previously missing platform, conformance, transfer and promotion evidence. The assessment concerns implementation assurance; it is not publication authorization or a general performance result. Per-tag qualification still gates publication at the exact approved source. No native speed claim follows from this build design.
Windows checker setup and source integrity
The full candidate run exposed Cargo/libgit2 failing on a locked Gleam snapshot path of 265 characters under the default Windows Cargo home. A short checker-only cache under RUNNER_TEMP/mck reduces the longest pinned path to 252 characters. A second diagnostic showed that transient parent Git options did not persist into the Rust submodule: inaccessible long fixtures produced false deleted/modified statuses. Preparation now configures LF and long-path handling locally in each checkout and requires clean source before and after compilation. An actual tracked-file edit is still refused and preserved. No source patch, ignored diff, changed corpus or reduced platform requirement was accepted.
Diagnostic run 40726235322 passes with clean parent/submodule state and a prepared Windows checker receipt. The manual mck-diagnostic mode builds only that checker and retains raw compiler and Git diagnostics; it cannot qualify or publish a release. Its separate concurrency group permits diagnosis while the full platform run continues. The normal workflow remains the promotion authority. The first Linux preparation of the shared checker and WASM adapter took 12m24s, followed by 1m31s for canonical execution and saved rechecks; these are assurance costs, not guest performance measurements.
Download assets and transformed-byte checks
The assembly category stages all five raw executables with checksums, a separately named process descriptor, and one deterministic gzip/tar download containing the canonical eleven-file process bundle. Archive metadata uses fixed ownership/timestamps and explicit executable permissions; every copied and archived payload must retain its original digest. Corrupt transfers, mixed source/claims/platforms, symlinks and extra files are refused without leaving partial assets. The candidate receipt and final checksum list bind the download bytes to assembly. The existing runtime-selection and source-pinned host checks consume the extracted archive, preserving the build-once contract. Actual 820c9d0 archive bytes pass local offline switching, installed-byte refusal and unsupported-platform refusal. The same manual/extension-tag workflow now requires canonical producer MCK rechecks before assembly. Final promotion copies verified raw process assets, their separate descriptor/archive, WASM assets and all producer evidence, then seals their final names. Manual runs stage this exact release set without publishing. An approved extension tag publishes it after all selected jobs pass.
Source-pinned host and timing observations
The standalone developer checker in tools/extension-qualification/ pins Rust host source d51daa8aea7ae09ae295ff254d72b27770c64104, independently of the compiler's published SDK dependency. Actual b38e00e WASM and macOS process bytes pass legacy initialization, v3/v4 type/value compile-generate-recompile, diagnostic parity, orderly close and a real timeout with child reaping. This is not executable MCK or new-draft qualification.
A release-built checker ran five fresh samples per workload on one macOS ARM64 machine. The small workload has one record and one function; the medium workload has 200 records and 100 typed conditional functions. Raw results, input files, exact artifact/checker/lock/toolchain hashes and descriptive timings are retained in .dev/qualification/timings-b38-release/. Startup includes fresh guest initialization and WASM module compilation. OS caches are not cleared; later v3/v4 operations share the session and WASM runs first. No general speed claim follows. The existing manual/release candidate consumer now owns source-pinned host checks and raw timing retention; it builds the checker once and consumes the already qualified guest bytes. The extension promotion job depends on that consumer. Estimated uncached Linux host build cost is 3 to 6 minutes; the observed local release build took 53 seconds. Actual CI timing remains to be measured.
Qualification result
All 16 implementation tasks are complete. Full candidate run 40726356123 succeeds at source bce5d859b524edf452b3031e2e2e56fc32f4da71. Later documentation, task and spec-sync edits preserve the production code and check definitions. Publication remains pending exact maintainer approval and per-tag validation.
| Obligation | Observed evidence |
|---|---|
| Platform and API agreement | All five process producers and WASM pass installed offline v3/v4 operation checks. Full process diagnostic records match native Python API 0.1.2 with compiler 0.5.0. Windows runtime imports pass. |
| Canonical current/legacy MEP | Six reports each contain 91 required observations with successful saved rechecks from clean merged85cf4352 source, totaling 546. Local transfer verification preserves original paths and exact source/tool/artifact/claim identities. |
| Assembly and download form | Five raw process targets combine into an eleven-file canonical bundle and deterministic archive. Extracted-archive switching, installed-byte tampering and foreign-platform refusal pass. An actual four-platform input set is refused. |
| Real hosts and timings | Pinned Rust host d51daa8 passes actual WASM/process legacy lifecycle, v3/v4 operations, diagnostics, timeout and child cleanup. Ten raw samples, five per small/medium workload, retain inputs and exact toolchain/artifact identities. No general speed claim. |
| Promotion and retention | Final staging has 39 assets plus SHA256SUMS. Every final checksum, six raw report/recheck pairs and all eleven archive payload/mode entries are independently checked. Historical 0.4.0/0.4.2 WASM assets remain uploaded with their original digests. |
Local validation passes 2632 tests with five intentional skips, strict typing, BDD, focused evidence/assembly tests, actionlint, site checks and strict OpenSpec validation. SDK 0.4.0 is published; Python/native/extension tags have not yet been published. The OpenSpec change remains active until that operational follow-up is complete.
CLI publication prerequisite
Approved native tag native-v0.1.2 at f2032e3adaf72854a3907dc092e8abb757f08b4a published successfully in workflow 40758932916; all six PyPI files match the qualified release bytes. Approved CLI tag python-v0.1.1 correctly failed workflow 40764461472 before publication: core 0.1.0 documentation links and its project URL changed after publication. Direct wheel comparison proves identical runtime files, with only METADATA and RECORD differing. Bug morphir-5hxy.6 prepares core 0.1.1 followed by CLI 0.1.2. The strict published-core byte guard remains unchanged, and the failed CLI tag is preserved. Native 0.1.2 and extension 0.5.0 retain their approved source and versions. Corrected core/CLI publication awaits exact approval after qualification.
The extension-tag run also exposed a separate routing error: its native candidate verifier received extension/python/v0.5.0 and correctly refused it as a non-native tag. Bug morphir-5hxy.7 keeps that verifier and the complete wheel-set check, using the source native version for extension/manual contexts while validating explicit native tags unchanged. A regression executes the production workflow shell for manual, native, extension and invalid-native routes; the extension route fails before the fix and all routes pass afterward. Protected extension publication remains held. Corrected-source approval is required before retargeting the unpublished extension tag and repeating full qualification; published native tags remain unchanged.
Selective PR CI also exposed a stale committed host-evidence fallback after earlier documentation input changes. Bug morphir-5hxy.8 replaces it with actual qualification from clean commit 6e88b295017d346007409bdd7182e8584c91c622: the currently documented 0.4.2 bundle passes the pinned Rust-host recipe and offline round trips, and both pinned MoonBit refusal probes pass. A regression verifies the real committed cache against all documented inputs, including when CI separately runs fresh host checks.
Repository evidence and references
- rust/crates/morphir-python-binding/src/lib.rs: shared PythonExtension traits and truthful current capabilities.
- rust/crates/morphir-python-native/src/lib.rs: current private two-function JSON bridge; it is not a full MEP transport.
- Published morphir-extension-sdk0.4.0: canonical request IDs, native session dispatch and model admission helpers.
- finos/morphir-rust 26d5e707e5ec94d16421db6380edbdc865d85c04: process launch/channel and stdio frontend fixture. Fixture framing must not be copied as a hardened server.
- finos/morphir-moonbit bd3adc52e6b8f6368fdaee50f76535799732f8d6: portable MEP embedding wrapper. No claim of Python host compatibility follows.
- Canonical MEP contract and model transport rules.
Related work: morphir-bj9r owns the selected CLI installation route; morphir-ukvn.5.98 tracks the earlier MoonBit extension qualification context.