From c7708752d3d173a7952ede101c60ea0c44e9cc02 Mon Sep 17 00:00:00 2001 From: Damian Reeves <957246+DamianReeves@users.noreply.github.com> Date: Tue, 27 Oct 2026 11:23:12 -0500 Subject: [PATCH 01/31] docs: plan Python binding migration Link the A4 migration to native Python API, CLI round trips and coordinated provider cutover without assuming a Rust speed advantage. Refs: morphir-ukvn.1.6 --- .../migrate-python-binding/.openspec.yaml | 2 + .../migrate-python-binding/design.html | 146 ++++++++++++++++++ .../migrate-python-binding/proposal.md | 37 +++++ .../specs/python/compiler/spec.md | 65 ++++++++ .../specs/python/extension/spec.md | 53 +++++++ .../specs/tools/python-codegen/spec.md | 69 +++++++++ .../changes/migrate-python-binding/tasks.md | 56 +++++++ 7 files changed, 428 insertions(+) create mode 100644 openspec/changes/migrate-python-binding/.openspec.yaml create mode 100644 openspec/changes/migrate-python-binding/design.html create mode 100644 openspec/changes/migrate-python-binding/proposal.md create mode 100644 openspec/changes/migrate-python-binding/specs/python/compiler/spec.md create mode 100644 openspec/changes/migrate-python-binding/specs/python/extension/spec.md create mode 100644 openspec/changes/migrate-python-binding/specs/tools/python-codegen/spec.md create mode 100644 openspec/changes/migrate-python-binding/tasks.md diff --git a/openspec/changes/migrate-python-binding/.openspec.yaml b/openspec/changes/migrate-python-binding/.openspec.yaml new file mode 100644 index 00000000..dda53b8a --- /dev/null +++ b/openspec/changes/migrate-python-binding/.openspec.yaml @@ -0,0 +1,2 @@ +schema: html-design +created: 2026-10-27 diff --git a/openspec/changes/migrate-python-binding/design.html b/openspec/changes/migrate-python-binding/design.html new file mode 100644 index 00000000..0214c351 --- /dev/null +++ b/openspec/changes/migrate-python-binding/design.html @@ -0,0 +1,146 @@ + + + + + +Migrate the Python binding: design + + + + + + + + +
+

Migrate the Python binding

+

OpenSpec change migrate-python-binding · Bead morphir-ukvn.1.6 · 2026-10-27 · Proposed

+ +

Context and evidence

+

See proposal.md for motivation. Inspection used the clean local morphir-rust checkout at f4298d0d416628ce3c910b662a78e224cf8eba8a, on chore/openspec-html-designs. This is an observed source revision, not a claim that it is the eventual migration base. Before copying, select the current upstream main revision and reconcile binding changes since this snapshot.

+ + + + + + + +
Observed codeConsequence
crates/morphir-python-binding, version 0.4.0; native PythonExtension and SDK WASM exportsThis is a Python language compiler implemented in Rust, currently neither a CPython module nor a PyPI distribution.
Ruff component crates pinned together to 0.0.15; core 0.2.0 and SDK 0.3.0 with sibling pathsMove compiler tests and lockfile resolution together; replace sibling paths with independently resolvable dependencies.
ADT, conditional, tuple, function, IR-version and module tests; Gherkin acceptance; native examplesMigration starts by preserving this behavior. More Python syntax is separate work under morphir-rust-72s.
Rust host tests include Python fixtures from the crate directory; release tasks and CI classify the crateDeleting the directory alone would break host tests, CI selection and bundle publishing.
Python core and tools use separate Hatch distributions; Python minimum 3.14; existing CLI only inspects/converts IRIntroduce compiler access without giving the dependency-free core a native build requirement.
Python supports Library and Specs across its versioned codecs; binding generation accepts only Library and a subset of nodesSuccessful IR decoding does not imply the Python backend can generate it. Report explicit subset diagnostics.
+

The approved ecosystem roadmap records independent language repositories and published Rust dependencies. Its A2 template qualification, morphir-ukvn.1.3.3, remains open. Preserve that dependency for final A4 delivery and reuse the canonical bundler rather than creating a new descriptor format.

+
+

Goals and boundaries

+

Own the compiler here; preserve native Rust and MEP callers; expose in-memory compilation and generation to Python; deliver CLI round trips. Python callers receive the existing Python IR models and ordinary immutable result types.

+

This increment preserves the existing language subset. It does not add general Python execution, arbitrary third-party imports, a second pure-Python compiler, full MCK conformance, new component hosts, or additional evaluation engines. The Python host wrapper in morphir-opua.1.4.4 remains independent. Existing IR inspection/conversion retains its broader version/distribution coverage.

+
+

Decisions

+

REPO-1. Keep a Rust half with independently resolvable dependencies

+
rust/
+  Cargo.toml
+  Cargo.lock
+  crates/morphir-python-binding/  # migrated compiler, tests and examples
+  crates/morphir-python-native/   # CPython bridge only
+packages/
+  morphir/                       # existing dependency-free library
+  morphir-tools/                 # public Python compiler API and CLI
+  morphir-python-native/         # native distribution metadata and typing
+
+

Use a Cargo workspace under rust/; retain the crate name and public Rust API. Keep PyO3 in the bridge crate so Cargo consumers and the WASM guest never link CPython. Use exact compatible published Morphir core/SDK versions after checking the current APIs. If the APIs require unreleased changes, pin all related Morphir crates to one exact git revision and record the publication dependency. No sibling paths, branches, vendored copies of core or hidden local patches. Carry workspace metadata explicitly and pin the four Ruff dependencies as a group. Cargo consumers can use the package from an exact morphir-python git revision before any crates.io publication.

+

Alternative: put PyO3 features in the compiler crate. Rejected because feature unification can pull Python linking into native Rust or WASM builds. A mechanical copy comes before language behavior changes, with source provenance and a file inventory.

+

API-1. Expose a Python API over an in-process bridge

+

Use PyO3 and Maturin to build morphir_python_native._native. The bridge invokes the same SDK frontend/backend traits as Rust callers and exchanges serialized requests/results internally. No host, subprocess, Extism runtime or MEP handshake is needed by Python applications. Detach from Python while owned Rust requests are compiled/generated; convert errors deliberately and do not expose panics or serialized envelopes as the public API.

+

Public entry points live in morphir_tools.python: compile_sources and generate_python. Inputs are frozen source-document and options types. Compilation returns a union of version-specific successes and a diagnostic failure; a success always contains its matching Python IR model. Generation returns a success with a tuple of frozen relative-path/text artifacts, or a diagnostic failure. Keep warnings on successes; use a severity enum and source locations distinct from IR member paths. API objects must not hold Rust AST objects or mutable SDK dictionaries. Compiler diagnostic codes are centralized consistently with morphir.ir.diagnostics.

+

Alternative: run the WASM extension from Python. It could reuse an existing bundle, but would add a runtime, host protocol and loading costs to every Python installation. A pure-Python compiler would duplicate a substantial implementation before evidence justifies it. Ordinary Python integration here means a Python API with a native implementation, not a second parser.

+

IR-1. Use versioned JSON as the internal compatibility boundary

+

Decode bridge output through morphir.ir.format_version and the matching v3/v4 codec before constructing a success. Encode Python models through those codecs before generation. Do not invent parallel Python domain models or encode the wire format in the CLI. Any explicit cross-version conversion goes through morphir.ir.migrate; generation does not silently downgrade or drop unsupported attributes.

+

Default new Python API and CLI compilation to stable v3.0 Library; allow explicit draft v4. Preserve the existing native Rust request choices. Generation initially accepts supported v3.0 and draft v4 Libraries only; Specs, dependency packages and unsupported language nodes retain explicit rejection. Legacy formats remain usable through the existing ir convert workflow before generation. JSON precision and parameter order must survive both directions. Cross-codec fixtures include arbitrary integers, unary nested function types, ADTs, tuples, private modules and independent IR. This is a qualification obligation, not assumed parity.

+

Alternative: expose every Rust model through PyO3. Rejected for the initial migration because it would duplicate this repository's versioned Python types and couple the API to Rust representation changes. JSON adds measurable cost, so benchmark it separately.

+

PKG-1. Add a native dependency without changing the core library

+

Add distribution morphir-python-native, containing the private extension, stubs and py.typed. Tools declare a compatible dependency range and resolve it automatically; the public API remains in the tools namespace. Keep the two existing distributions on Hatch, their current tags and independent versions. Give the new native distribution its own version/tag, native-v*, and keep the binding bundle on extension/python/v*. No release version is selected by this proposal. Qualify dependency installation using a local wheelhouse before changing release metadata.

+

Initial wheel qualification targets CPython 3.14 on Linux glibc x86_64/aarch64, macOS x86_64/arm64 and Windows x86_64. Pin a released PyO3/Maturin pair supporting the selected interpreter and use the stable ABI where qualified. Do not infer free-threaded support from an abi3 wheel; CPython 3.14t needs distinct qualification. Document source builds and other platforms separately. Build the native sdist with all referenced Rust workspace/path dependency files and prove a clean archive-only wheel build. Keep compiler imports lazy so existing IR commands do not initialize it.

+

Maturin's mixed-project guidance supports a private native submodule and typing files. PyO3's distribution guidance describes interpreter selection and wheel/ABI choices. These support the packaging approach, not a performance claim.

+

CLI-1. Keep I/O in tools and publish complete output

+

The compile command reads a single file or a sorted directory tree and makes one API request containing relative source paths. A single file uses its parent as root; a directory uses itself. Explicit exposure selections use repeated --expose options; the Python API also retains the distinction between omitted exposure and an explicitly empty tuple. Do not recursively follow symlink source directories or discover installed packages. Require a package identity rather than guessing it from filesystem names.

+

The generate command uses the shared reader/codec before the API. Require an absent or empty destination. Validate every artifact path, including collisions and symlink traversal, and write the complete artifact tree in a sibling temporary directory before publishing it. On failure remove staging, preserve existing files and report I/O errors without a traceback. Compile uses a temporary file and atomic replacement. Keep data/output and diagnostics separate; diagnostics go to stderr.

+

PERF-1. Measure the delivered path before claiming speed

+

Add a reproducible benchmark command with checked-in small and multi-module fixtures plus generated scale parameters. Measure native Rust compilation/generation, private bridge calls, public Python API calls and CLI end-to-end runs independently. Include warmup, repetitions, medians and spread, input bytes/module counts, target version, build profile, hardware, toolchains and serialization costs. Use release builds and separate cold import/startup from steady-state execution.

+

Python ast.parse can be reported only as a parsing baseline; it performs less work than compilation and cannot establish a compiler speed ratio. There is no equivalent pure-Python compiler here. State that limitation in results. Benchmarks report observations; noisy timing thresholds do not gate PRs, and a slower bridge result does not justify silently substituting a compiler.

+
+

Build and release assurance

+

Follow the personal build-master skill; this checkout has no repository copy. Preserve existing checks and add scoped Rust/native/bundle categories. No runtime or cost measurements were collected during planning, so estimated performance gains and CI savings are unknown.

+ + + + + + + +
Category and ownerTrigger, risk and deadlineProducer and consumer
Python, Python maintainersPython/API/IR or shared changes; regression detection before mergeExisting Ruff, mypy, pyright, pytest/coverage and behave; API tests consume one built local native wheel.
Rust/compiler, binding maintainersCompiler/dependency or shared changes; subset and protocol regressions before mergeCargo fmt/clippy/test including migrated acceptance and doc tests; external Cargo consumer proves independent use.
Wheel install, packaging maintainersBridge/packaging changes; ABI and missing-file failures before merge on representative Linux and platform-specific checks when touchedBuild once per selected platform/ABI; clean-environment API/CLI smoke consumes that exact wheel. No rebuild in consumers.
MEP bundle, extension maintainersBinding/bundler changes and cutover; host compatibility before mergeOne release WASM build; canonical bundling; digest/capability validation and isolated offline host round trip consume the exact bundle.
Release qualification, release maintainerTag or explicit qualification request; all advertised wheel platforms and source archive before promotionCandidate-bound wheel matrix, sdist rebuild, license/security scans and bundle receipts. Publication consumes qualified bytes; failed, skipped, absent or mismatched evidence blocks promotion.
Benchmarks, binding maintainersExplicit performance request or release assessment; no claim before resultsFixed-candidate release builds; retain environment and raw measurements with report. No live schedule is created here.
+

Test the change selector and aggregate for relevant/irrelevant paths, shared dependencies, cancellations, missing results and expected skips. Documentation-only changes do not install Rust. Keep release qualification callable without publishing; no wheel matrix, bundle publication or registry promotion runs merely because planning files changed. Cache keys include lockfiles, target, toolchain and features; caches never count as test evidence. Record candidate SHA, dependency lock hash, producer run and artifact digest in qualification receipts.

+
Build-master comparison and evidence limits +

Decision scope: proposed PR/release graph for native wheels and MEP bundles. Failure impact is moderate for compiler regressions and high for a broken or mismatched distribution. Merge checks protect touched behavior; release checks protect every advertised artifact. Maintainers named above respond to failures and block promotion until repaired.

+ + + + + + + +
CriterionWeightAll platforms on every PRScoped PR plus exact-candidate release
Risk and coverage fit25U, selectors unimplementedU, selectors unimplemented
Feedback and placement20U, no run measurementsU, no run measurements
Release evidence and trust20U, receipts unimplementedU, receipts unimplemented
Execution efficiency15U, no measured build graphU, no measured build graph
Ownership and structure102, category owners identified2, category owners identified
Operability and recovery10U, refusal cases untestedU, refusal cases untested
+

Totals withheld; 90% of rubric weight has unknown implementation evidence. Gate preservation, candidate identity, trust and failure routing are all unknown until executable checks verify them. Scoped CI is the provisional choice because the approved roadmap requires it. Reassess with actual run records during implementation. This design provides no merge or release approval.

+
+
+

Risks and trade-offs

+ + + + + + + +
RiskMitigation
Published SDK APIs lag the source snapshotResolve and compile against actual artifacts; use one exact git revision as a documented temporary fallback. Never copy shared core crates here.
Rust and Python v4 codecs differ despite matching namesCross-codec tests are required before marking Python API success; correct the supported mapping and reject unrepresentable attributes rather than dropping them.
Tools now depend on native wheelsKeep the core independent, test wheel-only installations and source archives, publish the compatible native dependency before tools, and state the qualified platform matrix.
JSON bridge overhead erases a possible Rust advantageRetain a replaceable private boundary and measure each stage. Performance remains a hypothesis.
Removing the crate breaks host fixture includes or CI matricesMove compiler tests here, replace host fixtures with independent protocol fixtures or pinned external provider input, and run affected workspace/selector gates.
A2 template qualification and live publication remain openImplement and validate local functionality first; preserve the roadmap dependency and record final release/cutover evidence before closing A4.
+
+

Migration and rollback

+
    +
  1. Select the upstream source revision, record provenance and inventory the binding, fixture includes, bundle registry, release selectors and parent CLI references. Preserve unrelated work in the sibling checkouts.
    2. +
  2. Move the crate and behavior tests here; establish standalone Cargo and WASM proofs using declared dependencies. Until cutover, the old provider remains the published provider.
    3. +
  3. Add the isolated CPython bridge, versioned Python-model boundary, typed API and CLI. Each group lands its tests and documentation. Qualify clean wheels and an archive-only source build.
    4. +
  4. Build and install the replacement bundle through the canonical morphir extension bundle path using pinned compatible tools. Validate checksum, provenance and claims. Run native, Python and offline MEP round trips for both IR versions.
    5. +
  5. Prepare coordinated morphir-rust removal and finos/morphir provider-reference changes on feature branches. Remove old release registrations and documentation ownership; retain historical releases. Validate the target repositories independently. Do not close A4 because only the destination builds.
    6. +
  6. After maintainer approval, publish qualified native wheels and bundle before releasing dependent tools or promoting ecosystem discovery references. Verify live installation resolves to this repository. Link merged revisions, release receipts and the independent Rust consumer proof in the epic, then close it only after all acceptance conditions hold.
    7. +
+

Rollback before publication means reverting the coordinated commits while the old provider remains available. After publication, retain immutable assets and select the last working provider/version through registry metadata and dependency constraints. Never delete or overwrite historical releases. Merges and real releases require the approval recorded by the repository rules; preparing their concrete artifacts and qualification evidence comes first.

+
+

Acceptance examples

+
# Run from a clean installed environment without Rust on PATH.
+morphir-python compile src --package acme/example --output model.json
+morphir-python ir inspect model.json
+morphir-python generate model.json --output-dir generated
+morphir-python compile generated --package acme/example --output rebuilt.json
+
+# Repeat using draft v4 and the same exposure choices.
+morphir-python compile src --package acme/example --ir-version v4 --output model-v4.json
+morphir-python generate model-v4.json --output-dir generated-v4
+
+

Compare decoded models after narrowly documented normalization, then execute generated fixtures in a separate test process to verify conditional branches, tuples, imports and closures. Add negative checks for unsupported constructs, unchanged existing files and malformed versions. Also install the exact MEP bundle into an isolated extension repository and run the ecosystem CLI with networking unavailable. Confirm the provider URL/provenance names finos/morphir-python and the old Rust directory is absent.

+

No unresolved design choice changes the scope or task breakdown. Exact release versions and approval timing are release inputs; source revisions and toolchain pins are implementation evidence to record before their dependent tasks.

+
+ +
+ + diff --git a/openspec/changes/migrate-python-binding/proposal.md b/openspec/changes/migrate-python-binding/proposal.md new file mode 100644 index 00000000..bba3c88f --- /dev/null +++ b/openspec/changes/migrate-python-binding/proposal.md @@ -0,0 +1,37 @@ +# Proposal + +## Why + +Python compilation and generation currently live in morphir-rust, separate from the Python IR library and CLI. Moving their implementation here lets Python contributors maintain the binding and gives Python callers and CLI users the same compiler without requiring knowledge of Rust or a separate Morphir executable. + +## What Changes + +- Move `morphir-python-binding`, its fixtures, examples, acceptance tests and supported-subset documentation into this repository's Rust half, preserving its native Rust API and MEP extension identity. +- Add an in-process CPython extension, a typed Python compilation/generation API, and prebuilt wheels. Keep `morphir` dependency-free; put compiler integration in the tools distribution and its native dependency. +- Add `morphir-python compile` and `morphir-python generate` for Python source to versioned IR and IR to Python, including multi-module inputs, diagnostics and safe output handling. +- Preserve the existing supported Python subset and IR v3/v4 behavior. Reject unsupported constructs explicitly. Validate the Python/Rust IR boundary against this repository's codecs instead of assuming matching model names imply compatibility. +- Build and qualify the extension bundle here, then cut over ecosystem distribution references. **BREAKING**: remove the old crate, Python bundle ownership and workspace-local fixture references from morphir-rust after replacement qualification. +- Measure compiler and Python integration costs with reproducible benchmarks. Treat a Rust performance advantage as unproven. + +## Capabilities + +### New Capabilities + +- `python/compiler`: Supported Python language mapping, deterministic compilation/generation, typed Python access and diagnostics across IR versions. +- `python/extension`: Independent Rust consumption, MEP compatibility, installable Python wheels and ecosystem extension distribution owned here. +- `tools/python-codegen`: CLI compilation and Python generation through the public Python API. + +### Modified Capabilities + +None. Existing `tools/ir-interchange` requirements continue to govern `ir inspect` and `ir convert`. + +## Beads + +- **Tracked by**: `morphir-ukvn.1.6`, reused as an epic for the existing A4 migration and the Python API/CLI requested here. +- **Related**: `morphir-ukvn.1.3` and `morphir-ukvn.1.3.3`, existing common-layout and release qualification dependencies; `morphir-rust-72s`, existing compiler foundations and broader language backlog; `morphir-nhbd`, prior Python bundle publication; `morphir-ukvn.8`, Python IR compatibility; `morphir-opua.1.4.4`, separate future Python component-host wrapper, not a dependency of the in-process compiler API. + +## Impact + +This repository gains a Rust workspace, a separate native wheel distribution, compiler API and CLI commands, scoped CI and extension release qualification. `morphir` and the current pure-Python tools wheel retain their own namespaces and release tags; installing tools also resolves a compatible native wheel. Native extension and bundle versions are explicit and independent of the Python CLI version. + +Coordinated changes are required in finos/morphir-rust for removal and in finos/morphir for bundle discovery, tests and pinned integration references. Published core/SDK dependencies, or exact committed git revisions where necessary, replace sibling-checkout paths. Real releases, merges and registry promotion remain maintainer-approved; qualification builds and draft changes do not imply publication. diff --git a/openspec/changes/migrate-python-binding/specs/python/compiler/spec.md b/openspec/changes/migrate-python-binding/specs/python/compiler/spec.md new file mode 100644 index 00000000..138377e4 --- /dev/null +++ b/openspec/changes/migrate-python-binding/specs/python/compiler/spec.md @@ -0,0 +1,65 @@ +# Spec Delta + +## Purpose + +Compile supported Python domain models to Morphir IR and generate executable Python through an ordinary typed Python API, sharing behavior with native Rust and MEP callers. + +## ADDED Requirements + +### Requirement: Compile the established Python subset +Compilation SHALL preserve the migrated binding's frozen dataclasses, named unions, fixed tuples, annotated pure functions, scalar literals and comparisons, conditionals, typed calls, unary contextual lambdas, local imports and module exposure behavior. It MUST reject unsupported syntax with structured diagnostics. (bead: morphir-ukvn.1.6) + +#### Scenario: Compile a multi-module model +- **WHEN** a caller submits models and rules modules with imported ADTs, tuple aliases and annotated conditional functions +- **THEN** compilation preserves their module identities, references, declaration order and configured public/private module access + +#### Scenario: Reject an unsupported construct +- **WHEN** a model uses an assignment or an unsupported external import +- **THEN** compilation reports the binding's diagnostic code and available source location without returning a successful partial model + +### Requirement: Source processing is static +Compilation and generation MUST NOT import or execute submitted source or generated Python. (bead: morphir-ukvn.1.6) + +#### Scenario: Input attempts a side effect +- **WHEN** source includes top-level file-writing code +- **THEN** compilation rejects the unsupported construct and no file is written by that source + +### Requirement: Versioned IR interoperates with the Python library +Compilation SHALL produce v3.0 Library IR by default and draft v4 Library IR on request. Generated models MUST decode with this repository's matching codec. Generation SHALL accept supported v3.0 and draft v4 Library models, reject unsupported versions and constructs, and preserve arbitrary-precision integers wherever the selected format permits them. (bead: morphir-ukvn.1.6) + +#### Scenario: Python codec reads compiler output +- **WHEN** a supported model is compiled to each advertised target version +- **THEN** the shared format reader and matching Python codec decode it without losing module, type or value semantics + +#### Scenario: Independently authored IR generates Python +- **WHEN** a Library model using supported constructs comes from the Python codec or an independent ecosystem fixture +- **THEN** generation produces valid Python without requiring that model to originate in this compiler + +#### Scenario: Unsupported distribution or format +- **WHEN** generation receives a Specs distribution or an unsupported format version +- **THEN** it reports an explicit diagnostic rather than inventing value bodies or silently migrating the model + +### Requirement: Typed Python API supports both directions +The Python API SHALL accept in-memory source documents, package identity, exposure and target-version options, and return immutable typed results with a version-specific Python IR model. Generation SHALL accept a supported Python IR model and return immutable relative-path source artifacts. Failures SHALL expose structured diagnostics with code, severity and available source location. (bead: morphir-ukvn.1.6) + +#### Scenario: Python caller compiles and generates +- **WHEN** a caller supplies source strings through the documented Python API and passes the compiled model to generation +- **THEN** both operations work in process without MEP envelopes, Rust types, temporary files, or a separate executable in the caller's code + +#### Scenario: Warning accompanies success +- **WHEN** the compiler returns valid IR with a warning +- **THEN** the result retains the warning and its severity separately from successful model data + +### Requirement: Round trips preserve supported semantics +For supported models, compile/generate/recompile SHALL preserve normalized module access, references, type definitions and value behavior in each target IR version. Source formatting and equivalent expression spelling need not be preserved. (bead: morphir-ukvn.1.6) + +#### Scenario: Conditional and higher-order behavior survives generation +- **WHEN** a model using conditional branches, a typed call and a unary closure is generated and recompiled +- **THEN** normalized IR remains equivalent and executing the generated code in a separate verification process produces the expected results + +### Requirement: Source document order does not affect results +Compilation SHALL produce equivalent models and diagnostics for permutations of the same source set, with deterministic artifact ordering during generation. (bead: morphir-ukvn.1.6) + +#### Scenario: Reordered modules +- **WHEN** callers compile the same dependent source modules in different orders +- **THEN** normalized models and diagnostic ordering agree, and subsequent generation returns the same ordered artifacts diff --git a/openspec/changes/migrate-python-binding/specs/python/extension/spec.md b/openspec/changes/migrate-python-binding/specs/python/extension/spec.md new file mode 100644 index 00000000..530f9d08 --- /dev/null +++ b/openspec/changes/migrate-python-binding/specs/python/extension/spec.md @@ -0,0 +1,53 @@ +# Spec Delta + +## Purpose + +Distribute the Python language compiler from morphir-python for native Rust consumers, ordinary Python installations and Morphir extension hosts, with one maintained implementation. + +## ADDED Requirements + +### Requirement: Rust consumers use an independently buildable crate +The migrated crate SHALL retain the `morphir-python-binding` package name and native `PythonExtension` frontend/backend API. A Rust consumer MUST be able to build and invoke it without a morphir-rust sibling checkout, CPython or Python extension linking. (bead: morphir-ukvn.1.6) + +#### Scenario: External Cargo consumer +- **WHEN** a standalone Cargo project depends on the binding at an exact committed revision in morphir-python +- **THEN** it builds and invokes compile and generate using declared core/SDK dependencies without local source overrides or Python development libraries + +### Requirement: MEP behavior remains compatible +The extension SHALL retain identity `morphir-python`, Python language/target identifiers, initialization and capability negotiation, compile/generate methods and advertised IR v3/v4 support. Native Rust, Python and packaged WASM invocations SHALL agree on supported model semantics and language diagnostic codes. (bead: morphir-ukvn.1.6) + +#### Scenario: Existing host installs the replacement bundle +- **WHEN** a compatible Morphir host installs the bundle built from morphir-python +- **THEN** initialization, capability discovery, Python compilation and Python generation succeed through the existing protocol + +### Requirement: Python installation hides Rust implementation details +Installing `morphir-python` on qualified CPython platforms SHALL install its compatible native compiler dependency automatically from wheels. Compilation and generation MUST require neither a Rust toolchain nor a separately installed Morphir CLI at runtime. The `morphir` distribution SHALL keep zero third-party runtime dependencies. (bead: morphir-ukvn.1.6) + +#### Scenario: Clean wheel-only installation +- **WHEN** a user installs the qualified distributions into a clean CPython 3.14 environment on a supported wheel platform with Rust absent +- **THEN** Python API compilation and generation and both new CLI commands succeed + +#### Scenario: Library-only user +- **WHEN** a user installs only `morphir` +- **THEN** IR and SDK imports work without the compiler dependency or native shared libraries + +### Requirement: Qualified artifacts are self-contained +Native wheels and source distributions SHALL include required runtime files, typing information and build inputs. Published artifact metadata SHALL declare supported interpreter/platform combinations and compatible package versions. Qualification MUST install built artifacts outside the source checkout. (bead: morphir-ukvn.1.6) + +#### Scenario: Build from the source archive +- **WHEN** a contributor builds the native wheel from its source archive with the documented build toolchain +- **THEN** the build succeeds without missing workspace files or sibling repositories and the wheel passes an installed API smoke test + +### Requirement: Python extension releases belong to morphir-python +Replacement Python extension bundles SHALL be built, qualified and released from finos/morphir-python with checksums, capability claims and source provenance. Ecosystem installation references SHALL resolve the replacement provider there; historical release assets MUST remain available. (bead: morphir-ukvn.1.6) + +#### Scenario: Ecosystem installation follows the new owner +- **WHEN** a qualified replacement is published and selected in the ecosystem extension repository +- **THEN** `morphir extension install morphir-python` obtains its bundle from morphir-python, verifies it and supports offline compile/generate/recompile + +### Requirement: Old implementation is removed after replacement qualification +The cutover SHALL remove the Python binding implementation, workspace membership and Python-specific publishing ownership from morphir-rust. Tests and integrations MUST use the new provider or independent host fixtures without depending on the deleted directory. (bead: morphir-ukvn.1.6) + +#### Scenario: Rust workspace after removal +- **WHEN** the replacement provider has passed migration qualification and the coordinated removal lands +- **THEN** the morphir-rust workspace and its CI remain valid with no Python compiler implementation or references requiring the former crate directory diff --git a/openspec/changes/migrate-python-binding/specs/tools/python-codegen/spec.md b/openspec/changes/migrate-python-binding/specs/tools/python-codegen/spec.md new file mode 100644 index 00000000..3d0ee305 --- /dev/null +++ b/openspec/changes/migrate-python-binding/specs/tools/python-codegen/spec.md @@ -0,0 +1,69 @@ +# Spec Delta + +## Purpose + +Let morphir-python CLI users compile supported Python models into versioned Morphir IR and generate Python packages from supported IR using the same Python API as application callers. + +## ADDED Requirements + +### Requirement: Compile Python sources from the CLI +`morphir-python compile SOURCE --package PACKAGE --output FILE` SHALL compile one Python file or a directory source tree through the public Python API. It SHALL default to IR v3 and support `--ir-version v4` and repeated `--expose MODULE` selections. Omitting exposure SHALL expose every module. Output SHALL be a versioned IR document readable by `ir inspect`. (bead: morphir-ukvn.1.6) + +#### Scenario: Compile a source directory +- **WHEN** the user compiles a directory containing supported models and locally importing rules modules +- **THEN** the tool writes one versioned Library document with correct module identities and requested exposure, and exits zero + +#### Scenario: Choose draft v4 +- **WHEN** the user requests `--ir-version v4` +- **THEN** the tool writes draft v4 and reports its draft status + +### Requirement: Generate Python from versioned IR +`morphir-python generate FILE --output-dir DIRECTORY` SHALL read IR with the shared format reader and matching Python codec, then call the public generation API. It SHALL write all generated relative-path Python modules while preserving their directory structure. (bead: morphir-ukvn.1.6) + +#### Scenario: Generate modules with imports +- **WHEN** a supported Library contains models and rules in nested modules +- **THEN** generation produces importable modules with the expected paths and supported behavior + +#### Scenario: Version parsing uses the shared contract +- **WHEN** an input has a malformed, missing or unsupported format version +- **THEN** generation reports the same version diagnostic code as the Python library and exits non-zero + +### Requirement: Report failures and warnings consistently +Both commands SHALL preserve Python API diagnostic codes, severities and available locations. Failures SHALL exit non-zero, with diagnostics on stderr. Success warnings SHALL remain distinguishable from errors and SHALL NOT cause a non-zero exit. (bead: morphir-ukvn.1.6) + +#### Scenario: Invalid source +- **WHEN** compilation encounters a supported parser error or unsupported construct +- **THEN** stderr identifies its code and available source location, and no successful IR output is written + +#### Scenario: Unsupported IR construct +- **WHEN** generation receives an unsupported value form +- **THEN** it reports the API diagnostic and writes no generated modules + +### Requirement: Output publication preserves existing user data +Commands MUST finish validation before publishing output. Compile failures MUST preserve an existing output file. Generation SHALL require a nonexistent or empty output directory, reject escaping or colliding artifact paths, and stage all files before publishing them. Failures MUST leave no partial generated output or overwritten user files. (bead: morphir-ukvn.1.6) + +#### Scenario: Existing generation directory contains files +- **WHEN** the user selects a nonempty output directory +- **THEN** the command fails clearly and preserves every existing file + +#### Scenario: Generation fails on a later module +- **WHEN** generation cannot handle one module after processing another +- **THEN** no final output module is published + +#### Scenario: Artifact path escapes the selected directory +- **WHEN** an artifact path is absolute, traverses a parent directory, or would traverse a symlink +- **THEN** the command rejects the artifact set before writing any output + +### Requirement: Existing IR commands remain independent +`ir inspect` and `ir convert` SHALL retain their existing options, shared library behavior and results without initializing the Python compiler. (bead: morphir-ukvn.1.6) + +#### Scenario: Inspect and convert an existing fixture +- **WHEN** the user runs either existing command after installing the updated tools +- **THEN** its existing compatibility and diagnostic tests continue to pass + +### Requirement: CLI supports a complete round trip +Generated Python in the supported subset SHALL be accepted by `compile` when supplied the original package and exposure options, yielding equivalent normalized IR in the chosen target version. (bead: morphir-ukvn.1.6) + +#### Scenario: Compile generate recompile +- **WHEN** the user compiles a supported multi-module model, generates Python, then recompiles that output with the same package, exposure and IR version +- **THEN** both IR documents describe equivalent supported definitions and generated Python returns the expected results diff --git a/openspec/changes/migrate-python-binding/tasks.md b/openspec/changes/migrate-python-binding/tasks.md new file mode 100644 index 00000000..0194fda9 --- /dev/null +++ b/openspec/changes/migrate-python-binding/tasks.md @@ -0,0 +1,56 @@ +# Tasks + +Implement against `design.html` and the three delta specs. Follow TDD within each behavior group; claim its bead before work and close it only with passing checks and commit evidence. Keep the A2 final-delivery dependency on the epic. Actual publication and live provider promotion require maintainer approval and are listed under workflow follow-up. + +## 1. Move and qualify the standalone Rust compiler (morphir-ukvn.1.6.1) + +- [ ] 1.1 Select and record the current upstream morphir-rust source revision, reconcile changes since inspected `f4298d0`, and inventory the crate, fixture consumers, release selectors and parent CLI provider references; verify the recorded file inventory and existing binding tests against that revision. +- [ ] 1.2 Move the compiler, examples, fixtures, acceptance tests and support documentation into `rust/crates/morphir-python-binding`; add the Rust workspace and lockfile using resolvable published core/SDK versions or one documented exact git revision; verify `cargo metadata --locked` and absence of sibling-checkout dependencies. +- [ ] 1.3 Preserve the native Rust API, MEP identity, Ruff pins and current v3/v4 subset; verify migrated unit/integration, Gherkin acceptance and doc tests, Cargo formatting/clippy, and a release WASM build without CPython linking. +- [ ] 1.4 Add the documented external Rust usage example and build prerequisites; verify an independent temporary Cargo consumer can compile and generate through the declared dependency without morphir-rust source paths or Python development libraries. + +## 2. Expose the compiler through a typed in-process Python API (morphir-ukvn.1.6.2) + +- [ ] 2.1 Add failing public API tests for in-memory documents, package/exposure options, v3 default and explicit v4, diagnostics and generated artifacts; implement the isolated `morphir-python-native` PyO3 bridge calling the existing compiler traits; verify the tests and that the compiler-only native/WASM builds still omit PyO3. +- [ ] 2.2 Add frozen result/source/artifact types and `morphir_tools.python.compile_sources` and `generate_python`; decode and encode only with existing versioned Python library APIs; verify strict mypy/pyright checks on the new public API and result states that cannot combine failure with successful model data. +- [ ] 2.3 Add cross-codec tests for ADTs, nested tuples, imports, visibility, unary higher-order functions, arbitrary integers and independently authored IR in both versions; verify normalized round trips, deterministic source permutations and explicit unsupported model/attribute diagnostics without silent migration. +- [ ] 2.4 Preserve language diagnostic codes, severities and available source locations, and document API examples and subset limits; verify warnings survive success, failure has no partial model, submitted code never executes, and documentation examples run through the public API. + +## 3. Package the native compiler for Python users (morphir-ukvn.1.6.3) + +- [ ] 3.1 Add the Maturin native distribution with a qualified pinned PyO3/Maturin pair, private module, stubs, `py.typed`, licenses and explicit compatible versions; verify wheel contents and package metadata while both existing distributions remain on Hatch. +- [ ] 3.2 Add the tools dependency and uv workspace wiring, keeping core dependencies empty and compiler loading lazy; verify library-only installation and existing `ir inspect`/`ir convert` tests without initializing the native compiler. +- [ ] 3.3 Produce the native source archive with all referenced Rust workspace files; verify building and installing its wheel in a clean directory with no source checkout or sibling repository available, then run an installed API round trip. +- [ ] 3.4 Document the qualified CPython/platform matrix, wheel-only installation and contributor source builds; verify clean wheelhouse installation of core, native and tools with Rust absent from PATH and record the exact artifacts used. + +## 4. Add Python compile and generate CLI commands (morphir-ukvn.1.6.4) + +- [ ] 4.1 Add compile tests and implement file/directory discovery, required package, repeated exposure selections and version options through the public API; verify deterministic module identities, local import resolution and `ir inspect` on written v3/v4 models. +- [ ] 4.2 Add generate tests and implement shared IR decoding plus API generation into an absent/empty output directory; verify nested modules, importable output, unsupported distribution/node diagnostics and parity with API codes. +- [ ] 4.3 Stage complete output before publication and render I/O failures consistently; verify injected write failures, late-module rejection, preservation of existing compile files, refusal of nonempty directories and artifact path/collision/symlink checks. +- [ ] 4.4 Add behave scenarios and user-facing CLI/API documentation for compile/generate/recompile; verify both versions, visibility settings and executable conditional/tuple/closure behavior in a separate process, plus all existing CLI regression tests. + +## 5. Qualify scoped CI and independently owned extension bundles (morphir-ukvn.1.6.5) + +- [ ] 5.1 Add mise tasks and scoped Python, Rust/native and bundle checks following the current common-layout contract; verify selector tests for documentation-only, native-only, compiler, shared dependency and workflow changes, plus aggregate failure/cancellation/missing-result handling. +- [ ] 5.2 Build one compatible artifact per target/ABI and reuse it for clean-install tests; verify no consumer rebuilds its input and preserve existing lint/typecheck/coverage/BDD/release checks, with runtime and cache observations recorded rather than estimated savings. +- [ ] 5.3 Build and package the WASM extension here with the canonical `morphir extension bundle` command against pinned tools; verify claims, checksums, source provenance and initialization/compile/generate/recompile through an isolated offline MEP host for both IR versions. +- [ ] 5.4 Add nonpublishing release qualification for the advertised wheel platforms, source archive and extension bundle, independent version/tag validation and exact-artifact promotion inputs; verify workflow syntax and refusal of failed/skipped/missing/mismatched candidate evidence, then document the approval and native-before-tools publication order. + +## 6. Measure native and Python integration costs (morphir-ukvn.1.6.6) + +- [ ] 6.1 Add a reproducible benchmark command with fixed fixtures and scale parameters, comparing native Rust, bridge, public API and CLI at matched target versions; verify warmup/repetition metadata, separate startup and JSON/codec costs, and recorded toolchains/build profile/input sizes. +- [ ] 6.2 Run release-build samples and record medians/spread in an HTML report with raw results in the local output directory; verify conclusions distinguish parsing-only `ast.parse` measurements from full compilation and make no Rust-versus-Python compiler claim without an equivalent implementation. + +## 7. Remove the old provider and prepare ecosystem cutover (morphir-ukvn.1.6.7) + +- [ ] 7.1 Prepare a coordinated morphir-rust feature branch removing the crate, workspace membership and Python release/CI registrations; move compiler-owned tests here and replace former-directory fixture includes with independent host fixtures or a pinned replacement provider; verify affected workspace tests, CI selector tests and no references that require the deleted directory. +- [ ] 7.2 Prepare finos/morphir provider/discovery, provenance, integration fixture and pin updates as required by the current registry implementation; verify the isolated extension install selects the bundle built here and offline ecosystem CLI round trips pass without the old Python crate. +- [ ] 7.3 Run final repository quality gates, installed Python API/CLI proofs and standalone Rust/native/WASM proofs for the coordinated revisions; verify recorded model equivalence, artifact digests and target-repository checks, and preserve unrelated sibling-checkout work. +- [ ] 7.4 Record reviewable commit ranges, qualification receipts, rollback steps and outstanding A2/release dependencies in beads; verify all implementation task evidence is linked and the epic remains open until approved publication and live provider cutover have actually satisfied its acceptance conditions. + +## Workflow follow-up + +- Present the exact qualified artifacts and coordinated repository changes for maintainer release/merge approval. Publish the native dependency and extension bundle before dependent tools or registry promotion; preserve historical releases. +- Verify live `morphir extension install morphir-python` fetches from finos/morphir-python and the published Python packages install and round trip without Rust. Record release URLs, digests and merged revisions before closing A4 and relink relevant compiler backlog work to this repository. +- Archive this OpenSpec change after implementation and the applicable review process; adjust HTML asset paths for the archive location, update the epic spec id and sync beads. From 55ced4694a552461ba42615f8ed5f29276261205 Mon Sep 17 00:00:00 2001 From: Damian Reeves <957246+DamianReeves@users.noreply.github.com> Date: Tue, 27 Oct 2026 12:37:11 -0500 Subject: [PATCH 02/31] docs: ground Python bridge in community examples --- .../changes/migrate-python-binding/design.html | 17 ++++++++++++++--- .../changes/migrate-python-binding/proposal.md | 3 ++- .../changes/migrate-python-binding/tasks.md | 12 ++++++------ 3 files changed, 22 insertions(+), 10 deletions(-) diff --git a/openspec/changes/migrate-python-binding/design.html b/openspec/changes/migrate-python-binding/design.html index 0214c351..388b0d3e 100644 --- a/openspec/changes/migrate-python-binding/design.html +++ b/openspec/changes/migrate-python-binding/design.html @@ -26,6 +26,7 @@

Migrate the Python binding