diff --git a/.claude/rules/docs.md b/.claude/rules/docs.md index 0cca94b10b..27ee15f982 100644 --- a/.claude/rules/docs.md +++ b/.claude/rules/docs.md @@ -13,4 +13,4 @@ paths: - Generated pages and regions under `docs/reference/` come from their generators (`settings_reference.py`, `docs_reference.py`); never hand-edit them. - Relative links and `#anchors` must resolve (`test_doc_links.py`). - Follow the [documentation style guide](../../docs/development/documentation-style.md). -- `i18n//` holds machine translations of the README, the quickstart and the use-case pages, generated by `scripts/i18n/translate.py` after an English edit lands on main (`.github/workflows/i18n.yml`); never edit them by hand, and edit the English page instead. A new page in that set goes into `SOURCES`, and the PR that adds it commits a translation for each entry in `LANGUAGES`, made with `python -m scripts.i18n.translate update` and AWS credentials that can invoke the model on Bedrock ([translations](../../CONTRIBUTING.md#translations); `test_i18n.py` requires every page); after a change to the prompt, a banner or the model, run the workflow with `all` (a new model needs the runner role's policy changed first). +- `i18n//` contains machine translations of tracked `README.md` and `docs/**/*.md`, including generated references. Edit English or its generator, never translations by hand. `.github/workflows/i18n.yml` queues bounded batches after English merges to main; ordinary PRs may have pending coverage. `python -m scripts.i18n.translate check` reports missing, stale and obsolete output; `check --skip-stale` still validates present files and links. Verified replacements wait in `i18n/.pending//` until their link dependencies permit publication. The updater reuses validated generated data from `i18n/update`, never branch code, and deletes only stamped obsolete output after inbound links are repaired. Paid calls require an explicit cumulative allowance and durable reservations ([translations](../../CONTRIBUTING.md#translations)). After a prompt change, request `update --all`; another model also needs the runner role's policy changed. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index dd6e154d42..8480bd9696 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -6,6 +6,12 @@ on: branches: [main, "release/**"] pull_request: branches: [main, "release/**"] + workflow_dispatch: + inputs: + expected_sha: + description: Exact generated translation commit to validate + required: true + type: string # A new push to a pull request cancels its superseded run. Every push (to main # or a release/X.Y line) runs to completion in its own group, so a failure on @@ -25,7 +31,26 @@ env: UV_VERSION: "0.12.19" jobs: + validate-ref: + name: validate requested commit + runs-on: ubuntu-latest + timeout-minutes: 2 + steps: + - name: Bind a dispatched run to its generated commit + env: + EVENT: ${{ github.event_name }} + REF: ${{ github.ref }} + SHA: ${{ github.sha }} + EXPECTED_SHA: ${{ inputs.expected_sha }} + run: | + if [ "$EVENT" = "workflow_dispatch" ]; then + test "$REF" = "refs/heads/i18n/update" + [[ "$EXPECTED_SHA" =~ ^[0-9a-f]{40}$ ]] + test "$SHA" = "$EXPECTED_SHA" + fi + clickhouse: + needs: validate-ref name: the ClickHouse release this run tests runs-on: ubuntu-latest timeout-minutes: 10 @@ -43,6 +68,7 @@ jobs: echo "image=$image" >> "$GITHUB_OUTPUT" lint: + needs: validate-ref name: ruff + mypy-gate runs-on: ubuntu-latest timeout-minutes: 15 @@ -112,7 +138,7 @@ jobs: # coverage data and applies the floors. unit: name: pytest unit part ${{ matrix.part }} - needs: clickhouse + needs: [validate-ref, clickhouse] runs-on: ubuntu-latest timeout-minutes: 30 strategy: @@ -186,7 +212,7 @@ jobs: unit-coverage: name: pytest unit coverage, floors and bench - needs: unit + needs: [validate-ref, unit] runs-on: ubuntu-latest timeout-minutes: 15 steps: @@ -244,6 +270,7 @@ jobs: path: coverage.xml sandbox: + needs: validate-ref name: repository-job sandbox (real bubblewrap) runs-on: ubuntu-24.04 timeout-minutes: 15 @@ -279,7 +306,7 @@ jobs: schema-check: name: schema.sql in sync with migrations - needs: clickhouse + needs: [validate-ref, clickhouse] runs-on: ubuntu-latest timeout-minutes: 15 env: @@ -318,6 +345,7 @@ jobs: run: make schema-check ts-typecheck: + needs: validate-ref name: tsc (frontend) runs-on: ubuntu-latest timeout-minutes: 15 @@ -340,6 +368,7 @@ jobs: run: npm run budget format: + needs: validate-ref name: format (biome, prettier, taplo, shfmt) runs-on: ubuntu-latest timeout-minutes: 15 @@ -366,6 +395,7 @@ jobs: run: git ls-files '*.sh' ':!tests/fixtures' | xargs shfmt -d helm: + needs: validate-ref name: helm chart (lint, kubeconform) runs-on: ubuntu-latest timeout-minutes: 10 @@ -390,7 +420,7 @@ jobs: assistant=(--set assistant.enabled=true --set assistant.existingSecret=ci) # TLS on Loom's port, which the assistant (in "all") cannot reach: its own render. tls=(--set tls.secretName=ci --set 'ingress.annotations.alb\.ingress\.kubernetes\.io/backend-protocol=HTTPS' --set workers.enabled=true --set networkPolicy.enabled=true "${on[@]}" "${extractor[@]}") - all=(--set workers.enabled=true --set workers.replicaCount=2 --set autoscaling.enabled=true --set networkPolicy.enabled=true --set alerts.prometheusRule.enabled=true --set mustMatchChange.publish=ci --set settingsFile.memory_ask_recency_weight=0.3 "${on[@]}" "${extractor[@]}" "${assistant[@]}") + all=(--set workers.enabled=true --set workers.replicaCount=2 --set autoscaling.enabled=true --set networkPolicy.enabled=true --set alerts.prometheusRule.enabled=true --set mustMatchChange.publish=ci --set cellProbe.port=8777 --set settingsFile.memory_ask_recency_weight=0.3 "${on[@]}" "${extractor[@]}" "${assistant[@]}") # The PrometheusRule's schema: the CRD catalog at a pinned commit. crd='https://raw.githubusercontent.com/datreeio/CRDs-catalog/c86448cbf1a840ca62d119259864863897c8748a/{{.Group}}/{{.ResourceKind}}_{{.ResourceAPIVersion}}.json' @@ -465,8 +495,11 @@ jobs: # TLS on Loom's port: the Secret's mount, the settings, HTTPS probes, and the refusals. - name: tls run: python3 -m scripts.gates.helm_tls + - name: private cell probe + run: python3 -m scripts.gates.helm_cell_probe agent-plugins: + needs: validate-ref name: coding-agent plugins (node --test) runs-on: ubuntu-latest timeout-minutes: 15 @@ -498,6 +531,7 @@ jobs: # The dashboard's vitest and Playwright suites share nothing but `npm ci`, so # each runs as its own job, split into parts on separate runners. frontend-vitest: + needs: validate-ref name: frontend vitest part ${{ matrix.part }} runs-on: ubuntu-latest timeout-minutes: 20 @@ -524,6 +558,7 @@ jobs: # With fullyParallel: false, --shard splits by spec file per project, not by test. frontend-e2e: + needs: validate-ref name: frontend playwright part ${{ matrix.part }} runs-on: ubuntu-latest timeout-minutes: 20 @@ -566,7 +601,7 @@ jobs: # analyzer corpus. integration: name: pytest integration ${{ matrix.part == 'isolation' && 'isolation gates' || format('part {0} of 4', matrix.part) }} - needs: clickhouse + needs: [validate-ref, clickhouse] runs-on: ubuntu-latest timeout-minutes: 45 strategy: @@ -721,13 +756,16 @@ jobs: done curl -sf http://127.0.0.1:5000/ >/dev/null echo "LOOM_TEST_S3_ENDPOINT=http://127.0.0.1:5000" >> "$GITHUB_ENV" - # The whole directory on every pull request, split over the parts; no test - # is slow enough to justify a separate merge-only tier. + # Split ordinary integration cases over the parts. Signed mirror HTTP cases + # run in the authenticated isolation step below on every pull request. - name: pytest (this part of tests/integration) if: matrix.part != 'isolation' env: LOOM_TEST_SHARD: ${{ matrix.part }}/4 - run: pytest tests/integration -q --durations=20 + run: >- + pytest tests/integration -q --durations=20 + --ignore=tests/integration/test_mirror_checked_search_http.py + --ignore=tests/integration/test_mirror_checked_list_http.py - name: Native Google ADK live tests (Python and TypeScript) if: matrix.part == '1' run: python -m tests.integration._google_adk_ci "$RUNNER_TEMP/loom-adk-client" @@ -777,6 +815,8 @@ jobs: tests/integration/test_run_begin_claims.py tests/integration/test_mcp_session_run_scope.py tests/integration/test_projection_failures_restricted.py + tests/integration/test_mirror_checked_search_http.py + tests/integration/test_mirror_checked_list_http.py # The statement corpus: every builder's shapes through # EXPLAIN QUERY TREE on the migrated service, and shapes.txt in step # with the builders. ~10 s; a rejection here is a statement that would @@ -796,7 +836,7 @@ jobs: integration-postgres: name: pytest integration (Postgres ${{ matrix.postgres }} memory authority) - needs: clickhouse + needs: [validate-ref, clickhouse] runs-on: ubuntu-latest timeout-minutes: 60 # The oldest supported major and the newest. @@ -929,6 +969,23 @@ jobs: tests/integration/test_cell_unset_gate_pg.py tests/integration/test_cell_prepare.py tests/integration/test_cell_org_triggers_pg.py + tests/integration/test_source_access_store.py + tests/integration/test_source_document_store.py + tests/integration/test_mirror_reader_evidence.py + tests/integration/test_mirror_session_latch_native.py + tests/integration/test_mirror_search_chunk_keys.py + tests/integration/test_mirror_title_projection.py + tests/integration/test_mirror_ask_http.py + tests/integration/test_marked_admin_events_native.py + tests/integration/test_marked_analytics_export_native.py + tests/integration/test_marked_assistant_ledger_native.py + tests/integration/test_marked_cost_summary_native.py + tests/integration/test_marked_derived_activity_native.py + tests/integration/test_marked_event_readers_native.py + tests/integration/test_marked_mentor_agents_native.py + tests/integration/test_marked_rollup_migration_native.py + tests/integration/test_marked_durable_folds_native.py + tests/integration/test_marked_sql_surface_native.py # The tenant-isolation gates again, against a second server that enforces auth: # the Postgres read paths (heads by key and by id, narrowed scopes) see real tokens. # They run whether or not the memory suites passed (`!cancelled()`). @@ -938,6 +995,7 @@ jobs: env: LOOM_AUTH_REQUIRED: "true" LOOM_INSTANCE_LABEL: "ci:integration-postgres-auth" + LOOM_CH_ANALYSIS_PROBE_INTERVAL_S: "10" run: | LOOM_WEB_WORKERS=3 PORT=7778 nohup sh scripts/serve.sh > server-auth.log 2>&1 & for _ in $(seq 1 60); do @@ -957,6 +1015,7 @@ jobs: env: LOOM_AUTH_REQUIRED: "true" LOOM_BASE_URL: http://localhost:7778 + LOOM_CH_ANALYSIS_PROBE_INTERVAL_S: "10" run: >- pytest -q tests/integration/test_cross_principal_leakage.py @@ -967,6 +1026,9 @@ jobs: tests/integration/test_mcp_session_run_scope.py tests/integration/test_onboard_tenant.py tests/integration/test_blacklist.py + tests/integration/test_mirror_checked_search_http.py + tests/integration/test_mirror_checked_list_http.py + tests/integration/test_analysis_isolation.py tests/integration/test_sensitivity_high_canary.py tests/integration/test_pg_authority.py -k "not test_a_duplicate_never_resolves_to_a_canonical_outside_the_scope @@ -986,9 +1048,12 @@ jobs: server-auth.log mixed-version: + needs: [validate-ref, clickhouse] name: builds N-1 and N on one ClickHouse runs-on: ubuntu-latest timeout-minutes: 30 + env: + LOOM_CLICKHOUSE_IMAGE: ${{ needs.clickhouse.outputs.image }} steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: @@ -1018,8 +1083,11 @@ jobs: # target branch's tip can carry a newer migration by the time the run starts. - name: Mixed-version gate run: python scripts/gates/mixed_version.py --require --base HEAD~1 + - name: Published build ordinary-event compatibility + run: python scripts/gates/mixed_version.py --require --base 1cbaf0411114512957b96ec5ea7bbb8058828543 sdk-typescript: + needs: validate-ref name: typescript sdk (generated types in sync, lint, tsc, vitest, dist) runs-on: ubuntu-latest timeout-minutes: 15 @@ -1064,6 +1132,7 @@ jobs: run: npm run check:dist sdk-python-adapters: + needs: validate-ref name: python sdk adapters (py ${{ matrix.python }}${{ matrix.lowest && ', lowest deps' || '' }}) runs-on: ubuntu-latest timeout-minutes: 15 @@ -1091,6 +1160,7 @@ jobs: run: make test-sdk-adapters SDK_ADAPTERS_PYTHON="$(which python)" SDK_ADAPTERS_RESOLUTION="${{ matrix.lowest && 'lowest-direct' || 'highest' }}" sdk-python-package: + needs: validate-ref name: python sdk wheel (py ${{ matrix.python }}${{ matrix.lowest && ', lowest deps' || '' }}) runs-on: ubuntu-latest timeout-minutes: 15 diff --git a/.github/workflows/docs-site.yml b/.github/workflows/docs-site.yml index eb81d4e88e..ff35d3117f 100644 --- a/.github/workflows/docs-site.yml +++ b/.github/workflows/docs-site.yml @@ -12,6 +12,7 @@ on: branches: [main] paths: - "docs/**" + - "i18n/**" - "mkdocs.yml" - "llms.txt" - "llms-install.md" diff --git a/.github/workflows/i18n.yml b/.github/workflows/i18n.yml index 4b1954769c..9e61a83ada 100644 --- a/.github/workflows/i18n.yml +++ b/.github/workflows/i18n.yml @@ -1,25 +1,24 @@ name: translations -# Keeps i18n// in step with the English pages it translates. On a push to main that -# changes the README or a page under docs/, and on a manual run, it translates the stale -# pages with scripts/i18n/translate.py and opens or updates one pull request from the -# branch i18n/update. After a change to the prompt, a banner or the model, run it by hand -# with "all" to translate every page again. A pull request opened with the workflow token -# starts no other workflow: close and reopen it to run CI. Needs "Allow GitHub Actions to -# create and approve pull requests" (Settings → Actions → General). -# -# Claude is called on Amazon Bedrock with no credential stored in GitHub: the job runs on a -# self-hosted runner in AWS whose instance role may invoke one model, and the SDK signs with -# that role. The repository variable LOOM_I18N_RUNNER holds the runner's labels as JSON; -# while it is empty the job is skipped. Only a push to main or a manual run reaches the -# runner, never a pull request. The runner group and the role are described in -# CONTRIBUTING.md#translations. +# Main supplies every executable byte. The pending branch contributes generated data +# only. Hosted CI is dispatched explicitly because GITHUB_TOKEN PRs suppress events. on: push: branches: [main] paths: - "README.md" - "docs/**" + - "scripts/i18n/**" + - "scripts/docs_site/messages.py" + - "scripts/docs_site/indexnow.py" + - "scripts/docs_site/paths.py" + - ".github/workflows/i18n.yml" + workflow_run: + workflows: [ci] + types: [completed] + branches: [i18n/update] + schedule: + - cron: "37 */6 * * *" workflow_dispatch: inputs: all: @@ -27,7 +26,7 @@ on: type: boolean default: false languages: - description: Language codes to translate, separated by spaces (empty for all) + description: Language codes separated by spaces (empty for all five or configured subset) type: string default: "" @@ -41,87 +40,78 @@ concurrency: jobs: update: name: update - if: github.ref == 'refs/heads/main' && vars.LOOM_I18N_RUNNER != '' + if: github.ref == 'refs/heads/main' && contains(fromJSON('["push","schedule","workflow_dispatch"]'), github.event_name) && vars.LOOM_I18N_RUNNER != '' runs-on: ${{ fromJSON(vars.LOOM_I18N_RUNNER) }} timeout-minutes: 45 - # Limited to main (Settings → Environments); it records each run. What keeps other - # workflows off the runner is the runner group's selected-workflow rule. + # Environment and runner-group restrictions admit this main-only workflow. environment: i18n permissions: + actions: write contents: write pull-requests: write + env: + LOOM_I18N_LANGUAGES: ${{ vars.LOOM_I18N_LANGUAGES }} + LOOM_I18N_BATCH_LANGUAGES: ${{ inputs.languages }} defaults: run: - # Explicit bash runs with -o pipefail, so a failed translation fails its step. shell: bash steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: ${{ github.sha }} + fetch-depth: 0 - uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0 with: python-version: "3.12" - - name: Find stale translations - id: check - env: - ALL: ${{ inputs.all }} - run: | - if [ "$ALL" != "true" ] && python -m scripts.i18n.translate check; then - echo "stale=false" >> "$GITHUB_OUTPUT" - else - echo "stale=true" >> "$GITHUB_OUTPUT" - fi - - name: Install the Bedrock client - if: steps.check.outputs.stale == 'true' + - name: Install translation runtime # A virtualenv in the job's temp directory: a self-hosted runner keeps its Python. run: | python -m venv "$RUNNER_TEMP/i18n" "$RUNNER_TEMP/i18n/bin/python" -m pip install --quiet --require-hashes --no-deps \ --only-binary=:all: -r scripts/i18n/requirements.txt echo "$RUNNER_TEMP/i18n/bin" >> "$GITHUB_PATH" - - name: Translate + - name: Preserve verified pending pages + id: prepare + run: python -m scripts.i18n.automation prepare + - name: Translate a bounded batch id: translate - if: steps.check.outputs.stale == 'true' env: - LOOM_I18N_MAX_USD: "5" - ALL: ${{ inputs.all && '--all' || '' }} - LANGUAGES: ${{ inputs.languages }} + LOOM_I18N_TOTAL_USD: ${{ vars.LOOM_I18N_TOTAL_USD || '0' }} + ALL: ${{ inputs.all }} run: | - # The runner's region, or us-east-1 when it sets none; the role's policy names it. export AWS_REGION="${AWS_REGION:-${AWS_DEFAULT_REGION:-us-east-1}}" - args=($ALL) - for code in $LANGUAGES; do args+=(--lang "$code"); done - if python -m scripts.i18n.translate update "${args[@]}" 2>&1 | tee -a "$GITHUB_STEP_SUMMARY"; then - echo "ok=true" >> "$GITHUB_OUTPUT" - else - echo "ok=false" >> "$GITHUB_OUTPUT" - fi - # The pages written passed the verifier; links, anchors and labels across i18n/ - # must hold before anything is pushed. - python -m scripts.i18n.translate check --skip-stale - - name: Open or update the pull request - if: steps.check.outputs.stale == 'true' + args=() + if [ "$ALL" = "true" ]; then args+=(--all); fi + python -m scripts.i18n.automation update "${args[@]}" 2>&1 | tee -a "$GITHUB_STEP_SUMMARY" + - name: Validate and publish for hosted CI + if: ${{ !cancelled() && steps.prepare.outcome == 'success' && steps.translate.outcome != 'skipped' }} env: GH_TOKEN: ${{ github.token }} - SOURCE_SHA: ${{ github.sha }} - run: | - git config user.name "github-actions[bot]" - git config user.email "41898282+github-actions[bot]@users.noreply.github.com" - git switch -C i18n/update - git add README.md i18n - if git diff --cached --quiet; then - echo "no page was translated" - exit 0 - fi - git commit -s -q -m "Translations follow ${SOURCE_SHA::9}" - git push --force origin i18n/update - title="Translations follow ${SOURCE_SHA::9}" - body="Machine translations of the pages changed up to ${SOURCE_SHA}, generated by \`scripts/i18n/translate.py\`. Close and reopen this pull request to run CI." - if [ -z "$(gh pr list --head i18n/update --state open --json number --jq '.[].number')" ]; then - gh pr create --head i18n/update --base main --title "$title" --body "$body" - else - gh pr edit i18n/update --title "$title" --body "$body" - fi - - name: Fail when a page was not translated - if: steps.translate.outputs.ok == 'false' + LOOM_I18N_PENDING_SHA: ${{ steps.prepare.outputs.pending_sha }} + run: python -m scripts.i18n.automation publish + + finalize: + name: merge validated translations + if: github.ref == 'refs/heads/main' && (github.event_name == 'schedule' || (github.event_name == 'workflow_run' && github.event.workflow_run.event == 'workflow_dispatch')) + runs-on: ubuntu-latest + timeout-minutes: 10 + permissions: + actions: write + contents: write + pull-requests: write + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: main + fetch-depth: 0 + - uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0 + with: + python-version: "3.12" + - name: Install the site builder run: | - echo "some pages failed; the pull request carries the ones that passed" - exit 1 + python -m pip install --require-hashes --no-deps --only-binary=:all: -r scripts/docs_site/requirements.txt + - name: Merge the successful exact head and publish the site + env: + GH_TOKEN: ${{ github.token }} + LOOM_I18N_LANGUAGES: ${{ vars.LOOM_I18N_LANGUAGES }} + run: python -m scripts.i18n.automation finalize diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 7350efe7e6..a7fba601c2 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -144,12 +144,19 @@ An operation or field may be renamed or removed. Say so in the PR description an ## Translations -`i18n//` holds machine translations of the README, the quickstart and the use-case pages. `scripts/i18n/translate.py` writes them and `.github/workflows/i18n.yml` runs it on main after an English page changes; never edit a translation by hand. +The translation inventory includes every tracked `README.md` and `docs/**/*.md`, including generated references. `i18n//` holds the available machine translations. Edit English or its generator; never edit translations by hand. Inventory discovery includes nested additions and renames and excludes untracked or deleted files. -The script calls Claude Sonnet 5.5 on Amazon Bedrock through the `anthropic` package's `AnthropicBedrock` client, which signs each request with the default AWS credential chain. To run it yourself, sign in to an AWS account where Bedrock serves the model and install the pinned client: +English is canonical and the default. Simplified Chinese (`zh-CN`), Japanese (`ja`), Korean (`ko`), Brazilian Portuguese (`pt-BR`) and Spanish (`es`) are the five active translation locales for default generation and localized website rollout. An additional language requires evidence that its relevant developer audience expects local-language technical documentation. Keep English for an audience that expects English technical documentation. Country, native language and developer population do not establish a translation need. This evidence requirement governs future additions. + +With an allowance and the restricted runner configured, translation follows an English merge to main. It is asynchronous: the English PR can merge while coverage is pending. The workflow queues a bounded batch after relevant main pushes and every six hours. Changed existing translations precede missing pages. Relative links use an available translated target and anchor, otherwise the English page. A verified replacement that would break surviving links waits in `i18n/.pending//`; later batches reuse it. Obsolete pages are removed only after surviving inbound links are repaired, and only with a matching generator stamp, language and source path. Unknown files are refused. A source deletion that breaks a published English fallback must include the generated link repair: run `LOOM_I18N_MAX_USD=0 python -m scripts.i18n.translate update` to repair links without provider calls. Repair uses a published translation only when its requested anchor exists; otherwise validation blocks the deletion until the incoming page is repaired. + +`python -m scripts.i18n.translate check` reports every missing, stale and obsolete page across all five translation locales. `check --skip-stale` permits pending coverage but still checks present stamps, banners, links, anchors and translated labels, plus the content invariants of current output. Ordinary PR tests follow this asynchronous contract. A changed prompt requires an explicit `update --all`; a generated refresh receipt tracks that request across bounded scheduled batches and pending-branch reuse. Completed source/output proofs prevent repeated calls; the receipt stays until main contains those results. A concurrent main translation is preserved when it differs from the refresh's recorded base output. Model and banner changes invalidate stamps automatically. + +The updater imports only validated generated blobs from `i18n/update`, preserving fresh main output. Owned output outside a selected batch can remain as hidden checkpoint data after its English source changes or disappears. Such archives never become publication routes; activation still requires current source, model and content validation. A conflicting existing checkpoint refuses replacement so both versions remain recoverable. It executes no code from that branch. It dispatches hosted CI for the exact bot commit because PR events created with `GITHUB_TOKEN` do not start CI. A hosted finalizer checks the successful commit, generated diff, current main parent, current language approval and repository review requirements before merging. Required human reviews remain required. After merge it explicitly dispatches the docs-site workflow; publication also requires the site's deployment configuration. A newer main commit defers merging until the producer composes another batch. + +The script calls Claude Sonnet 5.5 on Amazon Bedrock through `AnthropicBedrock`. Local runs also spend money and require separate authorization. For an authorized run: ```bash -# A separate virtualenv: the lock pins versions that differ from Loom's own. python -m venv .venv-i18n && . .venv-i18n/bin/activate python -m pip install --require-hashes --no-deps --only-binary=:all: -r scripts/i18n/requirements.txt aws sso login @@ -157,14 +164,16 @@ export AWS_REGION= python -m scripts.i18n.translate update ``` -The workflow stores no credential in GitHub. Its job runs on a self-hosted runner in AWS, and the SDK signs with that runner's EC2 instance role, which may invoke the one model and nothing else. Every step of the job can use the role, so the job only runs on a push to main or a manual run on main, never on a pull request. The runner group's selected-workflow rule keeps every other workflow off the runner; the job's environment `i18n`, limited to main, records each run. While the repository variable `LOOM_I18N_RUNNER` is empty, the job is skipped. +One batch shares an approximate USD 5 usage cap across every language. Before any workflow provider call, the updater books a conservative allowance covering every selected request, verifier retry, API retry and maximum output in the append-only `i18n/budget` branch. These reservations can greatly exceed the expected bill and are never refunded automatically, including after failures. `LOOM_I18N_TOTAL_USD` is the cumulative reservation ceiling, defaults to zero, and does not reset between runs or languages. An exhausted allowance defers work. A missing or invalid ledger refuses paid work. `LOOM_I18N_LANGUAGES` optionally restricts a batch to a space-separated subset of the five active translation locales. Unset or empty selects all five, as does the local update command by default. Other locale codes are refused. Manual language requests must stay within the configured subset and govern pending import, generation and publication together. Published output and checkpoints outside a selected batch stay intact. Automated maintenance changes only selected locales; present-link validation still covers retained files. A first translated README waits if publishing it would require changing an excluded locale or repairing a stale or conflicted README. Other selected pages continue. When a pending batch changes the English README’s generated language bar, importing it requires all affected locales. A partial run stops before generation or publication and retains the pending batch. An English deletion that breaks an unselected file requires a coordinated repair before it can pass validation. Raising the ceiling authorizes additional spending; a schedule cannot raise it. -To turn translations on: +The workflow stores no AWS credential in GitHub. Only main push, schedule and manual events may reach the role-bearing runner. Hosted CI and completion events cannot reach it. To activate authorized translation: -1. Create the environment `i18n` with deployment branches set to `main` only (Settings → Environments). -2. Get a runner group for this repository with "Allow public repositories" ticked (a group serves only private repositories otherwise, and the job would wait in the queue) and access limited to the selected workflow `ClickHouse/loom/.github/workflows/i18n.yml@refs/heads/main` (runner group settings → Workflow access → Selected workflows), so no other workflow and no fork's pull request can schedule a job on it. The runner image needs an OS that `actions/setup-python` supports, `git`, `bash` and the `gh` CLI. -3. Give the runners' instance role the policy below. In the account, accept Bedrock's model access for Claude Sonnet 5.5 once: the first call can otherwise fail with an access-denied error about `aws-marketplace:Subscribe`, and the role is not granted those Marketplace actions. -4. Set the repository variable `LOOM_I18N_RUNNER` to the runner group's labels as a JSON list, for example `["self-hosted","