From 62d32f0f1f4f48bea61e2a61d9c0c7d4cb6f8999 Mon Sep 17 00:00:00 2001 From: Scott Robinson Date: Sat, 31 Oct 2026 05:48:01 +0000 Subject: [PATCH] feat(backup): [backup.store], the store identity file, backup purge The `[backup.store]` section (the backup store crate's shape nested under `[backup]`, `deny_unknown_fields` on both), with the per-backend default for an absent section (a filesystem store at `backups/` beside a file-backed SQLite database, created 0700, whose validation or identity-check failure is a warning that leaves the store unconfigured while the catalog has no rows for it, and an unhealthy store or a startup error once it has; nothing on PostgreSQL, MongoDB, Cassandra and an in-memory database), `kind = "disabled"`, the spool path rule (required with an S3 store except beside a database file; refused inside or containing a filesystem root, or when it is or contains the database directory; a filesystem root that is or contains the database directory is refused the same way; an existing spool must be owner-only, parents included), relative store and spool paths made absolute at load, and the spool's `.lock`. Startup opens the configured store, validates it and checks its identity: `extenddb-store.json` at the root, created with the new `BackupStore::put_if_absent` when the root has no file and the catalog no rows for it (S3: `PutObject`/`CompleteMultipartUpload` with `If-None-Match: *`, a 412 is present, a 409 retried; filesystem: the temporary created O_EXCL and `link`ed to the key), re-read and compared with the catalog's id and with the rows' ids (`read_store_ids`), then the newest AVAILABLE row's manifest probed. A configuration error that needs no request stops startup naming the path or bucket; a filesystem store's checks run before the listener (under a 5 s bound that leaves the store `checking` when a mount stops answering after the root is opened; every later check runs under the same bound, 30 s for a bucket, the cycle selects it against shutdown, a filesystem root's abandoned call parks one thread until the mount answers and no check starts while it is, and the process exits within 5 s of SIGTERM with it parked), an S3 store's after it in the reconciler's first cycle (the component is `checking` until then); a transport, validation, identity or probe failure, a filesystem root's I/O failure at the probe write, or a check that does not finish within its bound leaves the server running with the store unhealthy: logged once, `/health` reports a `backup_store` component (status `degraded`, HTTP 200 kept, `reason` the problem's category alone), the restore and delete of a store-backed backup refuse with a ValidationException naming the kind and the category and not the root (or that the first check is pending), the reconciler acts on no row of the store, and the check runs again each cycle until it passes; the in-catalog create, which touches no store, is not refused. A check that cannot read the catalog changes nothing. DeleteBackup of a configured store's row now deletes the manifest, then the prefix, then marks the row DELETED; the reconciler finishes DELETING rows the same way. `BackupCatalog` gains `read_store_ids`, `maintenance_claim` and `remove_dead_creating_backup_row`, a `MaintenanceClaim` (heartbeat, release) held on `backup_maintenance` with each backend's claim mechanics (PostgreSQL advisory lock and row, SQLite row under the write lock, MongoDB conditional upsert, Cassandra LWT, its claim-row reads at LOCAL_SERIAL) and the stale rule; `check_maintenance_claim` reports a claim held while its heartbeat is within `stale_job_after_secs` (PostgreSQL: while the holder's session holds the advisory lock, tested with a transaction-scoped shared lock, then by the heartbeat; on SQLite a server's startup under the serve lock removes the record outright); the holder heartbeats it on a timer from the take to the release (`HeldClaim`, at `JOB_HEARTBEAT_INTERVAL`); `remove_dead_creating_backup_row` removes a CREATING row whose owner is dead. CreateBackup and RestoreTableFromBackup refuse with LimitExceededException while the claim is held, after every other check and before the permit. `ServerComponentsOptions::beside_server` opens a catalog for a read beside a server, without the SQLite serve lock and the startup recovery; on PostgreSQL the factory starts the GSI queue workers only for a server. `extenddb backup list [--failed]` (a read beside a server, on an unhealthy store too, that creates no identity file and reads it under the store's check bound) and `extenddb backup purge --confirm`: the latter takes the maintenance claim and heartbeats it for its whole run, on SQLite resolves the jobs the stopped server left (the reconciler's duties with a zero window: a RUNNING restore and the CREATING target of a FAILED one, an in-catalog CREATING backup, a DELETING row; a target that is no longer the restore target its record names is printed as left; a RUNNING `manifest` record, which no release before PR 9a writes, is left for its resumer and waited on), waits up to twice the stale window for CREATING or FAILED rows with a live owner and RUNNING restores, deletes each row's prefix and the row outright (a CREATING row with a dead owner included, the row before its prefix), leaves `extenddb-store.json`, `exports/` and prefixes with no row, and audit-logs each deletion; a row a DeleteBackup moves to DELETING or DELETED, or removes, between the command's read and its turn is finished and counted as deleted; on SQLite the owner window is zero (under the serve lock every owner is dead), so a CREATING or FAILED row with a young heartbeat is deleted, not waited on, and a server starting on the file clears a maintenance claim a killed purge left; SIGINT or SIGTERM releases the claim at once during the wait and before the next row during the deletion, and the wait's failures (jobs still running, the claim lost, the catalog unreadable, a signal) are each reported as themselves. The dry run is a read like `list` and exits 0, with no store configured as with one; `--config` is accepted after the subcommand; both commands say why there is no store when there is none. `extenddb destroy` says it leaves the store and that a re-created catalog needs the old root's identity file removed. Docs: the admin guide (`[backup.store]`, `/health` components and the reason list with `catalog unreadable`, `backup purge`, the shared-root rule), the limits doc's backup store row, the upgrade manual (configure a store before the release that stops the in-catalog create; `disabled` as the opt-out; the SQLite default root rule; the re-init sentence), the deployment guide's multi-instance notes (the hard-link requirement, a root that refuses `link(2)` reported as failed validation, the empty-catalog rule), the init template and sample config, the store crate's README and module docs, the S3 root label built from the endpoint the client uses, in one spelling; the identity file read up to 64 KiB; the console banner naming the store; design revision item 28; and the licence notices (the server binaries now link the store crate's AWS SDK tree). Co-authored-by: Lee Signed-off-by: Scott Robinson --- Cargo.lock | 10 + SOFTWARE-LICENSE-NOTICES-CASSANDRA.html | 1161 +++++++- SOFTWARE-LICENSE-NOTICES-DEV.html | 1190 +++++++- SOFTWARE-LICENSE-NOTICES-MONGODB.html | 936 ++++++- SOFTWARE-LICENSE-NOTICES.html | 1189 +++++++- crates/app/Cargo.toml | 2 + crates/app/src/cmd_backup.rs | 709 +++++ crates/app/src/cmd_destroy.rs | 14 +- crates/app/src/cmd_serve.rs | 19 +- crates/app/src/init_helpers.rs | 25 +- crates/app/src/lib.rs | 85 +- crates/backup-store/README.md | 20 + crates/backup-store/src/config.rs | 34 +- crates/backup-store/src/fs.rs | 321 ++- crates/backup-store/src/lib.rs | 77 +- crates/backup-store/src/s3.rs | 442 ++- crates/backup-store/tests/conformance.rs | 535 ++++ crates/config/Cargo.toml | 4 + crates/config/src/backup.rs | 674 ++++- crates/config/src/lib.rs | 15 +- crates/engine/Cargo.toml | 5 + crates/engine/src/backup.rs | 187 +- crates/engine/src/backup/maintenance.rs | 537 ++++ crates/engine/src/backup/reconciler.rs | 180 +- crates/engine/src/backup/runtime.rs | 691 ++++- crates/engine/src/backup/store_identity.rs | 665 +++++ crates/engine/src/backup/tests.rs | 2448 ++++++++++++++++- crates/engine/src/lib.rs | 6 +- crates/server/Cargo.toml | 3 + crates/server/src/backup_store.rs | 1343 +++++++++ crates/server/src/lib.rs | 37 +- crates/server/src/serve.rs | 20 +- .../storage-cassandra/src/backup_catalog.rs | 414 ++- .../storage-cassandra/src/cassandra_util.rs | 33 + crates/storage-cassandra/src/lib.rs | 18 +- .../tests/direct/backup_engine.rs | 253 +- crates/storage-mongodb/src/backup_catalog.rs | 200 +- .../storage-mongodb/src/backup_live_tests.rs | 241 +- crates/storage-postgres/Cargo.toml | 5 + crates/storage-postgres/src/backup_catalog.rs | 195 +- crates/storage-postgres/src/backup_engine.rs | 8 + crates/storage-postgres/src/lib.rs | 40 +- .../storage-postgres/tests/backup_restore.rs | 849 +++++- crates/storage-sqlite/src/backup.rs | 218 +- crates/storage-sqlite/src/backup_catalog.rs | 164 +- crates/storage-sqlite/src/config.rs | 4 + crates/storage-sqlite/src/lib.rs | 235 +- crates/storage/src/backup_catalog.rs | 74 +- crates/storage/src/config.rs | 10 + crates/storage/src/lib.rs | 4 +- crates/storage/src/server_components.rs | 9 + devtools/run-tests | 2 +- docs/design/15-backup-restore.md | 30 +- docs/dynamodb-limits.md | 3 +- docs/manuals/05-admin-guide.md | 54 +- docs/manuals/07-upgrade-manual.md | 4 +- docs/manuals/11-deployment-guide.md | 3 +- extenddb.sample.toml | 25 +- tests/test_backup_store_config.py | 1331 +++++++++ 59 files changed, 17284 insertions(+), 726 deletions(-) create mode 100644 crates/app/src/cmd_backup.rs create mode 100644 crates/engine/src/backup/maintenance.rs create mode 100644 crates/engine/src/backup/store_identity.rs create mode 100644 crates/server/src/backup_store.rs create mode 100644 tests/test_backup_store_config.py diff --git a/Cargo.lock b/Cargo.lock index 443c6c8fa..7a5ca02d0 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1705,11 +1705,13 @@ dependencies = [ "clap", "daemonize", "extenddb-auth", + "extenddb-backup-store", "extenddb-config", "extenddb-core", "extenddb-engine", "extenddb-server", "extenddb-storage", + "futures", "libc", "rcgen", "rustls", @@ -1785,9 +1787,11 @@ version = "0.1.14" dependencies = [ "anyhow", "config", + "extenddb-backup-store", "extenddb-core", "extenddb-storage", "serde", + "tempfile", "toml", "tracing", ] @@ -1815,6 +1819,7 @@ dependencies = [ "base64 0.22.1", "crc32fast", "extenddb-auth", + "extenddb-backup-store", "extenddb-core", "extenddb-storage", "futures", @@ -1841,6 +1846,7 @@ dependencies = [ "bcrypt", "crc32fast", "extenddb-auth", + "extenddb-backup-store", "extenddb-cache", "extenddb-config", "extenddb-core", @@ -1855,6 +1861,7 @@ dependencies = [ "serde", "serde_json", "syslog-tracing", + "tempfile", "time", "tokio", "tower", @@ -1959,7 +1966,9 @@ dependencies = [ "bcrypt", "crc32fast", "extenddb-auth", + "extenddb-backup-store", "extenddb-core", + "extenddb-engine", "extenddb-storage", "extenddb-storage-postgres", "futures", @@ -1969,6 +1978,7 @@ dependencies = [ "serde", "serde_json", "sqlx", + "tempfile", "time", "tokio", "toml", diff --git a/SOFTWARE-LICENSE-NOTICES-CASSANDRA.html b/SOFTWARE-LICENSE-NOTICES-CASSANDRA.html index 867d1265f..02c2fa6eb 100644 --- a/SOFTWARE-LICENSE-NOTICES-CASSANDRA.html +++ b/SOFTWARE-LICENSE-NOTICES-CASSANDRA.html @@ -49,14 +49,14 @@

ExtendDB Software License Notices

Overview of licenses:

All license text:

@@ -1741,6 +1741,424 @@

Used by:

See the License for the specific language governing permissions and limitations under the License. + + +
  • +

    Apache License 2.0

    +

    Used by:

    + +
                                     Apache License
    +                           Version 2.0, January 2004
    +                        http://www.apache.org/licenses/
    +
    +   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
    +
    +   1. Definitions.
    +
    +      "License" shall mean the terms and conditions for use, reproduction,
    +      and distribution as defined by Sections 1 through 9 of this document.
    +
    +      "Licensor" shall mean the copyright owner or entity authorized by
    +      the copyright owner that is granting the License.
    +
    +      "Legal Entity" shall mean the union of the acting entity and all
    +      other entities that control, are controlled by, or are under common
    +      control with that entity. For the purposes of this definition,
    +      "control" means (i) the power, direct or indirect, to cause the
    +      direction or management of such entity, whether by contract or
    +      otherwise, or (ii) ownership of fifty percent (50%) or more of the
    +      outstanding shares, or (iii) beneficial ownership of such entity.
    +
    +      "You" (or "Your") shall mean an individual or Legal Entity
    +      exercising permissions granted by this License.
    +
    +      "Source" form shall mean the preferred form for making modifications,
    +      including but not limited to software source code, documentation
    +      source, and configuration files.
    +
    +      "Object" form shall mean any form resulting from mechanical
    +      transformation or translation of a Source form, including but
    +      not limited to compiled object code, generated documentation,
    +      and conversions to other media types.
    +
    +      "Work" shall mean the work of authorship, whether in Source or
    +      Object form, made available under the License, as indicated by a
    +      copyright notice that is included in or attached to the work
    +      (an example is provided in the Appendix below).
    +
    +      "Derivative Works" shall mean any work, whether in Source or Object
    +      form, that is based on (or derived from) the Work and for which the
    +      editorial revisions, annotations, elaborations, or other modifications
    +      represent, as a whole, an original work of authorship. For the purposes
    +      of this License, Derivative Works shall not include works that remain
    +      separable from, or merely link (or bind by name) to the interfaces of,
    +      the Work and Derivative Works thereof.
    +
    +      "Contribution" shall mean any work of authorship, including
    +      the original version of the Work and any modifications or additions
    +      to that Work or Derivative Works thereof, that is intentionally
    +      submitted to Licensor for inclusion in the Work by the copyright owner
    +      or by an individual or Legal Entity authorized to submit on behalf of
    +      the copyright owner. For the purposes of this definition, "submitted"
    +      means any form of electronic, verbal, or written communication sent
    +      to the Licensor or its representatives, including but not limited to
    +      communication on electronic mailing lists, source code control systems,
    +      and issue tracking systems that are managed by, or on behalf of, the
    +      Licensor for the purpose of discussing and improving the Work, but
    +      excluding communication that is conspicuously marked or otherwise
    +      designated in writing by the copyright owner as "Not a Contribution."
    +
    +      "Contributor" shall mean Licensor and any individual or Legal Entity
    +      on behalf of whom a Contribution has been received by Licensor and
    +      subsequently incorporated within the Work.
    +
    +   2. Grant of Copyright License. Subject to the terms and conditions of
    +      this License, each Contributor hereby grants to You a perpetual,
    +      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +      copyright license to reproduce, prepare Derivative Works of,
    +      publicly display, publicly perform, sublicense, and distribute the
    +      Work and such Derivative Works in Source or Object form.
    +
    +   3. Grant of Patent License. Subject to the terms and conditions of
    +      this License, each Contributor hereby grants to You a perpetual,
    +      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +      (except as stated in this section) patent license to make, have made,
    +      use, offer to sell, sell, import, and otherwise transfer the Work,
    +      where such license applies only to those patent claims licensable
    +      by such Contributor that are necessarily infringed by their
    +      Contribution(s) alone or by combination of their Contribution(s)
    +      with the Work to which such Contribution(s) was submitted. If You
    +      institute patent litigation against any entity (including a
    +      cross-claim or counterclaim in a lawsuit) alleging that the Work
    +      or a Contribution incorporated within the Work constitutes direct
    +      or contributory patent infringement, then any patent licenses
    +      granted to You under this License for that Work shall terminate
    +      as of the date such litigation is filed.
    +
    +   4. Redistribution. You may reproduce and distribute copies of the
    +      Work or Derivative Works thereof in any medium, with or without
    +      modifications, and in Source or Object form, provided that You
    +      meet the following conditions:
    +
    +      (a) You must give any other recipients of the Work or
    +          Derivative Works a copy of this License; and
    +
    +      (b) You must cause any modified files to carry prominent notices
    +          stating that You changed the files; and
    +
    +      (c) You must retain, in the Source form of any Derivative Works
    +          that You distribute, all copyright, patent, trademark, and
    +          attribution notices from the Source form of the Work,
    +          excluding those notices that do not pertain to any part of
    +          the Derivative Works; and
    +
    +      (d) If the Work includes a "NOTICE" text file as part of its
    +          distribution, then any Derivative Works that You distribute must
    +          include a readable copy of the attribution notices contained
    +          within such NOTICE file, excluding those notices that do not
    +          pertain to any part of the Derivative Works, in at least one
    +          of the following places: within a NOTICE text file distributed
    +          as part of the Derivative Works; within the Source form or
    +          documentation, if provided along with the Derivative Works; or,
    +          within a display generated by the Derivative Works, if and
    +          wherever such third-party notices normally appear. The contents
    +          of the NOTICE file are for informational purposes only and
    +          do not modify the License. You may add Your own attribution
    +          notices within Derivative Works that You distribute, alongside
    +          or as an addendum to the NOTICE text from the Work, provided
    +          that such additional attribution notices cannot be construed
    +          as modifying the License.
    +
    +      You may add Your own copyright statement to Your modifications and
    +      may provide additional or different license terms and conditions
    +      for use, reproduction, or distribution of Your modifications, or
    +      for any such Derivative Works as a whole, provided Your use,
    +      reproduction, and distribution of the Work otherwise complies with
    +      the conditions stated in this License.
    +
    +   5. Submission of Contributions. Unless You explicitly state otherwise,
    +      any Contribution intentionally submitted for inclusion in the Work
    +      by You to the Licensor shall be under the terms and conditions of
    +      this License, without any additional terms or conditions.
    +      Notwithstanding the above, nothing herein shall supersede or modify
    +      the terms of any separate license agreement you may have executed
    +      with Licensor regarding such Contributions.
    +
    +   6. Trademarks. This License does not grant permission to use the trade
    +      names, trademarks, service marks, or product names of the Licensor,
    +      except as required for reasonable and customary use in describing the
    +      origin of the Work and reproducing the content of the NOTICE file.
    +
    +   7. Disclaimer of Warranty. Unless required by applicable law or
    +      agreed to in writing, Licensor provides the Work (and each
    +      Contributor provides its Contributions) on an "AS IS" BASIS,
    +      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
    +      implied, including, without limitation, any warranties or conditions
    +      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
    +      PARTICULAR PURPOSE. You are solely responsible for determining the
    +      appropriateness of using or redistributing the Work and assume any
    +      risks associated with Your exercise of permissions under this License.
    +
    +   8. Limitation of Liability. In no event and under no legal theory,
    +      whether in tort (including negligence), contract, or otherwise,
    +      unless required by applicable law (such as deliberate and grossly
    +      negligent acts) or agreed to in writing, shall any Contributor be
    +      liable to You for damages, including any direct, indirect, special,
    +      incidental, or consequential damages of any character arising as a
    +      result of this License or out of the use or inability to use the
    +      Work (including but not limited to damages for loss of goodwill,
    +      work stoppage, computer failure or malfunction, or any and all
    +      other commercial damages or losses), even if such Contributor
    +      has been advised of the possibility of such damages.
    +
    +   9. Accepting Warranty or Additional Liability. While redistributing
    +      the Work or Derivative Works thereof, You may choose to offer,
    +      and charge a fee for, acceptance of support, warranty, indemnity,
    +      or other liability obligations and/or rights consistent with this
    +      License. However, in accepting such obligations, You may act only
    +      on Your own behalf and on Your sole responsibility, not on behalf
    +      of any other Contributor, and only if You agree to indemnify,
    +      defend, and hold each Contributor harmless for any liability
    +      incurred by, or claims asserted against, such Contributor by reason
    +      of your accepting any such warranty or additional liability.
    +
    +   END OF TERMS AND CONDITIONS
    +
    +   APPENDIX: How to apply the Apache License to your work.
    +
    +      To apply the Apache License to your work, attach the following
    +      boilerplate notice, with the fields enclosed by brackets "[]"
    +      replaced with your own identifying information. (Don't include
    +      the brackets!)  The text should be enclosed in the appropriate
    +      comment syntax for the file format. We also recommend that a
    +      file or class name and description of purpose be included on the
    +      same "printed page" as the copyright notice for easier
    +      identification within third-party archives.
    +
    +   Copyright [yyyy] [name of copyright owner]
    +
    +   Licensed under the Apache License, Version 2.0 (the "License");
    +   you may not use this file except in compliance with the License.
    +   You may obtain a copy of the License at
    +
    +       http://www.apache.org/licenses/LICENSE-2.0
    +
    +   Unless required by applicable law or agreed to in writing, software
    +   distributed under the License is distributed on an "AS IS" BASIS,
    +   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +   See the License for the specific language governing permissions and
    +   limitations under the License.
    +
    +
  • +
  • +

    Apache License 2.0

    +

    Used by:

    + +
                                     Apache License
    +                           Version 2.0, January 2004
    +                        http://www.apache.org/licenses/
    +
    +   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
    +
    +   1. Definitions.
    +
    +      "License" shall mean the terms and conditions for use, reproduction,
    +      and distribution as defined by Sections 1 through 9 of this document.
    +
    +      "Licensor" shall mean the copyright owner or entity authorized by
    +      the copyright owner that is granting the License.
    +
    +      "Legal Entity" shall mean the union of the acting entity and all
    +      other entities that control, are controlled by, or are under common
    +      control with that entity. For the purposes of this definition,
    +      "control" means (i) the power, direct or indirect, to cause the
    +      direction or management of such entity, whether by contract or
    +      otherwise, or (ii) ownership of fifty percent (50%) or more of the
    +      outstanding shares, or (iii) beneficial ownership of such entity.
    +
    +      "You" (or "Your") shall mean an individual or Legal Entity
    +      exercising permissions granted by this License.
    +
    +      "Source" form shall mean the preferred form for making modifications,
    +      including but not limited to software source code, documentation
    +      source, and configuration files.
    +
    +      "Object" form shall mean any form resulting from mechanical
    +      transformation or translation of a Source form, including but
    +      not limited to compiled object code, generated documentation,
    +      and conversions to other media types.
    +
    +      "Work" shall mean the work of authorship, whether in Source or
    +      Object form, made available under the License, as indicated by a
    +      copyright notice that is included in or attached to the work
    +      (an example is provided in the Appendix below).
    +
    +      "Derivative Works" shall mean any work, whether in Source or Object
    +      form, that is based on (or derived from) the Work and for which the
    +      editorial revisions, annotations, elaborations, or other modifications
    +      represent, as a whole, an original work of authorship. For the purposes
    +      of this License, Derivative Works shall not include works that remain
    +      separable from, or merely link (or bind by name) to the interfaces of,
    +      the Work and Derivative Works thereof.
    +
    +      "Contribution" shall mean any work of authorship, including
    +      the original version of the Work and any modifications or additions
    +      to that Work or Derivative Works thereof, that is intentionally
    +      submitted to Licensor for inclusion in the Work by the copyright owner
    +      or by an individual or Legal Entity authorized to submit on behalf of
    +      the copyright owner. For the purposes of this definition, "submitted"
    +      means any form of electronic, verbal, or written communication sent
    +      to the Licensor or its representatives, including but not limited to
    +      communication on electronic mailing lists, source code control systems,
    +      and issue tracking systems that are managed by, or on behalf of, the
    +      Licensor for the purpose of discussing and improving the Work, but
    +      excluding communication that is conspicuously marked or otherwise
    +      designated in writing by the copyright owner as "Not a Contribution."
    +
    +      "Contributor" shall mean Licensor and any individual or Legal Entity
    +      on behalf of whom a Contribution has been received by Licensor and
    +      subsequently incorporated within the Work.
    +
    +   2. Grant of Copyright License. Subject to the terms and conditions of
    +      this License, each Contributor hereby grants to You a perpetual,
    +      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +      copyright license to reproduce, prepare Derivative Works of,
    +      publicly display, publicly perform, sublicense, and distribute the
    +      Work and such Derivative Works in Source or Object form.
    +
    +   3. Grant of Patent License. Subject to the terms and conditions of
    +      this License, each Contributor hereby grants to You a perpetual,
    +      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +      (except as stated in this section) patent license to make, have made,
    +      use, offer to sell, sell, import, and otherwise transfer the Work,
    +      where such license applies only to those patent claims licensable
    +      by such Contributor that are necessarily infringed by their
    +      Contribution(s) alone or by combination of their Contribution(s)
    +      with the Work to which such Contribution(s) was submitted. If You
    +      institute patent litigation against any entity (including a
    +      cross-claim or counterclaim in a lawsuit) alleging that the Work
    +      or a Contribution incorporated within the Work constitutes direct
    +      or contributory patent infringement, then any patent licenses
    +      granted to You under this License for that Work shall terminate
    +      as of the date such litigation is filed.
    +
    +   4. Redistribution. You may reproduce and distribute copies of the
    +      Work or Derivative Works thereof in any medium, with or without
    +      modifications, and in Source or Object form, provided that You
    +      meet the following conditions:
    +
    +      (a) You must give any other recipients of the Work or
    +          Derivative Works a copy of this License; and
    +
    +      (b) You must cause any modified files to carry prominent notices
    +          stating that You changed the files; and
    +
    +      (c) You must retain, in the Source form of any Derivative Works
    +          that You distribute, all copyright, patent, trademark, and
    +          attribution notices from the Source form of the Work,
    +          excluding those notices that do not pertain to any part of
    +          the Derivative Works; and
    +
    +      (d) If the Work includes a "NOTICE" text file as part of its
    +          distribution, then any Derivative Works that You distribute must
    +          include a readable copy of the attribution notices contained
    +          within such NOTICE file, excluding those notices that do not
    +          pertain to any part of the Derivative Works, in at least one
    +          of the following places: within a NOTICE text file distributed
    +          as part of the Derivative Works; within the Source form or
    +          documentation, if provided along with the Derivative Works; or,
    +          within a display generated by the Derivative Works, if and
    +          wherever such third-party notices normally appear. The contents
    +          of the NOTICE file are for informational purposes only and
    +          do not modify the License. You may add Your own attribution
    +          notices within Derivative Works that You distribute, alongside
    +          or as an addendum to the NOTICE text from the Work, provided
    +          that such additional attribution notices cannot be construed
    +          as modifying the License.
    +
    +      You may add Your own copyright statement to Your modifications and
    +      may provide additional or different license terms and conditions
    +      for use, reproduction, or distribution of Your modifications, or
    +      for any such Derivative Works as a whole, provided Your use,
    +      reproduction, and distribution of the Work otherwise complies with
    +      the conditions stated in this License.
    +
    +   5. Submission of Contributions. Unless You explicitly state otherwise,
    +      any Contribution intentionally submitted for inclusion in the Work
    +      by You to the Licensor shall be under the terms and conditions of
    +      this License, without any additional terms or conditions.
    +      Notwithstanding the above, nothing herein shall supersede or modify
    +      the terms of any separate license agreement you may have executed
    +      with Licensor regarding such Contributions.
    +
    +   6. Trademarks. This License does not grant permission to use the trade
    +      names, trademarks, service marks, or product names of the Licensor,
    +      except as required for reasonable and customary use in describing the
    +      origin of the Work and reproducing the content of the NOTICE file.
    +
    +   7. Disclaimer of Warranty. Unless required by applicable law or
    +      agreed to in writing, Licensor provides the Work (and each
    +      Contributor provides its Contributions) on an "AS IS" BASIS,
    +      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
    +      implied, including, without limitation, any warranties or conditions
    +      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
    +      PARTICULAR PURPOSE. You are solely responsible for determining the
    +      appropriateness of using or redistributing the Work and assume any
    +      risks associated with Your exercise of permissions under this License.
    +
    +   8. Limitation of Liability. In no event and under no legal theory,
    +      whether in tort (including negligence), contract, or otherwise,
    +      unless required by applicable law (such as deliberate and grossly
    +      negligent acts) or agreed to in writing, shall any Contributor be
    +      liable to You for damages, including any direct, indirect, special,
    +      incidental, or consequential damages of any character arising as a
    +      result of this License or out of the use or inability to use the
    +      Work (including but not limited to damages for loss of goodwill,
    +      work stoppage, computer failure or malfunction, or any and all
    +      other commercial damages or losses), even if such Contributor
    +      has been advised of the possibility of such damages.
    +
    +   9. Accepting Warranty or Additional Liability. While redistributing
    +      the Work or Derivative Works thereof, You may choose to offer,
    +      and charge a fee for, acceptance of support, warranty, indemnity,
    +      or other liability obligations and/or rights consistent with this
    +      License. However, in accepting such obligations, You may act only
    +      on Your own behalf and on Your sole responsibility, not on behalf
    +      of any other Contributor, and only if You agree to indemnify,
    +      defend, and hold each Contributor harmless for any liability
    +      incurred by, or claims asserted against, such Contributor by reason
    +      of your accepting any such warranty or additional liability.
    +
    +   END OF TERMS AND CONDITIONS
    +
    +   APPENDIX: How to apply the Apache License to your work.
    +
    +      To apply the Apache License to your work, attach the following
    +      boilerplate notice, with the fields enclosed by brackets "{}"
    +      replaced with your own identifying information. (Don't include
    +      the brackets!)  The text should be enclosed in the appropriate
    +      comment syntax for the file format. We also recommend that a
    +      file or class name and description of purpose be included on the
    +      same "printed page" as the copyright notice for easier
    +      identification within third-party archives.
    +
    +   Copyright 2017 Juniper Networks, Inc.
    +
    +   Licensed under the Apache License, Version 2.0 (the "License");
    +   you may not use this file except in compliance with the License.
    +   You may obtain a copy of the License at
    +
    +       http://www.apache.org/licenses/LICENSE-2.0
    +
    +   Unless required by applicable law or agreed to in writing, software
    +   distributed under the License is distributed on an "AS IS" BASIS,
    +   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +   See the License for the specific language governing permissions and
    +   limitations under the License.
     
  • @@ -1767,7 +2185,220 @@

    Used by:

  • toml_edit 0.22.27
  • toml_write 0.1.2
  • -
                                     Apache License
    +                
                                     Apache License
    +                           Version 2.0, January 2004
    +                        http://www.apache.org/licenses/
    +
    +   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
    +
    +   1. Definitions.
    +
    +      "License" shall mean the terms and conditions for use, reproduction,
    +      and distribution as defined by Sections 1 through 9 of this document.
    +
    +      "Licensor" shall mean the copyright owner or entity authorized by
    +      the copyright owner that is granting the License.
    +
    +      "Legal Entity" shall mean the union of the acting entity and all
    +      other entities that control, are controlled by, or are under common
    +      control with that entity. For the purposes of this definition,
    +      "control" means (i) the power, direct or indirect, to cause the
    +      direction or management of such entity, whether by contract or
    +      otherwise, or (ii) ownership of fifty percent (50%) or more of the
    +      outstanding shares, or (iii) beneficial ownership of such entity.
    +
    +      "You" (or "Your") shall mean an individual or Legal Entity
    +      exercising permissions granted by this License.
    +
    +      "Source" form shall mean the preferred form for making modifications,
    +      including but not limited to software source code, documentation
    +      source, and configuration files.
    +
    +      "Object" form shall mean any form resulting from mechanical
    +      transformation or translation of a Source form, including but
    +      not limited to compiled object code, generated documentation,
    +      and conversions to other media types.
    +
    +      "Work" shall mean the work of authorship, whether in Source or
    +      Object form, made available under the License, as indicated by a
    +      copyright notice that is included in or attached to the work
    +      (an example is provided in the Appendix below).
    +
    +      "Derivative Works" shall mean any work, whether in Source or Object
    +      form, that is based on (or derived from) the Work and for which the
    +      editorial revisions, annotations, elaborations, or other modifications
    +      represent, as a whole, an original work of authorship. For the purposes
    +      of this License, Derivative Works shall not include works that remain
    +      separable from, or merely link (or bind by name) to the interfaces of,
    +      the Work and Derivative Works thereof.
    +
    +      "Contribution" shall mean any work of authorship, including
    +      the original version of the Work and any modifications or additions
    +      to that Work or Derivative Works thereof, that is intentionally
    +      submitted to Licensor for inclusion in the Work by the copyright owner
    +      or by an individual or Legal Entity authorized to submit on behalf of
    +      the copyright owner. For the purposes of this definition, "submitted"
    +      means any form of electronic, verbal, or written communication sent
    +      to the Licensor or its representatives, including but not limited to
    +      communication on electronic mailing lists, source code control systems,
    +      and issue tracking systems that are managed by, or on behalf of, the
    +      Licensor for the purpose of discussing and improving the Work, but
    +      excluding communication that is conspicuously marked or otherwise
    +      designated in writing by the copyright owner as "Not a Contribution."
    +
    +      "Contributor" shall mean Licensor and any individual or Legal Entity
    +      on behalf of whom a Contribution has been received by Licensor and
    +      subsequently incorporated within the Work.
    +
    +   2. Grant of Copyright License. Subject to the terms and conditions of
    +      this License, each Contributor hereby grants to You a perpetual,
    +      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +      copyright license to reproduce, prepare Derivative Works of,
    +      publicly display, publicly perform, sublicense, and distribute the
    +      Work and such Derivative Works in Source or Object form.
    +
    +   3. Grant of Patent License. Subject to the terms and conditions of
    +      this License, each Contributor hereby grants to You a perpetual,
    +      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +      (except as stated in this section) patent license to make, have made,
    +      use, offer to sell, sell, import, and otherwise transfer the Work,
    +      where such license applies only to those patent claims licensable
    +      by such Contributor that are necessarily infringed by their
    +      Contribution(s) alone or by combination of their Contribution(s)
    +      with the Work to which such Contribution(s) was submitted. If You
    +      institute patent litigation against any entity (including a
    +      cross-claim or counterclaim in a lawsuit) alleging that the Work
    +      or a Contribution incorporated within the Work constitutes direct
    +      or contributory patent infringement, then any patent licenses
    +      granted to You under this License for that Work shall terminate
    +      as of the date such litigation is filed.
    +
    +   4. Redistribution. You may reproduce and distribute copies of the
    +      Work or Derivative Works thereof in any medium, with or without
    +      modifications, and in Source or Object form, provided that You
    +      meet the following conditions:
    +
    +      (a) You must give any other recipients of the Work or
    +          Derivative Works a copy of this License; and
    +
    +      (b) You must cause any modified files to carry prominent notices
    +          stating that You changed the files; and
    +
    +      (c) You must retain, in the Source form of any Derivative Works
    +          that You distribute, all copyright, patent, trademark, and
    +          attribution notices from the Source form of the Work,
    +          excluding those notices that do not pertain to any part of
    +          the Derivative Works; and
    +
    +      (d) If the Work includes a "NOTICE" text file as part of its
    +          distribution, then any Derivative Works that You distribute must
    +          include a readable copy of the attribution notices contained
    +          within such NOTICE file, excluding those notices that do not
    +          pertain to any part of the Derivative Works, in at least one
    +          of the following places: within a NOTICE text file distributed
    +          as part of the Derivative Works; within the Source form or
    +          documentation, if provided along with the Derivative Works; or,
    +          within a display generated by the Derivative Works, if and
    +          wherever such third-party notices normally appear. The contents
    +          of the NOTICE file are for informational purposes only and
    +          do not modify the License. You may add Your own attribution
    +          notices within Derivative Works that You distribute, alongside
    +          or as an addendum to the NOTICE text from the Work, provided
    +          that such additional attribution notices cannot be construed
    +          as modifying the License.
    +
    +      You may add Your own copyright statement to Your modifications and
    +      may provide additional or different license terms and conditions
    +      for use, reproduction, or distribution of Your modifications, or
    +      for any such Derivative Works as a whole, provided Your use,
    +      reproduction, and distribution of the Work otherwise complies with
    +      the conditions stated in this License.
    +
    +   5. Submission of Contributions. Unless You explicitly state otherwise,
    +      any Contribution intentionally submitted for inclusion in the Work
    +      by You to the Licensor shall be under the terms and conditions of
    +      this License, without any additional terms or conditions.
    +      Notwithstanding the above, nothing herein shall supersede or modify
    +      the terms of any separate license agreement you may have executed
    +      with Licensor regarding such Contributions.
    +
    +   6. Trademarks. This License does not grant permission to use the trade
    +      names, trademarks, service marks, or product names of the Licensor,
    +      except as required for reasonable and customary use in describing the
    +      origin of the Work and reproducing the content of the NOTICE file.
    +
    +   7. Disclaimer of Warranty. Unless required by applicable law or
    +      agreed to in writing, Licensor provides the Work (and each
    +      Contributor provides its Contributions) on an "AS IS" BASIS,
    +      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
    +      implied, including, without limitation, any warranties or conditions
    +      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
    +      PARTICULAR PURPOSE. You are solely responsible for determining the
    +      appropriateness of using or redistributing the Work and assume any
    +      risks associated with Your exercise of permissions under this License.
    +
    +   8. Limitation of Liability. In no event and under no legal theory,
    +      whether in tort (including negligence), contract, or otherwise,
    +      unless required by applicable law (such as deliberate and grossly
    +      negligent acts) or agreed to in writing, shall any Contributor be
    +      liable to You for damages, including any direct, indirect, special,
    +      incidental, or consequential damages of any character arising as a
    +      result of this License or out of the use or inability to use the
    +      Work (including but not limited to damages for loss of goodwill,
    +      work stoppage, computer failure or malfunction, or any and all
    +      other commercial damages or losses), even if such Contributor
    +      has been advised of the possibility of such damages.
    +
    +   9. Accepting Warranty or Additional Liability. While redistributing
    +      the Work or Derivative Works thereof, You may choose to offer,
    +      and charge a fee for, acceptance of support, warranty, indemnity,
    +      or other liability obligations and/or rights consistent with this
    +      License. However, in accepting such obligations, You may act only
    +      on Your own behalf and on Your sole responsibility, not on behalf
    +      of any other Contributor, and only if You agree to indemnify,
    +      defend, and hold each Contributor harmless for any liability
    +      incurred by, or claims asserted against, such Contributor by reason
    +      of your accepting any such warranty or additional liability.
    +
    +   END OF TERMS AND CONDITIONS
    +
    +   APPENDIX: How to apply the Apache License to your work.
    +
    +      To apply the Apache License to your work, attach the following
    +      boilerplate notice, with the fields enclosed by brackets "{}"
    +      replaced with your own identifying information. (Don't include
    +      the brackets!)  The text should be enclosed in the appropriate
    +      comment syntax for the file format. We also recommend that a
    +      file or class name and description of purpose be included on the
    +      same "printed page" as the copyright notice for easier
    +      identification within third-party archives.
    +
    +   Copyright {yyyy} {name of copyright owner}
    +
    +   Licensed under the Apache License, Version 2.0 (the "License");
    +   you may not use this file except in compliance with the License.
    +   You may obtain a copy of the License at
    +
    +       http://www.apache.org/licenses/LICENSE-2.0
    +
    +   Unless required by applicable law or agreed to in writing, software
    +   distributed under the License is distributed on an "AS IS" BASIS,
    +   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +   See the License for the specific language governing permissions and
    +   limitations under the License.
    +
    +
    + +
  • +

    Apache License 2.0

    +

    Used by:

    + +
                                    Apache License
                                Version 2.0, January 2004
                             http://www.apache.org/licenses/
     
    @@ -1955,7 +2586,7 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. - Copyright {yyyy} {name of copyright owner} + Copyright 2018-2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -1975,10 +2606,227 @@

    Used by:

    Apache License 2.0

    Used by:

    +
                                  Apache License
    +                        Version 2.0 January 2004
    +                     http://www.apache.org/licenses/
    +
    +TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
    +
    +1. Definitions.
    +
    +   "License" shall mean the terms and conditions for use, reproduction,
    +   and distribution as defined by Sections 1 through 9 of this document.
    +
    +   "Licensor" shall mean the copyright owner or entity authorized by
    +   the copyright owner that is granting the License.
    +
    +   "Legal Entity" shall mean the union of the acting entity and all
    +   other entities that control, are controlled by, or are under common
    +   control with that entity. For the purposes of this definition,
    +   "control" means (i) the power, direct or indirect, to cause the
    +   direction or management of such entity, whether by contract or
    +   otherwise, or (ii) ownership of fifty percent (50%) or more of the
    +   outstanding shares, or (iii) beneficial ownership of such entity.
    +
    +   "You" (or "Your") shall mean an individual or Legal Entity
    +   exercising permissions granted by this License.
    +
    +   "Source" form shall mean the preferred form for making modifications,
    +   including but not limited to software source code, documentation
    +   source, and configuration files.
    +
    +   "Object" form shall mean any form resulting from mechanical
    +   transformation or translation of a Source form, including but
    +   not limited to compiled object code, generated documentation,
    +   and conversions to other media types.
    +
    +   "Work" shall mean the work of authorship, whether in Source or
    +   Object form, made available under the License, as indicated by a
    +   copyright notice that is included in or attached to the work
    +   (an example is provided in the Appendix below).
    +
    +   "Derivative Works" shall mean any work, whether in Source or Object
    +   form, that is based on (or derived from) the Work and for which the
    +   editorial revisions, annotations, elaborations, or other modifications
    +   represent, as a whole, an original work of authorship. For the purposes
    +   of this License, Derivative Works shall not include works that remain
    +   separable from, or merely link (or bind by name) to the interfaces of,
    +   the Work and Derivative Works thereof.
    +
    +   "Contribution" shall mean any work of authorship, including
    +   the original version of the Work and any modifications or additions
    +   to that Work or Derivative Works thereof, that is intentionally
    +   submitted to Licensor for inclusion in the Work by the copyright owner
    +   or by an individual or Legal Entity authorized to submit on behalf of
    +   the copyright owner. For the purposes of this definition, "submitted"
    +   means any form of electronic, verbal, or written communication sent
    +   to the Licensor or its representatives, including but not limited to
    +   communication on electronic mailing lists, source code control systems,
    +   and issue tracking systems that are managed by, or on behalf of, the
    +   Licensor for the purpose of discussing and improving the Work, but
    +   excluding communication that is conspicuously marked or otherwise
    +   designated in writing by the copyright owner as "Not a Contribution."
    +
    +   "Contributor" shall mean Licensor and any individual or Legal Entity
    +   on behalf of whom a Contribution has been received by Licensor and
    +   subsequently incorporated within the Work.
    +
    +2. Grant of Copyright License. Subject to the terms and conditions of
    +   this License, each Contributor hereby grants to You a perpetual,
    +   worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +   copyright license to reproduce, prepare Derivative Works of,
    +   publicly display, publicly perform, sublicense, and distribute the
    +   Work and such Derivative Works in Source or Object form.
    +
    +3. Grant of Patent License. Subject to the terms and conditions of
    +   this License, each Contributor hereby grants to You a perpetual,
    +   worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +   (except as stated in this section) patent license to make, have made,
    +   use, offer to sell, sell, import, and otherwise transfer the Work,
    +   where such license applies only to those patent claims licensable
    +   by such Contributor that are necessarily infringed by their
    +   Contribution(s) alone or by combination of their Contribution(s)
    +   with the Work to which such Contribution(s) was submitted. If You
    +   institute patent litigation against any entity (including a
    +   cross-claim or counterclaim in a lawsuit) alleging that the Work
    +   or a Contribution incorporated within the Work constitutes direct
    +   or contributory patent infringement, then any patent licenses
    +   granted to You under this License for that Work shall terminate
    +   as of the date such litigation is filed.
    +
    +4. Redistribution. You may reproduce and distribute copies of the
    +   Work or Derivative Works thereof in any medium, with or without
    +   modifications, and in Source or Object form, provided that You
    +   meet the following conditions:
    +
    +   (a) You must give any other recipients of the Work or
    +       Derivative Works a copy of this License; and
    +
    +   (b) You must cause any modified files to carry prominent notices
    +       stating that You changed the files; and
    +
    +   (c) You must retain, in the Source form of any Derivative Works
    +       that You distribute, all copyright, patent, trademark, and
    +       attribution notices from the Source form of the Work,
    +       excluding those notices that do not pertain to any part of
    +       the Derivative Works; and
    +
    +   (d) If the Work includes a "NOTICE" text file as part of its
    +       distribution, then any Derivative Works that You distribute must
    +       include a readable copy of the attribution notices contained
    +       within such NOTICE file, excluding those notices that do not
    +       pertain to any part of the Derivative Works, in at least one
    +       of the following places: within a NOTICE text file distributed
    +       as part of the Derivative Works; within the Source form or
    +       documentation, if provided along with the Derivative Works; or,
    +       within a display generated by the Derivative Works, if and
    +       wherever such third-party notices normally appear. The contents
    +       of the NOTICE file are for informational purposes only and
    +       do not modify the License. You may add Your own attribution
    +       notices within Derivative Works that You distribute, alongside
    +       or as an addendum to the NOTICE text from the Work, provided
    +       that such additional attribution notices cannot be construed
    +       as modifying the License.
    +
    +   You may add Your own copyright statement to Your modifications and
    +   may provide additional or different license terms and conditions
    +   for use, reproduction, or distribution of Your modifications, or
    +   for any such Derivative Works as a whole, provided Your use,
    +   reproduction, and distribution of the Work otherwise complies with
    +   the conditions stated in this License.
    +
    +5. Submission of Contributions. Unless You explicitly state otherwise,
    +   any Contribution intentionally submitted for inclusion in the Work
    +   by You to the Licensor shall be under the terms and conditions of
    +   this License, without any additional terms or conditions.
    +   Notwithstanding the above, nothing herein shall supersede or modify
    +   the terms of any separate license agreement you may have executed
    +   with Licensor regarding such Contributions.
    +
    +6. Trademarks. This License does not grant permission to use the trade
    +   names, trademarks, service marks, or product names of the Licensor,
    +   except as required for reasonable and customary use in describing the
    +   origin of the Work and reproducing the content of the NOTICE file.
    +
    +7. Disclaimer of Warranty. Unless required by applicable law or
    +   agreed to in writing, Licensor provides the Work (and each
    +   Contributor provides its Contributions) on an "AS IS" BASIS,
    +   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
    +   implied, including, without limitation, any warranties or conditions
    +   of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
    +   PARTICULAR PURPOSE. You are solely responsible for determining the
    +   appropriateness of using or redistributing the Work and assume any
    +   risks associated with Your exercise of permissions under this License.
    +
    +8. Limitation of Liability. In no event and under no legal theory,
    +   whether in tort (including negligence), contract, or otherwise,
    +   unless required by applicable law (such as deliberate and grossly
    +   negligent acts) or agreed to in writing, shall any Contributor be
    +   liable to You for damages, including any direct, indirect, special,
    +   incidental, or consequential damages of any character arising as a
    +   result of this License or out of the use or inability to use the
    +   Work (including but not limited to damages for loss of goodwill,
    +   work stoppage, computer failure or malfunction, or any and all
    +   other commercial damages or losses), even if such Contributor
    +   has been advised of the possibility of such damages.
    +
    +9. Accepting Warranty or Additional Liability. While redistributing
    +   the Work or Derivative Works thereof, You may choose to offer,
    +   and charge a fee for, acceptance of support, warranty, indemnity,
    +   or other liability obligations and/or rights consistent with this
    +   License. However, in accepting such obligations, You may act only
    +   on Your own behalf and on Your sole responsibility, not on behalf
    +   of any other Contributor, and only if You agree to indemnify,
    +   defend, and hold each Contributor harmless for any liability
    +   incurred by, or claims asserted against, such Contributor by reason
    +   of your accepting any such warranty or additional liability.
    +
    +END OF TERMS AND CONDITIONS
    +
    +APPENDIX: How to apply the Apache License to your work.
    +
    +   To apply the Apache License to your work, attach the following
    +   boilerplate notice, with the fields enclosed by brackets "[]"
    +   replaced with your own identifying information. (Don't include
    +   the brackets!)  The text should be enclosed in the appropriate
    +   comment syntax for the file format. We also recommend that a
    +   file or class name and description of purpose be included on the
    +   same "printed page" as the copyright notice for easier
    +   identification within third-party archives.
    +
    +Copyright [yyyy] [name of copyright owner]
    +
    +Licensed under the Apache License, Version 2.0 (the "License");
    +you may not use this file except in compliance with the License.
    +You may obtain a copy of the License at
    +
    +	http://www.apache.org/licenses/LICENSE-2.0
    +
    +Unless required by applicable law or agreed to in writing, software
    +distributed under the License is distributed on an "AS IS" BASIS,
    +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +See the License for the specific language governing permissions and
    +limitations under the License.
    +
    +
  • +
  • +

    Apache License 2.0

    +

    Used by:

    +
                                  Apache License
    -                        Version 2.0 January 2004
    +                        Version 2.0, January 2004
                          http://www.apache.org/licenses/
     
     TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
    @@ -2165,7 +3013,8 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright [yyyy] [name of copyright owner] +Copyright (c) 2016 Alex Crichton +Copyright (c) 2017 The Tokio Authors Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -2184,15 +3033,7 @@

    Used by:

    Apache License 2.0

    Used by:

                                  Apache License
                             Version 2.0, January 2004
    @@ -2382,8 +3223,7 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright (c) 2016 Alex Crichton -Copyright (c) 2017 The Tokio Authors +Copyright (c) 2019 Matthias Einwag Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -2402,7 +3242,7 @@

    Used by:

    Apache License 2.0

    Used by:

                                  Apache License
                             Version 2.0, January 2004
    @@ -2592,7 +3432,7 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright (c) 2019 Matthias Einwag +Copyright 2014 Paho Lurie-Gregg Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -2604,14 +3444,13 @@

    Used by:

    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and -limitations under the License. -
    +limitations under the License.
  • Apache License 2.0

    Used by:

                                  Apache License
                             Version 2.0, January 2004
    @@ -2801,7 +3640,7 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright 2014 Paho Lurie-Gregg +Copyright 2017 The Rust Project Developers Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -2813,13 +3652,15 @@

    Used by:

    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and -limitations under the License.
    +limitations under the License. +
  • Apache License 2.0

    Used by:

                                  Apache License
                             Version 2.0, January 2004
    @@ -3009,7 +3850,7 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright 2017 The Rust Project Developers +Copyright 2017 http-rs authors Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -3028,7 +3869,7 @@

    Used by:

    Apache License 2.0

    Used by:

                                  Apache License
                             Version 2.0, January 2004
    @@ -3218,7 +4059,7 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright 2017 http-rs authors +Copyright 2017 quininer kel Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -3237,7 +4078,7 @@

    Used by:

    Apache License 2.0

    Used by:

                                  Apache License
                             Version 2.0, January 2004
    @@ -3427,7 +4268,7 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright 2017 quininer kel +Copyright 2018 The pin-utils authors Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -4503,6 +5344,7 @@

    Used by:

  • base64 0.21.7
  • base64 0.22.1
  • bitflags 2.11.1
  • +
  • bytes-utils 0.1.4
  • cc 1.2.62
  • cfg-if 1.0.4
  • cmake 0.1.58
  • @@ -4521,15 +5363,18 @@

    Used by:

  • errno 0.3.14
  • event-listener-strategy 0.5.4
  • event-listener 5.4.2
  • +
  • fastrand 2.5.0
  • find-msvc-tools 0.1.9
  • flate2 1.1.9
  • fnv 1.0.7
  • form_urlencoded 1.2.2
  • hashbrown 0.14.5
  • hashbrown 0.15.5
  • +
  • hashbrown 0.16.1
  • hashbrown 0.17.1
  • heck 0.5.0
  • httparse 1.10.1
  • +
  • hyper-rustls 0.27.10
  • idna 1.1.0
  • idna_adapter 1.2.2
  • indexmap 2.14.0
  • @@ -4544,6 +5389,7 @@

    Used by:

  • num-integer 0.1.46
  • num-traits 0.2.19
  • once_cell 1.21.4
  • +
  • openssl-probe 0.2.1
  • parking 2.2.1
  • parking_lot 0.12.5
  • parking_lot_core 0.9.12
  • @@ -4554,9 +5400,12 @@

    Used by:

  • pest_meta 2.8.6
  • pkg-config 0.3.33
  • regex-automata 0.4.14
  • +
  • regex-lite 0.1.9
  • regex-syntax 0.8.10
  • +
  • ring 0.17.14
  • ron 0.8.1
  • rustc_version 0.4.1
  • +
  • rustls-native-certs 0.8.4
  • rustls 0.23.46
  • scopeguard 1.2.0
  • signal-hook-registry 1.4.8
  • @@ -4574,6 +5423,7 @@

    Used by:

  • url 2.5.8
  • uuid 1.23.1
  • version_check 0.9.5
  • +
  • xmlparser 0.13.6
  • yasna 0.6.0
  •                               Apache License
    @@ -5426,6 +6276,7 @@ 

    Used by:

  • md-5 0.10.6
  • opaque-debug 0.3.1
  • polyval 0.6.2
  • +
  • sha1 0.10.6
  • sha2 0.10.9
  • universal-hash 0.5.1
  • @@ -7167,6 +8018,7 @@

    Used by:

    +
  • +
  • +

    ISC License

    +

    Used by:

    + +
    Copyright 2015-2025 Brian Smith.
    +
    +Permission to use, copy, modify, and/or distribute this software for any
    +purpose with or without fee is hereby granted, provided that the above
    +copyright notice and this permission notice appear in all copies.
    +
    +THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
    +WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
    +MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY
    +SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
    +WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION
    +OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN
    +CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
     
  • @@ -8168,6 +9058,63 @@

    Used by:

    OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +
  • +
  • +

    MIT License

    +

    Used by:

    + +
    Copyright (c) 2018-2019 Sean McArthur
    +
    +Permission is hereby granted, free of charge, to any person obtaining a copy
    +of this software and associated documentation files (the "Software"), to deal
    +in the Software without restriction, including without limitation the rights
    +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    +copies of the Software, and to permit persons to whom the Software is
    +furnished to do so, subject to the following conditions:
    +
    +The above copyright notice and this permission notice shall be included in
    +all copies or substantial portions of the Software.
    +
    +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
    +THE SOFTWARE.
    +
    +
    +
  • +
  • +

    MIT License

    +

    Used by:

    + +
    Copyright (c) 2018-2023 Sean McArthur
    +Copyright (c) 2016 Alex Crichton
    +
    +Permission is hereby granted, free of charge, to any person obtaining a copy
    +of this software and associated documentation files (the "Software"), to deal
    +in the Software without restriction, including without limitation the rights
    +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    +copies of the Software, and to permit persons to whom the Software is
    +furnished to do so, subject to the following conditions:
    +
    +The above copyright notice and this permission notice shall be included in
    +all copies or substantial portions of the Software.
    +
    +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
    +THE SOFTWARE.
    +
     
  • @@ -8255,6 +9202,39 @@

    Used by:

    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +
  • +
  • +

    MIT License

    +

    Used by:

    + +
    Copyright (c) 2019 Hyper Contributors
    +
    +Permission is hereby granted, free of charge, to any
    +person obtaining a copy of this software and associated
    +documentation files (the "Software"), to deal in the
    +Software without restriction, including without
    +limitation the rights to use, copy, modify, merge,
    +publish, distribute, sublicense, and/or sell copies of
    +the Software, and to permit persons to whom the Software
    +is furnished to do so, subject to the following
    +conditions:
    +
    +The above copyright notice and this permission notice
    +shall be included in all copies or substantial portions
    +of the Software.
    +
    +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF
    +ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED
    +TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A
    +PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT
    +SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
    +CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
    +OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR
    +IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
    +DEALINGS IN THE SOFTWARE.
     
  • @@ -8572,6 +9552,34 @@

    Used by:

    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. +
  • +
  • +

    MIT License

    +

    Used by:

    + +
    MIT License
    +
    +Copyright (c) 2016 Jerome Froelich
    +
    +Permission is hereby granted, free of charge, to any person obtaining a copy
    +of this software and associated documentation files (the "Software"), to deal
    +in the Software without restriction, including without limitation the rights
    +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    +copies of the Software, and to permit persons to whom the Software is
    +furnished to do so, subject to the following conditions:
    +
    +The above copyright notice and this permission notice shall be included in all
    +copies or substantial portions of the Software.
    +
    +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
    +SOFTWARE.
  • MIT License

    @@ -8781,6 +9789,34 @@

    Used by:

    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. +
  • +
  • +

    MIT License

    +

    Used by:

    + +
    MIT License
    +
    +Copyright (c) 2022 Nugine
    +
    +Permission is hereby granted, free of charge, to any person obtaining a copy
    +of this software and associated documentation files (the "Software"), to deal
    +in the Software without restriction, including without limitation the rights
    +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    +copies of the Software, and to permit persons to whom the Software is
    +furnished to do so, subject to the following conditions:
    +
    +The above copyright notice and this permission notice shall be included in all
    +copies or substantial portions of the Software.
    +
    +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
    +SOFTWARE.
  • MIT License

    @@ -8814,7 +9850,9 @@

    Used by:

    MIT License

    Used by:

    MIT License
     
    @@ -8953,6 +9991,34 @@ 

    Used by:

    OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
    +
  • +
  • +

    MIT License

    +

    Used by:

    + +
    The MIT License (MIT)
    +
    +Copyright (c) 2014 Mathijs van de Nes
    +
    +Permission is hereby granted, free of charge, to any person obtaining a copy
    +of this software and associated documentation files (the "Software"), to deal
    +in the Software without restriction, including without limitation the rights
    +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    +copies of the Software, and to permit persons to whom the Software is
    +furnished to do so, subject to the following conditions:
    +
    +The above copyright notice and this permission notice shall be included in all
    +copies or substantial portions of the Software.
    +
    +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
    +SOFTWARE.
  • MIT License

    @@ -9324,6 +10390,34 @@

    Used by:

    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. +
  • +
  • +

    MIT License

    +

    Used by:

    + +
    © 2016 Bertram Truong
    +© 2021 Kornel Lesiński
    +
    +Permission is hereby granted, free of charge, to any person obtaining a copy
    +of this software and associated documentation files (the "Software"), to deal
    +in the Software without restriction, including without limitation the rights
    +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    +copies of the Software, and to permit persons to whom the Software is
    +furnished to do so, subject to the following conditions:
    +
    +The above copyright notice and this permission notice shall be included in
    +all copies or substantial portions of the Software.
    +
    +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
    +THE SOFTWARE.
    +
  • Unicode License v3

    @@ -9448,6 +10542,7 @@

    zlib License

    Used by:

    Copyright (c) 2024 Orson Peters
     
    diff --git a/SOFTWARE-LICENSE-NOTICES-DEV.html b/SOFTWARE-LICENSE-NOTICES-DEV.html
    index 34cdc961b..63a36061d 100644
    --- a/SOFTWARE-LICENSE-NOTICES-DEV.html
    +++ b/SOFTWARE-LICENSE-NOTICES-DEV.html
    @@ -49,14 +49,14 @@ 

    ExtendDB Software License Notices

    Overview of licenses:

    All license text:

    @@ -1537,25 +1537,216 @@

    Used by:

    Apache License 2.0

    Used by:

    +
                                     Apache License
    +                           Version 2.0, January 2004
    +                        http://www.apache.org/licenses/
    +
    +   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
    +
    +   1. Definitions.
    +
    +      "License" shall mean the terms and conditions for use, reproduction,
    +      and distribution as defined by Sections 1 through 9 of this document.
    +
    +      "Licensor" shall mean the copyright owner or entity authorized by
    +      the copyright owner that is granting the License.
    +
    +      "Legal Entity" shall mean the union of the acting entity and all
    +      other entities that control, are controlled by, or are under common
    +      control with that entity. For the purposes of this definition,
    +      "control" means (i) the power, direct or indirect, to cause the
    +      direction or management of such entity, whether by contract or
    +      otherwise, or (ii) ownership of fifty percent (50%) or more of the
    +      outstanding shares, or (iii) beneficial ownership of such entity.
    +
    +      "You" (or "Your") shall mean an individual or Legal Entity
    +      exercising permissions granted by this License.
    +
    +      "Source" form shall mean the preferred form for making modifications,
    +      including but not limited to software source code, documentation
    +      source, and configuration files.
    +
    +      "Object" form shall mean any form resulting from mechanical
    +      transformation or translation of a Source form, including but
    +      not limited to compiled object code, generated documentation,
    +      and conversions to other media types.
    +
    +      "Work" shall mean the work of authorship, whether in Source or
    +      Object form, made available under the License, as indicated by a
    +      copyright notice that is included in or attached to the work
    +      (an example is provided in the Appendix below).
    +
    +      "Derivative Works" shall mean any work, whether in Source or Object
    +      form, that is based on (or derived from) the Work and for which the
    +      editorial revisions, annotations, elaborations, or other modifications
    +      represent, as a whole, an original work of authorship. For the purposes
    +      of this License, Derivative Works shall not include works that remain
    +      separable from, or merely link (or bind by name) to the interfaces of,
    +      the Work and Derivative Works thereof.
    +
    +      "Contribution" shall mean any work of authorship, including
    +      the original version of the Work and any modifications or additions
    +      to that Work or Derivative Works thereof, that is intentionally
    +      submitted to Licensor for inclusion in the Work by the copyright owner
    +      or by an individual or Legal Entity authorized to submit on behalf of
    +      the copyright owner. For the purposes of this definition, "submitted"
    +      means any form of electronic, verbal, or written communication sent
    +      to the Licensor or its representatives, including but not limited to
    +      communication on electronic mailing lists, source code control systems,
    +      and issue tracking systems that are managed by, or on behalf of, the
    +      Licensor for the purpose of discussing and improving the Work, but
    +      excluding communication that is conspicuously marked or otherwise
    +      designated in writing by the copyright owner as "Not a Contribution."
    +
    +      "Contributor" shall mean Licensor and any individual or Legal Entity
    +      on behalf of whom a Contribution has been received by Licensor and
    +      subsequently incorporated within the Work.
    +
    +   2. Grant of Copyright License. Subject to the terms and conditions of
    +      this License, each Contributor hereby grants to You a perpetual,
    +      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +      copyright license to reproduce, prepare Derivative Works of,
    +      publicly display, publicly perform, sublicense, and distribute the
    +      Work and such Derivative Works in Source or Object form.
    +
    +   3. Grant of Patent License. Subject to the terms and conditions of
    +      this License, each Contributor hereby grants to You a perpetual,
    +      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +      (except as stated in this section) patent license to make, have made,
    +      use, offer to sell, sell, import, and otherwise transfer the Work,
    +      where such license applies only to those patent claims licensable
    +      by such Contributor that are necessarily infringed by their
    +      Contribution(s) alone or by combination of their Contribution(s)
    +      with the Work to which such Contribution(s) was submitted. If You
    +      institute patent litigation against any entity (including a
    +      cross-claim or counterclaim in a lawsuit) alleging that the Work
    +      or a Contribution incorporated within the Work constitutes direct
    +      or contributory patent infringement, then any patent licenses
    +      granted to You under this License for that Work shall terminate
    +      as of the date such litigation is filed.
    +
    +   4. Redistribution. You may reproduce and distribute copies of the
    +      Work or Derivative Works thereof in any medium, with or without
    +      modifications, and in Source or Object form, provided that You
    +      meet the following conditions:
    +
    +      (a) You must give any other recipients of the Work or
    +          Derivative Works a copy of this License; and
    +
    +      (b) You must cause any modified files to carry prominent notices
    +          stating that You changed the files; and
    +
    +      (c) You must retain, in the Source form of any Derivative Works
    +          that You distribute, all copyright, patent, trademark, and
    +          attribution notices from the Source form of the Work,
    +          excluding those notices that do not pertain to any part of
    +          the Derivative Works; and
    +
    +      (d) If the Work includes a "NOTICE" text file as part of its
    +          distribution, then any Derivative Works that You distribute must
    +          include a readable copy of the attribution notices contained
    +          within such NOTICE file, excluding those notices that do not
    +          pertain to any part of the Derivative Works, in at least one
    +          of the following places: within a NOTICE text file distributed
    +          as part of the Derivative Works; within the Source form or
    +          documentation, if provided along with the Derivative Works; or,
    +          within a display generated by the Derivative Works, if and
    +          wherever such third-party notices normally appear. The contents
    +          of the NOTICE file are for informational purposes only and
    +          do not modify the License. You may add Your own attribution
    +          notices within Derivative Works that You distribute, alongside
    +          or as an addendum to the NOTICE text from the Work, provided
    +          that such additional attribution notices cannot be construed
    +          as modifying the License.
    +
    +      You may add Your own copyright statement to Your modifications and
    +      may provide additional or different license terms and conditions
    +      for use, reproduction, or distribution of Your modifications, or
    +      for any such Derivative Works as a whole, provided Your use,
    +      reproduction, and distribution of the Work otherwise complies with
    +      the conditions stated in this License.
    +
    +   5. Submission of Contributions. Unless You explicitly state otherwise,
    +      any Contribution intentionally submitted for inclusion in the Work
    +      by You to the Licensor shall be under the terms and conditions of
    +      this License, without any additional terms or conditions.
    +      Notwithstanding the above, nothing herein shall supersede or modify
    +      the terms of any separate license agreement you may have executed
    +      with Licensor regarding such Contributions.
    +
    +   6. Trademarks. This License does not grant permission to use the trade
    +      names, trademarks, service marks, or product names of the Licensor,
    +      except as required for reasonable and customary use in describing the
    +      origin of the Work and reproducing the content of the NOTICE file.
    +
    +   7. Disclaimer of Warranty. Unless required by applicable law or
    +      agreed to in writing, Licensor provides the Work (and each
    +      Contributor provides its Contributions) on an "AS IS" BASIS,
    +      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
    +      implied, including, without limitation, any warranties or conditions
    +      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
    +      PARTICULAR PURPOSE. You are solely responsible for determining the
    +      appropriateness of using or redistributing the Work and assume any
    +      risks associated with Your exercise of permissions under this License.
    +
    +   8. Limitation of Liability. In no event and under no legal theory,
    +      whether in tort (including negligence), contract, or otherwise,
    +      unless required by applicable law (such as deliberate and grossly
    +      negligent acts) or agreed to in writing, shall any Contributor be
    +      liable to You for damages, including any direct, indirect, special,
    +      incidental, or consequential damages of any character arising as a
    +      result of this License or out of the use or inability to use the
    +      Work (including but not limited to damages for loss of goodwill,
    +      work stoppage, computer failure or malfunction, or any and all
    +      other commercial damages or losses), even if such Contributor
    +      has been advised of the possibility of such damages.
    +
    +   9. Accepting Warranty or Additional Liability. While redistributing
    +      the Work or Derivative Works thereof, You may choose to offer,
    +      and charge a fee for, acceptance of support, warranty, indemnity,
    +      or other liability obligations and/or rights consistent with this
    +      License. However, in accepting such obligations, You may act only
    +      on Your own behalf and on Your sole responsibility, not on behalf
    +      of any other Contributor, and only if You agree to indemnify,
    +      defend, and hold each Contributor harmless for any liability
    +      incurred by, or claims asserted against, such Contributor by reason
    +      of your accepting any such warranty or additional liability.
    +
    +   END OF TERMS AND CONDITIONS
    +
    +   APPENDIX: How to apply the Apache License to your work.
    +
    +      To apply the Apache License to your work, attach the following
    +      boilerplate notice, with the fields enclosed by brackets "[]"
    +      replaced with your own identifying information. (Don't include
    +      the brackets!)  The text should be enclosed in the appropriate
    +      comment syntax for the file format. We also recommend that a
    +      file or class name and description of purpose be included on the
    +      same "printed page" as the copyright notice for easier
    +      identification within third-party archives.
    +
    +   Copyright [yyyy] [name of copyright owner]
    +
    +   Licensed under the Apache License, Version 2.0 (the "License");
    +   you may not use this file except in compliance with the License.
    +   You may obtain a copy of the License at
    +
    +       http://www.apache.org/licenses/LICENSE-2.0
    +
    +   Unless required by applicable law or agreed to in writing, software
    +   distributed under the License is distributed on an "AS IS" BASIS,
    +   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +   See the License for the specific language governing permissions and
    +   limitations under the License.
    +
    +
  • +
  • +

    Apache License 2.0

    +

    Used by:

    +
                                     Apache License
                                Version 2.0, January 2004
    @@ -1745,30 +1936,687 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. - Copyright {yyyy} {name of copyright owner} + Copyright 2017 Juniper Networks, Inc. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +
    +
  • +
  • +

    Apache License 2.0

    +

    Used by:

    + +
                                     Apache License
    +                           Version 2.0, January 2004
    +                        http://www.apache.org/licenses/
    +
    +   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
    +
    +   1. Definitions.
    +
    +      "License" shall mean the terms and conditions for use, reproduction,
    +      and distribution as defined by Sections 1 through 9 of this document.
    +
    +      "Licensor" shall mean the copyright owner or entity authorized by
    +      the copyright owner that is granting the License.
    +
    +      "Legal Entity" shall mean the union of the acting entity and all
    +      other entities that control, are controlled by, or are under common
    +      control with that entity. For the purposes of this definition,
    +      "control" means (i) the power, direct or indirect, to cause the
    +      direction or management of such entity, whether by contract or
    +      otherwise, or (ii) ownership of fifty percent (50%) or more of the
    +      outstanding shares, or (iii) beneficial ownership of such entity.
    +
    +      "You" (or "Your") shall mean an individual or Legal Entity
    +      exercising permissions granted by this License.
    +
    +      "Source" form shall mean the preferred form for making modifications,
    +      including but not limited to software source code, documentation
    +      source, and configuration files.
    +
    +      "Object" form shall mean any form resulting from mechanical
    +      transformation or translation of a Source form, including but
    +      not limited to compiled object code, generated documentation,
    +      and conversions to other media types.
    +
    +      "Work" shall mean the work of authorship, whether in Source or
    +      Object form, made available under the License, as indicated by a
    +      copyright notice that is included in or attached to the work
    +      (an example is provided in the Appendix below).
    +
    +      "Derivative Works" shall mean any work, whether in Source or Object
    +      form, that is based on (or derived from) the Work and for which the
    +      editorial revisions, annotations, elaborations, or other modifications
    +      represent, as a whole, an original work of authorship. For the purposes
    +      of this License, Derivative Works shall not include works that remain
    +      separable from, or merely link (or bind by name) to the interfaces of,
    +      the Work and Derivative Works thereof.
    +
    +      "Contribution" shall mean any work of authorship, including
    +      the original version of the Work and any modifications or additions
    +      to that Work or Derivative Works thereof, that is intentionally
    +      submitted to Licensor for inclusion in the Work by the copyright owner
    +      or by an individual or Legal Entity authorized to submit on behalf of
    +      the copyright owner. For the purposes of this definition, "submitted"
    +      means any form of electronic, verbal, or written communication sent
    +      to the Licensor or its representatives, including but not limited to
    +      communication on electronic mailing lists, source code control systems,
    +      and issue tracking systems that are managed by, or on behalf of, the
    +      Licensor for the purpose of discussing and improving the Work, but
    +      excluding communication that is conspicuously marked or otherwise
    +      designated in writing by the copyright owner as "Not a Contribution."
    +
    +      "Contributor" shall mean Licensor and any individual or Legal Entity
    +      on behalf of whom a Contribution has been received by Licensor and
    +      subsequently incorporated within the Work.
    +
    +   2. Grant of Copyright License. Subject to the terms and conditions of
    +      this License, each Contributor hereby grants to You a perpetual,
    +      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +      copyright license to reproduce, prepare Derivative Works of,
    +      publicly display, publicly perform, sublicense, and distribute the
    +      Work and such Derivative Works in Source or Object form.
    +
    +   3. Grant of Patent License. Subject to the terms and conditions of
    +      this License, each Contributor hereby grants to You a perpetual,
    +      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +      (except as stated in this section) patent license to make, have made,
    +      use, offer to sell, sell, import, and otherwise transfer the Work,
    +      where such license applies only to those patent claims licensable
    +      by such Contributor that are necessarily infringed by their
    +      Contribution(s) alone or by combination of their Contribution(s)
    +      with the Work to which such Contribution(s) was submitted. If You
    +      institute patent litigation against any entity (including a
    +      cross-claim or counterclaim in a lawsuit) alleging that the Work
    +      or a Contribution incorporated within the Work constitutes direct
    +      or contributory patent infringement, then any patent licenses
    +      granted to You under this License for that Work shall terminate
    +      as of the date such litigation is filed.
    +
    +   4. Redistribution. You may reproduce and distribute copies of the
    +      Work or Derivative Works thereof in any medium, with or without
    +      modifications, and in Source or Object form, provided that You
    +      meet the following conditions:
    +
    +      (a) You must give any other recipients of the Work or
    +          Derivative Works a copy of this License; and
    +
    +      (b) You must cause any modified files to carry prominent notices
    +          stating that You changed the files; and
    +
    +      (c) You must retain, in the Source form of any Derivative Works
    +          that You distribute, all copyright, patent, trademark, and
    +          attribution notices from the Source form of the Work,
    +          excluding those notices that do not pertain to any part of
    +          the Derivative Works; and
    +
    +      (d) If the Work includes a "NOTICE" text file as part of its
    +          distribution, then any Derivative Works that You distribute must
    +          include a readable copy of the attribution notices contained
    +          within such NOTICE file, excluding those notices that do not
    +          pertain to any part of the Derivative Works, in at least one
    +          of the following places: within a NOTICE text file distributed
    +          as part of the Derivative Works; within the Source form or
    +          documentation, if provided along with the Derivative Works; or,
    +          within a display generated by the Derivative Works, if and
    +          wherever such third-party notices normally appear. The contents
    +          of the NOTICE file are for informational purposes only and
    +          do not modify the License. You may add Your own attribution
    +          notices within Derivative Works that You distribute, alongside
    +          or as an addendum to the NOTICE text from the Work, provided
    +          that such additional attribution notices cannot be construed
    +          as modifying the License.
    +
    +      You may add Your own copyright statement to Your modifications and
    +      may provide additional or different license terms and conditions
    +      for use, reproduction, or distribution of Your modifications, or
    +      for any such Derivative Works as a whole, provided Your use,
    +      reproduction, and distribution of the Work otherwise complies with
    +      the conditions stated in this License.
    +
    +   5. Submission of Contributions. Unless You explicitly state otherwise,
    +      any Contribution intentionally submitted for inclusion in the Work
    +      by You to the Licensor shall be under the terms and conditions of
    +      this License, without any additional terms or conditions.
    +      Notwithstanding the above, nothing herein shall supersede or modify
    +      the terms of any separate license agreement you may have executed
    +      with Licensor regarding such Contributions.
    +
    +   6. Trademarks. This License does not grant permission to use the trade
    +      names, trademarks, service marks, or product names of the Licensor,
    +      except as required for reasonable and customary use in describing the
    +      origin of the Work and reproducing the content of the NOTICE file.
    +
    +   7. Disclaimer of Warranty. Unless required by applicable law or
    +      agreed to in writing, Licensor provides the Work (and each
    +      Contributor provides its Contributions) on an "AS IS" BASIS,
    +      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
    +      implied, including, without limitation, any warranties or conditions
    +      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
    +      PARTICULAR PURPOSE. You are solely responsible for determining the
    +      appropriateness of using or redistributing the Work and assume any
    +      risks associated with Your exercise of permissions under this License.
    +
    +   8. Limitation of Liability. In no event and under no legal theory,
    +      whether in tort (including negligence), contract, or otherwise,
    +      unless required by applicable law (such as deliberate and grossly
    +      negligent acts) or agreed to in writing, shall any Contributor be
    +      liable to You for damages, including any direct, indirect, special,
    +      incidental, or consequential damages of any character arising as a
    +      result of this License or out of the use or inability to use the
    +      Work (including but not limited to damages for loss of goodwill,
    +      work stoppage, computer failure or malfunction, or any and all
    +      other commercial damages or losses), even if such Contributor
    +      has been advised of the possibility of such damages.
    +
    +   9. Accepting Warranty or Additional Liability. While redistributing
    +      the Work or Derivative Works thereof, You may choose to offer,
    +      and charge a fee for, acceptance of support, warranty, indemnity,
    +      or other liability obligations and/or rights consistent with this
    +      License. However, in accepting such obligations, You may act only
    +      on Your own behalf and on Your sole responsibility, not on behalf
    +      of any other Contributor, and only if You agree to indemnify,
    +      defend, and hold each Contributor harmless for any liability
    +      incurred by, or claims asserted against, such Contributor by reason
    +      of your accepting any such warranty or additional liability.
    +
    +   END OF TERMS AND CONDITIONS
    +
    +   APPENDIX: How to apply the Apache License to your work.
    +
    +      To apply the Apache License to your work, attach the following
    +      boilerplate notice, with the fields enclosed by brackets "{}"
    +      replaced with your own identifying information. (Don't include
    +      the brackets!)  The text should be enclosed in the appropriate
    +      comment syntax for the file format. We also recommend that a
    +      file or class name and description of purpose be included on the
    +      same "printed page" as the copyright notice for easier
    +      identification within third-party archives.
    +
    +   Copyright {yyyy} {name of copyright owner}
    +
    +   Licensed under the Apache License, Version 2.0 (the "License");
    +   you may not use this file except in compliance with the License.
    +   You may obtain a copy of the License at
    +
    +       http://www.apache.org/licenses/LICENSE-2.0
    +
    +   Unless required by applicable law or agreed to in writing, software
    +   distributed under the License is distributed on an "AS IS" BASIS,
    +   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +   See the License for the specific language governing permissions and
    +   limitations under the License.
    +
    +
    +
  • +
  • +

    Apache License 2.0

    +

    Used by:

    + +
                                    Apache License
    +                           Version 2.0, January 2004
    +                        http://www.apache.org/licenses/
    +
    +   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
    +
    +   1. Definitions.
    +
    +      "License" shall mean the terms and conditions for use, reproduction,
    +      and distribution as defined by Sections 1 through 9 of this document.
    +
    +      "Licensor" shall mean the copyright owner or entity authorized by
    +      the copyright owner that is granting the License.
    +
    +      "Legal Entity" shall mean the union of the acting entity and all
    +      other entities that control, are controlled by, or are under common
    +      control with that entity. For the purposes of this definition,
    +      "control" means (i) the power, direct or indirect, to cause the
    +      direction or management of such entity, whether by contract or
    +      otherwise, or (ii) ownership of fifty percent (50%) or more of the
    +      outstanding shares, or (iii) beneficial ownership of such entity.
    +
    +      "You" (or "Your") shall mean an individual or Legal Entity
    +      exercising permissions granted by this License.
    +
    +      "Source" form shall mean the preferred form for making modifications,
    +      including but not limited to software source code, documentation
    +      source, and configuration files.
    +
    +      "Object" form shall mean any form resulting from mechanical
    +      transformation or translation of a Source form, including but
    +      not limited to compiled object code, generated documentation,
    +      and conversions to other media types.
    +
    +      "Work" shall mean the work of authorship, whether in Source or
    +      Object form, made available under the License, as indicated by a
    +      copyright notice that is included in or attached to the work
    +      (an example is provided in the Appendix below).
    +
    +      "Derivative Works" shall mean any work, whether in Source or Object
    +      form, that is based on (or derived from) the Work and for which the
    +      editorial revisions, annotations, elaborations, or other modifications
    +      represent, as a whole, an original work of authorship. For the purposes
    +      of this License, Derivative Works shall not include works that remain
    +      separable from, or merely link (or bind by name) to the interfaces of,
    +      the Work and Derivative Works thereof.
    +
    +      "Contribution" shall mean any work of authorship, including
    +      the original version of the Work and any modifications or additions
    +      to that Work or Derivative Works thereof, that is intentionally
    +      submitted to Licensor for inclusion in the Work by the copyright owner
    +      or by an individual or Legal Entity authorized to submit on behalf of
    +      the copyright owner. For the purposes of this definition, "submitted"
    +      means any form of electronic, verbal, or written communication sent
    +      to the Licensor or its representatives, including but not limited to
    +      communication on electronic mailing lists, source code control systems,
    +      and issue tracking systems that are managed by, or on behalf of, the
    +      Licensor for the purpose of discussing and improving the Work, but
    +      excluding communication that is conspicuously marked or otherwise
    +      designated in writing by the copyright owner as "Not a Contribution."
    +
    +      "Contributor" shall mean Licensor and any individual or Legal Entity
    +      on behalf of whom a Contribution has been received by Licensor and
    +      subsequently incorporated within the Work.
    +
    +   2. Grant of Copyright License. Subject to the terms and conditions of
    +      this License, each Contributor hereby grants to You a perpetual,
    +      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +      copyright license to reproduce, prepare Derivative Works of,
    +      publicly display, publicly perform, sublicense, and distribute the
    +      Work and such Derivative Works in Source or Object form.
    +
    +   3. Grant of Patent License. Subject to the terms and conditions of
    +      this License, each Contributor hereby grants to You a perpetual,
    +      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +      (except as stated in this section) patent license to make, have made,
    +      use, offer to sell, sell, import, and otherwise transfer the Work,
    +      where such license applies only to those patent claims licensable
    +      by such Contributor that are necessarily infringed by their
    +      Contribution(s) alone or by combination of their Contribution(s)
    +      with the Work to which such Contribution(s) was submitted. If You
    +      institute patent litigation against any entity (including a
    +      cross-claim or counterclaim in a lawsuit) alleging that the Work
    +      or a Contribution incorporated within the Work constitutes direct
    +      or contributory patent infringement, then any patent licenses
    +      granted to You under this License for that Work shall terminate
    +      as of the date such litigation is filed.
    +
    +   4. Redistribution. You may reproduce and distribute copies of the
    +      Work or Derivative Works thereof in any medium, with or without
    +      modifications, and in Source or Object form, provided that You
    +      meet the following conditions:
    +
    +      (a) You must give any other recipients of the Work or
    +          Derivative Works a copy of this License; and
    +
    +      (b) You must cause any modified files to carry prominent notices
    +          stating that You changed the files; and
    +
    +      (c) You must retain, in the Source form of any Derivative Works
    +          that You distribute, all copyright, patent, trademark, and
    +          attribution notices from the Source form of the Work,
    +          excluding those notices that do not pertain to any part of
    +          the Derivative Works; and
    +
    +      (d) If the Work includes a "NOTICE" text file as part of its
    +          distribution, then any Derivative Works that You distribute must
    +          include a readable copy of the attribution notices contained
    +          within such NOTICE file, excluding those notices that do not
    +          pertain to any part of the Derivative Works, in at least one
    +          of the following places: within a NOTICE text file distributed
    +          as part of the Derivative Works; within the Source form or
    +          documentation, if provided along with the Derivative Works; or,
    +          within a display generated by the Derivative Works, if and
    +          wherever such third-party notices normally appear. The contents
    +          of the NOTICE file are for informational purposes only and
    +          do not modify the License. You may add Your own attribution
    +          notices within Derivative Works that You distribute, alongside
    +          or as an addendum to the NOTICE text from the Work, provided
    +          that such additional attribution notices cannot be construed
    +          as modifying the License.
    +
    +      You may add Your own copyright statement to Your modifications and
    +      may provide additional or different license terms and conditions
    +      for use, reproduction, or distribution of Your modifications, or
    +      for any such Derivative Works as a whole, provided Your use,
    +      reproduction, and distribution of the Work otherwise complies with
    +      the conditions stated in this License.
    +
    +   5. Submission of Contributions. Unless You explicitly state otherwise,
    +      any Contribution intentionally submitted for inclusion in the Work
    +      by You to the Licensor shall be under the terms and conditions of
    +      this License, without any additional terms or conditions.
    +      Notwithstanding the above, nothing herein shall supersede or modify
    +      the terms of any separate license agreement you may have executed
    +      with Licensor regarding such Contributions.
    +
    +   6. Trademarks. This License does not grant permission to use the trade
    +      names, trademarks, service marks, or product names of the Licensor,
    +      except as required for reasonable and customary use in describing the
    +      origin of the Work and reproducing the content of the NOTICE file.
    +
    +   7. Disclaimer of Warranty. Unless required by applicable law or
    +      agreed to in writing, Licensor provides the Work (and each
    +      Contributor provides its Contributions) on an "AS IS" BASIS,
    +      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
    +      implied, including, without limitation, any warranties or conditions
    +      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
    +      PARTICULAR PURPOSE. You are solely responsible for determining the
    +      appropriateness of using or redistributing the Work and assume any
    +      risks associated with Your exercise of permissions under this License.
    +
    +   8. Limitation of Liability. In no event and under no legal theory,
    +      whether in tort (including negligence), contract, or otherwise,
    +      unless required by applicable law (such as deliberate and grossly
    +      negligent acts) or agreed to in writing, shall any Contributor be
    +      liable to You for damages, including any direct, indirect, special,
    +      incidental, or consequential damages of any character arising as a
    +      result of this License or out of the use or inability to use the
    +      Work (including but not limited to damages for loss of goodwill,
    +      work stoppage, computer failure or malfunction, or any and all
    +      other commercial damages or losses), even if such Contributor
    +      has been advised of the possibility of such damages.
    +
    +   9. Accepting Warranty or Additional Liability. While redistributing
    +      the Work or Derivative Works thereof, You may choose to offer,
    +      and charge a fee for, acceptance of support, warranty, indemnity,
    +      or other liability obligations and/or rights consistent with this
    +      License. However, in accepting such obligations, You may act only
    +      on Your own behalf and on Your sole responsibility, not on behalf
    +      of any other Contributor, and only if You agree to indemnify,
    +      defend, and hold each Contributor harmless for any liability
    +      incurred by, or claims asserted against, such Contributor by reason
    +      of your accepting any such warranty or additional liability.
    +
    +   END OF TERMS AND CONDITIONS
    +
    +   APPENDIX: How to apply the Apache License to your work.
    +
    +      To apply the Apache License to your work, attach the following
    +      boilerplate notice, with the fields enclosed by brackets "{}"
    +      replaced with your own identifying information. (Don't include
    +      the brackets!)  The text should be enclosed in the appropriate
    +      comment syntax for the file format. We also recommend that a
    +      file or class name and description of purpose be included on the
    +      same "printed page" as the copyright notice for easier
    +      identification within third-party archives.
    +
    +   Copyright 2018-2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.
    +
    +   Licensed under the Apache License, Version 2.0 (the "License");
    +   you may not use this file except in compliance with the License.
    +   You may obtain a copy of the License at
    +
    +       http://www.apache.org/licenses/LICENSE-2.0
    +
    +   Unless required by applicable law or agreed to in writing, software
    +   distributed under the License is distributed on an "AS IS" BASIS,
    +   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +   See the License for the specific language governing permissions and
    +   limitations under the License.
    +
    +
    +
  • +
  • +

    Apache License 2.0

    +

    Used by:

    + +
                                  Apache License
    +                        Version 2.0 January 2004
    +                     http://www.apache.org/licenses/
    +
    +TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
    +
    +1. Definitions.
    +
    +   "License" shall mean the terms and conditions for use, reproduction,
    +   and distribution as defined by Sections 1 through 9 of this document.
    +
    +   "Licensor" shall mean the copyright owner or entity authorized by
    +   the copyright owner that is granting the License.
    +
    +   "Legal Entity" shall mean the union of the acting entity and all
    +   other entities that control, are controlled by, or are under common
    +   control with that entity. For the purposes of this definition,
    +   "control" means (i) the power, direct or indirect, to cause the
    +   direction or management of such entity, whether by contract or
    +   otherwise, or (ii) ownership of fifty percent (50%) or more of the
    +   outstanding shares, or (iii) beneficial ownership of such entity.
    +
    +   "You" (or "Your") shall mean an individual or Legal Entity
    +   exercising permissions granted by this License.
    +
    +   "Source" form shall mean the preferred form for making modifications,
    +   including but not limited to software source code, documentation
    +   source, and configuration files.
    +
    +   "Object" form shall mean any form resulting from mechanical
    +   transformation or translation of a Source form, including but
    +   not limited to compiled object code, generated documentation,
    +   and conversions to other media types.
    +
    +   "Work" shall mean the work of authorship, whether in Source or
    +   Object form, made available under the License, as indicated by a
    +   copyright notice that is included in or attached to the work
    +   (an example is provided in the Appendix below).
    +
    +   "Derivative Works" shall mean any work, whether in Source or Object
    +   form, that is based on (or derived from) the Work and for which the
    +   editorial revisions, annotations, elaborations, or other modifications
    +   represent, as a whole, an original work of authorship. For the purposes
    +   of this License, Derivative Works shall not include works that remain
    +   separable from, or merely link (or bind by name) to the interfaces of,
    +   the Work and Derivative Works thereof.
    +
    +   "Contribution" shall mean any work of authorship, including
    +   the original version of the Work and any modifications or additions
    +   to that Work or Derivative Works thereof, that is intentionally
    +   submitted to Licensor for inclusion in the Work by the copyright owner
    +   or by an individual or Legal Entity authorized to submit on behalf of
    +   the copyright owner. For the purposes of this definition, "submitted"
    +   means any form of electronic, verbal, or written communication sent
    +   to the Licensor or its representatives, including but not limited to
    +   communication on electronic mailing lists, source code control systems,
    +   and issue tracking systems that are managed by, or on behalf of, the
    +   Licensor for the purpose of discussing and improving the Work, but
    +   excluding communication that is conspicuously marked or otherwise
    +   designated in writing by the copyright owner as "Not a Contribution."
    +
    +   "Contributor" shall mean Licensor and any individual or Legal Entity
    +   on behalf of whom a Contribution has been received by Licensor and
    +   subsequently incorporated within the Work.
    +
    +2. Grant of Copyright License. Subject to the terms and conditions of
    +   this License, each Contributor hereby grants to You a perpetual,
    +   worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +   copyright license to reproduce, prepare Derivative Works of,
    +   publicly display, publicly perform, sublicense, and distribute the
    +   Work and such Derivative Works in Source or Object form.
    +
    +3. Grant of Patent License. Subject to the terms and conditions of
    +   this License, each Contributor hereby grants to You a perpetual,
    +   worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +   (except as stated in this section) patent license to make, have made,
    +   use, offer to sell, sell, import, and otherwise transfer the Work,
    +   where such license applies only to those patent claims licensable
    +   by such Contributor that are necessarily infringed by their
    +   Contribution(s) alone or by combination of their Contribution(s)
    +   with the Work to which such Contribution(s) was submitted. If You
    +   institute patent litigation against any entity (including a
    +   cross-claim or counterclaim in a lawsuit) alleging that the Work
    +   or a Contribution incorporated within the Work constitutes direct
    +   or contributory patent infringement, then any patent licenses
    +   granted to You under this License for that Work shall terminate
    +   as of the date such litigation is filed.
    +
    +4. Redistribution. You may reproduce and distribute copies of the
    +   Work or Derivative Works thereof in any medium, with or without
    +   modifications, and in Source or Object form, provided that You
    +   meet the following conditions:
    +
    +   (a) You must give any other recipients of the Work or
    +       Derivative Works a copy of this License; and
    +
    +   (b) You must cause any modified files to carry prominent notices
    +       stating that You changed the files; and
    +
    +   (c) You must retain, in the Source form of any Derivative Works
    +       that You distribute, all copyright, patent, trademark, and
    +       attribution notices from the Source form of the Work,
    +       excluding those notices that do not pertain to any part of
    +       the Derivative Works; and
    +
    +   (d) If the Work includes a "NOTICE" text file as part of its
    +       distribution, then any Derivative Works that You distribute must
    +       include a readable copy of the attribution notices contained
    +       within such NOTICE file, excluding those notices that do not
    +       pertain to any part of the Derivative Works, in at least one
    +       of the following places: within a NOTICE text file distributed
    +       as part of the Derivative Works; within the Source form or
    +       documentation, if provided along with the Derivative Works; or,
    +       within a display generated by the Derivative Works, if and
    +       wherever such third-party notices normally appear. The contents
    +       of the NOTICE file are for informational purposes only and
    +       do not modify the License. You may add Your own attribution
    +       notices within Derivative Works that You distribute, alongside
    +       or as an addendum to the NOTICE text from the Work, provided
    +       that such additional attribution notices cannot be construed
    +       as modifying the License.
    +
    +   You may add Your own copyright statement to Your modifications and
    +   may provide additional or different license terms and conditions
    +   for use, reproduction, or distribution of Your modifications, or
    +   for any such Derivative Works as a whole, provided Your use,
    +   reproduction, and distribution of the Work otherwise complies with
    +   the conditions stated in this License.
    +
    +5. Submission of Contributions. Unless You explicitly state otherwise,
    +   any Contribution intentionally submitted for inclusion in the Work
    +   by You to the Licensor shall be under the terms and conditions of
    +   this License, without any additional terms or conditions.
    +   Notwithstanding the above, nothing herein shall supersede or modify
    +   the terms of any separate license agreement you may have executed
    +   with Licensor regarding such Contributions.
    +
    +6. Trademarks. This License does not grant permission to use the trade
    +   names, trademarks, service marks, or product names of the Licensor,
    +   except as required for reasonable and customary use in describing the
    +   origin of the Work and reproducing the content of the NOTICE file.
    +
    +7. Disclaimer of Warranty. Unless required by applicable law or
    +   agreed to in writing, Licensor provides the Work (and each
    +   Contributor provides its Contributions) on an "AS IS" BASIS,
    +   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
    +   implied, including, without limitation, any warranties or conditions
    +   of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
    +   PARTICULAR PURPOSE. You are solely responsible for determining the
    +   appropriateness of using or redistributing the Work and assume any
    +   risks associated with Your exercise of permissions under this License.
    +
    +8. Limitation of Liability. In no event and under no legal theory,
    +   whether in tort (including negligence), contract, or otherwise,
    +   unless required by applicable law (such as deliberate and grossly
    +   negligent acts) or agreed to in writing, shall any Contributor be
    +   liable to You for damages, including any direct, indirect, special,
    +   incidental, or consequential damages of any character arising as a
    +   result of this License or out of the use or inability to use the
    +   Work (including but not limited to damages for loss of goodwill,
    +   work stoppage, computer failure or malfunction, or any and all
    +   other commercial damages or losses), even if such Contributor
    +   has been advised of the possibility of such damages.
    +
    +9. Accepting Warranty or Additional Liability. While redistributing
    +   the Work or Derivative Works thereof, You may choose to offer,
    +   and charge a fee for, acceptance of support, warranty, indemnity,
    +   or other liability obligations and/or rights consistent with this
    +   License. However, in accepting such obligations, You may act only
    +   on Your own behalf and on Your sole responsibility, not on behalf
    +   of any other Contributor, and only if You agree to indemnify,
    +   defend, and hold each Contributor harmless for any liability
    +   incurred by, or claims asserted against, such Contributor by reason
    +   of your accepting any such warranty or additional liability.
    +
    +END OF TERMS AND CONDITIONS
    +
    +APPENDIX: How to apply the Apache License to your work.
     
    -   Licensed under the Apache License, Version 2.0 (the "License");
    -   you may not use this file except in compliance with the License.
    -   You may obtain a copy of the License at
    +   To apply the Apache License to your work, attach the following
    +   boilerplate notice, with the fields enclosed by brackets "[]"
    +   replaced with your own identifying information. (Don't include
    +   the brackets!)  The text should be enclosed in the appropriate
    +   comment syntax for the file format. We also recommend that a
    +   file or class name and description of purpose be included on the
    +   same "printed page" as the copyright notice for easier
    +   identification within third-party archives.
     
    -       http://www.apache.org/licenses/LICENSE-2.0
    +Copyright [yyyy] [name of copyright owner]
     
    -   Unless required by applicable law or agreed to in writing, software
    -   distributed under the License is distributed on an "AS IS" BASIS,
    -   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    -   See the License for the specific language governing permissions and
    -   limitations under the License.
    +Licensed under the Apache License, Version 2.0 (the "License");
    +you may not use this file except in compliance with the License.
    +You may obtain a copy of the License at
    +
    +	http://www.apache.org/licenses/LICENSE-2.0
     
    +Unless required by applicable law or agreed to in writing, software
    +distributed under the License is distributed on an "AS IS" BASIS,
    +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +See the License for the specific language governing permissions and
    +limitations under the License.
     
  • Apache License 2.0

    Used by:

                                  Apache License
    -                        Version 2.0 January 2004
    +                        Version 2.0, January 2004
                          http://www.apache.org/licenses/
     
     TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
    @@ -1955,7 +2803,8 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright [yyyy] [name of copyright owner] +Copyright (c) 2016 Alex Crichton +Copyright (c) 2017 The Tokio Authors Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -1974,15 +2823,7 @@

    Used by:

    Apache License 2.0

    Used by:

                                  Apache License
                             Version 2.0, January 2004
    @@ -2172,8 +3013,7 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright (c) 2016 Alex Crichton -Copyright (c) 2017 The Tokio Authors +Copyright (c) 2019 Matthias Einwag Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -2192,7 +3032,7 @@

    Used by:

    Apache License 2.0

    Used by:

                                  Apache License
                             Version 2.0, January 2004
    @@ -2382,7 +3222,7 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright (c) 2019 Matthias Einwag +Copyright 2014 Paho Lurie-Gregg Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -2394,14 +3234,13 @@

    Used by:

    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and -limitations under the License. -
    +limitations under the License.
  • Apache License 2.0

    Used by:

                                  Apache License
                             Version 2.0, January 2004
    @@ -2591,7 +3430,7 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright 2014 Paho Lurie-Gregg +Copyright 2017 The Rust Project Developers Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -2603,13 +3442,15 @@

    Used by:

    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and -limitations under the License.
    +limitations under the License. +
  • Apache License 2.0

    Used by:

                                  Apache License
                             Version 2.0, January 2004
    @@ -2799,7 +3640,7 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright 2017 The Rust Project Developers +Copyright 2017 http-rs authors Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -2818,7 +3659,7 @@

    Used by:

    Apache License 2.0

    Used by:

                                  Apache License
                             Version 2.0, January 2004
    @@ -3008,7 +3849,7 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright 2017 http-rs authors +Copyright 2017 quininer kel Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -3027,7 +3868,7 @@

    Used by:

    Apache License 2.0

    Used by:

                                  Apache License
                             Version 2.0, January 2004
    @@ -3217,7 +4058,7 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright 2017 quininer kel +Copyright 2018 The pin-utils authors Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -4083,6 +4924,7 @@

    Used by:

  • base64 0.21.7
  • base64 0.22.1
  • bitflags 2.11.1
  • +
  • bytes-utils 0.1.4
  • cc 1.2.62
  • cfg-if 1.0.4
  • cmake 0.1.58
  • @@ -4100,15 +4942,18 @@

    Used by:

  • errno 0.3.14
  • event-listener-strategy 0.5.4
  • event-listener 5.4.2
  • +
  • fastrand 2.5.0
  • find-msvc-tools 0.1.9
  • flate2 1.1.9
  • fnv 1.0.7
  • form_urlencoded 1.2.2
  • hashbrown 0.14.5
  • hashbrown 0.15.5
  • +
  • hashbrown 0.16.1
  • hashbrown 0.17.1
  • heck 0.5.0
  • httparse 1.10.1
  • +
  • hyper-rustls 0.27.10
  • idna 1.1.0
  • idna_adapter 1.2.2
  • indexmap 2.14.0
  • @@ -4121,6 +4966,7 @@

    Used by:

  • num-integer 0.1.46
  • num-traits 0.2.19
  • once_cell 1.21.4
  • +
  • openssl-probe 0.2.1
  • parking 2.2.1
  • parking_lot 0.12.5
  • parking_lot_core 0.9.12
  • @@ -4131,8 +4977,12 @@

    Used by:

  • pest_meta 2.8.6
  • pkg-config 0.3.33
  • regex-automata 0.4.14
  • +
  • regex-lite 0.1.9
  • regex-syntax 0.8.10
  • +
  • ring 0.17.14
  • ron 0.8.1
  • +
  • rustc_version 0.4.1
  • +
  • rustls-native-certs 0.8.4
  • rustls 0.23.46
  • scopeguard 1.2.0
  • signal-hook-registry 1.4.8
  • @@ -4148,6 +4998,7 @@

    Used by:

  • url 2.5.8
  • uuid 1.23.1
  • version_check 0.9.5
  • +
  • xmlparser 0.13.6
  • yasna 0.6.0
  •                               Apache License
    @@ -4998,6 +5849,7 @@ 

    Used by:

  • md-5 0.10.6
  • opaque-debug 0.3.1
  • polyval 0.6.2
  • +
  • sha1 0.10.6
  • sha2 0.10.9
  • universal-hash 0.5.1
  • @@ -7087,6 +7939,7 @@

    Used by:

    +
  • +
  • +

    ISC License

    +

    Used by:

    + +
    Copyright 2015-2025 Brian Smith.
    +
    +Permission to use, copy, modify, and/or distribute this software for any
    +purpose with or without fee is hereby granted, provided that the above
    +copyright notice and this permission notice appear in all copies.
    +
    +THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
    +WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
    +MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY
    +SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
    +WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION
    +OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN
    +CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
     
  • @@ -7825,6 +8717,63 @@

    Used by:

    OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +
  • +
  • +

    MIT License

    +

    Used by:

    + +
    Copyright (c) 2018-2019 Sean McArthur
    +
    +Permission is hereby granted, free of charge, to any person obtaining a copy
    +of this software and associated documentation files (the "Software"), to deal
    +in the Software without restriction, including without limitation the rights
    +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    +copies of the Software, and to permit persons to whom the Software is
    +furnished to do so, subject to the following conditions:
    +
    +The above copyright notice and this permission notice shall be included in
    +all copies or substantial portions of the Software.
    +
    +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
    +THE SOFTWARE.
    +
    +
    +
  • +
  • +

    MIT License

    +

    Used by:

    + +
    Copyright (c) 2018-2023 Sean McArthur
    +Copyright (c) 2016 Alex Crichton
    +
    +Permission is hereby granted, free of charge, to any person obtaining a copy
    +of this software and associated documentation files (the "Software"), to deal
    +in the Software without restriction, including without limitation the rights
    +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    +copies of the Software, and to permit persons to whom the Software is
    +furnished to do so, subject to the following conditions:
    +
    +The above copyright notice and this permission notice shall be included in
    +all copies or substantial portions of the Software.
    +
    +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
    +THE SOFTWARE.
    +
     
  • @@ -7912,6 +8861,39 @@

    Used by:

    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +
  • +
  • +

    MIT License

    +

    Used by:

    + +
    Copyright (c) 2019 Hyper Contributors
    +
    +Permission is hereby granted, free of charge, to any
    +person obtaining a copy of this software and associated
    +documentation files (the "Software"), to deal in the
    +Software without restriction, including without
    +limitation the rights to use, copy, modify, merge,
    +publish, distribute, sublicense, and/or sell copies of
    +the Software, and to permit persons to whom the Software
    +is furnished to do so, subject to the following
    +conditions:
    +
    +The above copyright notice and this permission notice
    +shall be included in all copies or substantial portions
    +of the Software.
    +
    +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF
    +ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED
    +TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A
    +PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT
    +SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
    +CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
    +OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR
    +IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
    +DEALINGS IN THE SOFTWARE.
     
  • @@ -8229,6 +9211,34 @@

    Used by:

    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. +
  • +
  • +

    MIT License

    +

    Used by:

    + +
    MIT License
    +
    +Copyright (c) 2016 Jerome Froelich
    +
    +Permission is hereby granted, free of charge, to any person obtaining a copy
    +of this software and associated documentation files (the "Software"), to deal
    +in the Software without restriction, including without limitation the rights
    +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    +copies of the Software, and to permit persons to whom the Software is
    +furnished to do so, subject to the following conditions:
    +
    +The above copyright notice and this permission notice shall be included in all
    +copies or substantial portions of the Software.
    +
    +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
    +SOFTWARE.
  • MIT License

    @@ -8443,7 +9453,37 @@

    Used by:

    MIT License

    Used by:

    +
    MIT License
    +
    +Copyright (c) 2022 Nugine
    +
    +Permission is hereby granted, free of charge, to any person obtaining a copy
    +of this software and associated documentation files (the "Software"), to deal
    +in the Software without restriction, including without limitation the rights
    +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    +copies of the Software, and to permit persons to whom the Software is
    +furnished to do so, subject to the following conditions:
    +
    +The above copyright notice and this permission notice shall be included in all
    +copies or substantial portions of the Software.
    +
    +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
    +SOFTWARE.
    +
  • +
  • +

    MIT License

    +

    Used by:

    +
    MIT License
     
    @@ -8587,6 +9627,7 @@ 

    Used by:

    MIT License

    Used by:

    The MIT License (MIT)
    @@ -8863,6 +9904,34 @@ 

    Used by:

    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
    +
  • +
  • +

    MIT License

    +

    Used by:

    + +
    © 2016 Bertram Truong
    +© 2021 Kornel Lesiński
    +
    +Permission is hereby granted, free of charge, to any person obtaining a copy
    +of this software and associated documentation files (the "Software"), to deal
    +in the Software without restriction, including without limitation the rights
    +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    +copies of the Software, and to permit persons to whom the Software is
    +furnished to do so, subject to the following conditions:
    +
    +The above copyright notice and this permission notice shall be included in
    +all copies or substantial portions of the Software.
    +
    +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
    +THE SOFTWARE.
    +
  • Unicode License v3

    @@ -8987,6 +10056,7 @@

    zlib License

    Used by:

    Copyright (c) 2024 Orson Peters
     
    diff --git a/SOFTWARE-LICENSE-NOTICES-MONGODB.html b/SOFTWARE-LICENSE-NOTICES-MONGODB.html
    index 3e5a85b19..c86c6d1ae 100644
    --- a/SOFTWARE-LICENSE-NOTICES-MONGODB.html
    +++ b/SOFTWARE-LICENSE-NOTICES-MONGODB.html
    @@ -49,14 +49,14 @@ 

    ExtendDB Software License Notices

    Overview of licenses:

    All license text:

    @@ -1534,6 +1534,215 @@

    Used by:

    See the License for the specific language governing permissions and limitations under the License. +
    +
  • +
  • +

    Apache License 2.0

    +

    Used by:

    + +
                                     Apache License
    +                           Version 2.0, January 2004
    +                        http://www.apache.org/licenses/
    +
    +   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
    +
    +   1. Definitions.
    +
    +      "License" shall mean the terms and conditions for use, reproduction,
    +      and distribution as defined by Sections 1 through 9 of this document.
    +
    +      "Licensor" shall mean the copyright owner or entity authorized by
    +      the copyright owner that is granting the License.
    +
    +      "Legal Entity" shall mean the union of the acting entity and all
    +      other entities that control, are controlled by, or are under common
    +      control with that entity. For the purposes of this definition,
    +      "control" means (i) the power, direct or indirect, to cause the
    +      direction or management of such entity, whether by contract or
    +      otherwise, or (ii) ownership of fifty percent (50%) or more of the
    +      outstanding shares, or (iii) beneficial ownership of such entity.
    +
    +      "You" (or "Your") shall mean an individual or Legal Entity
    +      exercising permissions granted by this License.
    +
    +      "Source" form shall mean the preferred form for making modifications,
    +      including but not limited to software source code, documentation
    +      source, and configuration files.
    +
    +      "Object" form shall mean any form resulting from mechanical
    +      transformation or translation of a Source form, including but
    +      not limited to compiled object code, generated documentation,
    +      and conversions to other media types.
    +
    +      "Work" shall mean the work of authorship, whether in Source or
    +      Object form, made available under the License, as indicated by a
    +      copyright notice that is included in or attached to the work
    +      (an example is provided in the Appendix below).
    +
    +      "Derivative Works" shall mean any work, whether in Source or Object
    +      form, that is based on (or derived from) the Work and for which the
    +      editorial revisions, annotations, elaborations, or other modifications
    +      represent, as a whole, an original work of authorship. For the purposes
    +      of this License, Derivative Works shall not include works that remain
    +      separable from, or merely link (or bind by name) to the interfaces of,
    +      the Work and Derivative Works thereof.
    +
    +      "Contribution" shall mean any work of authorship, including
    +      the original version of the Work and any modifications or additions
    +      to that Work or Derivative Works thereof, that is intentionally
    +      submitted to Licensor for inclusion in the Work by the copyright owner
    +      or by an individual or Legal Entity authorized to submit on behalf of
    +      the copyright owner. For the purposes of this definition, "submitted"
    +      means any form of electronic, verbal, or written communication sent
    +      to the Licensor or its representatives, including but not limited to
    +      communication on electronic mailing lists, source code control systems,
    +      and issue tracking systems that are managed by, or on behalf of, the
    +      Licensor for the purpose of discussing and improving the Work, but
    +      excluding communication that is conspicuously marked or otherwise
    +      designated in writing by the copyright owner as "Not a Contribution."
    +
    +      "Contributor" shall mean Licensor and any individual or Legal Entity
    +      on behalf of whom a Contribution has been received by Licensor and
    +      subsequently incorporated within the Work.
    +
    +   2. Grant of Copyright License. Subject to the terms and conditions of
    +      this License, each Contributor hereby grants to You a perpetual,
    +      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +      copyright license to reproduce, prepare Derivative Works of,
    +      publicly display, publicly perform, sublicense, and distribute the
    +      Work and such Derivative Works in Source or Object form.
    +
    +   3. Grant of Patent License. Subject to the terms and conditions of
    +      this License, each Contributor hereby grants to You a perpetual,
    +      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +      (except as stated in this section) patent license to make, have made,
    +      use, offer to sell, sell, import, and otherwise transfer the Work,
    +      where such license applies only to those patent claims licensable
    +      by such Contributor that are necessarily infringed by their
    +      Contribution(s) alone or by combination of their Contribution(s)
    +      with the Work to which such Contribution(s) was submitted. If You
    +      institute patent litigation against any entity (including a
    +      cross-claim or counterclaim in a lawsuit) alleging that the Work
    +      or a Contribution incorporated within the Work constitutes direct
    +      or contributory patent infringement, then any patent licenses
    +      granted to You under this License for that Work shall terminate
    +      as of the date such litigation is filed.
    +
    +   4. Redistribution. You may reproduce and distribute copies of the
    +      Work or Derivative Works thereof in any medium, with or without
    +      modifications, and in Source or Object form, provided that You
    +      meet the following conditions:
    +
    +      (a) You must give any other recipients of the Work or
    +          Derivative Works a copy of this License; and
    +
    +      (b) You must cause any modified files to carry prominent notices
    +          stating that You changed the files; and
    +
    +      (c) You must retain, in the Source form of any Derivative Works
    +          that You distribute, all copyright, patent, trademark, and
    +          attribution notices from the Source form of the Work,
    +          excluding those notices that do not pertain to any part of
    +          the Derivative Works; and
    +
    +      (d) If the Work includes a "NOTICE" text file as part of its
    +          distribution, then any Derivative Works that You distribute must
    +          include a readable copy of the attribution notices contained
    +          within such NOTICE file, excluding those notices that do not
    +          pertain to any part of the Derivative Works, in at least one
    +          of the following places: within a NOTICE text file distributed
    +          as part of the Derivative Works; within the Source form or
    +          documentation, if provided along with the Derivative Works; or,
    +          within a display generated by the Derivative Works, if and
    +          wherever such third-party notices normally appear. The contents
    +          of the NOTICE file are for informational purposes only and
    +          do not modify the License. You may add Your own attribution
    +          notices within Derivative Works that You distribute, alongside
    +          or as an addendum to the NOTICE text from the Work, provided
    +          that such additional attribution notices cannot be construed
    +          as modifying the License.
    +
    +      You may add Your own copyright statement to Your modifications and
    +      may provide additional or different license terms and conditions
    +      for use, reproduction, or distribution of Your modifications, or
    +      for any such Derivative Works as a whole, provided Your use,
    +      reproduction, and distribution of the Work otherwise complies with
    +      the conditions stated in this License.
    +
    +   5. Submission of Contributions. Unless You explicitly state otherwise,
    +      any Contribution intentionally submitted for inclusion in the Work
    +      by You to the Licensor shall be under the terms and conditions of
    +      this License, without any additional terms or conditions.
    +      Notwithstanding the above, nothing herein shall supersede or modify
    +      the terms of any separate license agreement you may have executed
    +      with Licensor regarding such Contributions.
    +
    +   6. Trademarks. This License does not grant permission to use the trade
    +      names, trademarks, service marks, or product names of the Licensor,
    +      except as required for reasonable and customary use in describing the
    +      origin of the Work and reproducing the content of the NOTICE file.
    +
    +   7. Disclaimer of Warranty. Unless required by applicable law or
    +      agreed to in writing, Licensor provides the Work (and each
    +      Contributor provides its Contributions) on an "AS IS" BASIS,
    +      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
    +      implied, including, without limitation, any warranties or conditions
    +      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
    +      PARTICULAR PURPOSE. You are solely responsible for determining the
    +      appropriateness of using or redistributing the Work and assume any
    +      risks associated with Your exercise of permissions under this License.
    +
    +   8. Limitation of Liability. In no event and under no legal theory,
    +      whether in tort (including negligence), contract, or otherwise,
    +      unless required by applicable law (such as deliberate and grossly
    +      negligent acts) or agreed to in writing, shall any Contributor be
    +      liable to You for damages, including any direct, indirect, special,
    +      incidental, or consequential damages of any character arising as a
    +      result of this License or out of the use or inability to use the
    +      Work (including but not limited to damages for loss of goodwill,
    +      work stoppage, computer failure or malfunction, or any and all
    +      other commercial damages or losses), even if such Contributor
    +      has been advised of the possibility of such damages.
    +
    +   9. Accepting Warranty or Additional Liability. While redistributing
    +      the Work or Derivative Works thereof, You may choose to offer,
    +      and charge a fee for, acceptance of support, warranty, indemnity,
    +      or other liability obligations and/or rights consistent with this
    +      License. However, in accepting such obligations, You may act only
    +      on Your own behalf and on Your sole responsibility, not on behalf
    +      of any other Contributor, and only if You agree to indemnify,
    +      defend, and hold each Contributor harmless for any liability
    +      incurred by, or claims asserted against, such Contributor by reason
    +      of your accepting any such warranty or additional liability.
    +
    +   END OF TERMS AND CONDITIONS
    +
    +   APPENDIX: How to apply the Apache License to your work.
    +
    +      To apply the Apache License to your work, attach the following
    +      boilerplate notice, with the fields enclosed by brackets "[]"
    +      replaced with your own identifying information. (Don't include
    +      the brackets!)  The text should be enclosed in the appropriate
    +      comment syntax for the file format. We also recommend that a
    +      file or class name and description of purpose be included on the
    +      same "printed page" as the copyright notice for easier
    +      identification within third-party archives.
    +
    +   Copyright [yyyy] [name of copyright owner]
    +
    +   Licensed under the Apache License, Version 2.0 (the "License");
    +   you may not use this file except in compliance with the License.
    +   You may obtain a copy of the License at
    +
    +       http://www.apache.org/licenses/LICENSE-2.0
    +
    +   Unless required by applicable law or agreed to in writing, software
    +   distributed under the License is distributed on an "AS IS" BASIS,
    +   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +   See the License for the specific language governing permissions and
    +   limitations under the License.
     
  • @@ -2177,24 +2386,454 @@

    Used by:

    you may not use this file except in compliance with the License. You may obtain a copy of the License at - https://www.apache.org/licenses/LICENSE-2.0 + https://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +
  • +
  • +

    Apache License 2.0

    +

    Used by:

    + +
                                    Apache License
    +                           Version 2.0, January 2004
    +                        http://www.apache.org/licenses/
    +
    +   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
    +
    +   1. Definitions.
    +
    +      "License" shall mean the terms and conditions for use, reproduction,
    +      and distribution as defined by Sections 1 through 9 of this document.
    +
    +      "Licensor" shall mean the copyright owner or entity authorized by
    +      the copyright owner that is granting the License.
    +
    +      "Legal Entity" shall mean the union of the acting entity and all
    +      other entities that control, are controlled by, or are under common
    +      control with that entity. For the purposes of this definition,
    +      "control" means (i) the power, direct or indirect, to cause the
    +      direction or management of such entity, whether by contract or
    +      otherwise, or (ii) ownership of fifty percent (50%) or more of the
    +      outstanding shares, or (iii) beneficial ownership of such entity.
    +
    +      "You" (or "Your") shall mean an individual or Legal Entity
    +      exercising permissions granted by this License.
    +
    +      "Source" form shall mean the preferred form for making modifications,
    +      including but not limited to software source code, documentation
    +      source, and configuration files.
    +
    +      "Object" form shall mean any form resulting from mechanical
    +      transformation or translation of a Source form, including but
    +      not limited to compiled object code, generated documentation,
    +      and conversions to other media types.
    +
    +      "Work" shall mean the work of authorship, whether in Source or
    +      Object form, made available under the License, as indicated by a
    +      copyright notice that is included in or attached to the work
    +      (an example is provided in the Appendix below).
    +
    +      "Derivative Works" shall mean any work, whether in Source or Object
    +      form, that is based on (or derived from) the Work and for which the
    +      editorial revisions, annotations, elaborations, or other modifications
    +      represent, as a whole, an original work of authorship. For the purposes
    +      of this License, Derivative Works shall not include works that remain
    +      separable from, or merely link (or bind by name) to the interfaces of,
    +      the Work and Derivative Works thereof.
    +
    +      "Contribution" shall mean any work of authorship, including
    +      the original version of the Work and any modifications or additions
    +      to that Work or Derivative Works thereof, that is intentionally
    +      submitted to Licensor for inclusion in the Work by the copyright owner
    +      or by an individual or Legal Entity authorized to submit on behalf of
    +      the copyright owner. For the purposes of this definition, "submitted"
    +      means any form of electronic, verbal, or written communication sent
    +      to the Licensor or its representatives, including but not limited to
    +      communication on electronic mailing lists, source code control systems,
    +      and issue tracking systems that are managed by, or on behalf of, the
    +      Licensor for the purpose of discussing and improving the Work, but
    +      excluding communication that is conspicuously marked or otherwise
    +      designated in writing by the copyright owner as "Not a Contribution."
    +
    +      "Contributor" shall mean Licensor and any individual or Legal Entity
    +      on behalf of whom a Contribution has been received by Licensor and
    +      subsequently incorporated within the Work.
    +
    +   2. Grant of Copyright License. Subject to the terms and conditions of
    +      this License, each Contributor hereby grants to You a perpetual,
    +      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +      copyright license to reproduce, prepare Derivative Works of,
    +      publicly display, publicly perform, sublicense, and distribute the
    +      Work and such Derivative Works in Source or Object form.
    +
    +   3. Grant of Patent License. Subject to the terms and conditions of
    +      this License, each Contributor hereby grants to You a perpetual,
    +      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +      (except as stated in this section) patent license to make, have made,
    +      use, offer to sell, sell, import, and otherwise transfer the Work,
    +      where such license applies only to those patent claims licensable
    +      by such Contributor that are necessarily infringed by their
    +      Contribution(s) alone or by combination of their Contribution(s)
    +      with the Work to which such Contribution(s) was submitted. If You
    +      institute patent litigation against any entity (including a
    +      cross-claim or counterclaim in a lawsuit) alleging that the Work
    +      or a Contribution incorporated within the Work constitutes direct
    +      or contributory patent infringement, then any patent licenses
    +      granted to You under this License for that Work shall terminate
    +      as of the date such litigation is filed.
    +
    +   4. Redistribution. You may reproduce and distribute copies of the
    +      Work or Derivative Works thereof in any medium, with or without
    +      modifications, and in Source or Object form, provided that You
    +      meet the following conditions:
    +
    +      (a) You must give any other recipients of the Work or
    +          Derivative Works a copy of this License; and
    +
    +      (b) You must cause any modified files to carry prominent notices
    +          stating that You changed the files; and
    +
    +      (c) You must retain, in the Source form of any Derivative Works
    +          that You distribute, all copyright, patent, trademark, and
    +          attribution notices from the Source form of the Work,
    +          excluding those notices that do not pertain to any part of
    +          the Derivative Works; and
    +
    +      (d) If the Work includes a "NOTICE" text file as part of its
    +          distribution, then any Derivative Works that You distribute must
    +          include a readable copy of the attribution notices contained
    +          within such NOTICE file, excluding those notices that do not
    +          pertain to any part of the Derivative Works, in at least one
    +          of the following places: within a NOTICE text file distributed
    +          as part of the Derivative Works; within the Source form or
    +          documentation, if provided along with the Derivative Works; or,
    +          within a display generated by the Derivative Works, if and
    +          wherever such third-party notices normally appear. The contents
    +          of the NOTICE file are for informational purposes only and
    +          do not modify the License. You may add Your own attribution
    +          notices within Derivative Works that You distribute, alongside
    +          or as an addendum to the NOTICE text from the Work, provided
    +          that such additional attribution notices cannot be construed
    +          as modifying the License.
    +
    +      You may add Your own copyright statement to Your modifications and
    +      may provide additional or different license terms and conditions
    +      for use, reproduction, or distribution of Your modifications, or
    +      for any such Derivative Works as a whole, provided Your use,
    +      reproduction, and distribution of the Work otherwise complies with
    +      the conditions stated in this License.
    +
    +   5. Submission of Contributions. Unless You explicitly state otherwise,
    +      any Contribution intentionally submitted for inclusion in the Work
    +      by You to the Licensor shall be under the terms and conditions of
    +      this License, without any additional terms or conditions.
    +      Notwithstanding the above, nothing herein shall supersede or modify
    +      the terms of any separate license agreement you may have executed
    +      with Licensor regarding such Contributions.
    +
    +   6. Trademarks. This License does not grant permission to use the trade
    +      names, trademarks, service marks, or product names of the Licensor,
    +      except as required for reasonable and customary use in describing the
    +      origin of the Work and reproducing the content of the NOTICE file.
    +
    +   7. Disclaimer of Warranty. Unless required by applicable law or
    +      agreed to in writing, Licensor provides the Work (and each
    +      Contributor provides its Contributions) on an "AS IS" BASIS,
    +      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
    +      implied, including, without limitation, any warranties or conditions
    +      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
    +      PARTICULAR PURPOSE. You are solely responsible for determining the
    +      appropriateness of using or redistributing the Work and assume any
    +      risks associated with Your exercise of permissions under this License.
    +
    +   8. Limitation of Liability. In no event and under no legal theory,
    +      whether in tort (including negligence), contract, or otherwise,
    +      unless required by applicable law (such as deliberate and grossly
    +      negligent acts) or agreed to in writing, shall any Contributor be
    +      liable to You for damages, including any direct, indirect, special,
    +      incidental, or consequential damages of any character arising as a
    +      result of this License or out of the use or inability to use the
    +      Work (including but not limited to damages for loss of goodwill,
    +      work stoppage, computer failure or malfunction, or any and all
    +      other commercial damages or losses), even if such Contributor
    +      has been advised of the possibility of such damages.
    +
    +   9. Accepting Warranty or Additional Liability. While redistributing
    +      the Work or Derivative Works thereof, You may choose to offer,
    +      and charge a fee for, acceptance of support, warranty, indemnity,
    +      or other liability obligations and/or rights consistent with this
    +      License. However, in accepting such obligations, You may act only
    +      on Your own behalf and on Your sole responsibility, not on behalf
    +      of any other Contributor, and only if You agree to indemnify,
    +      defend, and hold each Contributor harmless for any liability
    +      incurred by, or claims asserted against, such Contributor by reason
    +      of your accepting any such warranty or additional liability.
    +
    +   END OF TERMS AND CONDITIONS
    +
    +   APPENDIX: How to apply the Apache License to your work.
    +
    +      To apply the Apache License to your work, attach the following
    +      boilerplate notice, with the fields enclosed by brackets "{}"
    +      replaced with your own identifying information. (Don't include
    +      the brackets!)  The text should be enclosed in the appropriate
    +      comment syntax for the file format. We also recommend that a
    +      file or class name and description of purpose be included on the
    +      same "printed page" as the copyright notice for easier
    +      identification within third-party archives.
    +
    +   Copyright 2018-2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.
    +
    +   Licensed under the Apache License, Version 2.0 (the "License");
    +   you may not use this file except in compliance with the License.
    +   You may obtain a copy of the License at
    +
    +       http://www.apache.org/licenses/LICENSE-2.0
    +
    +   Unless required by applicable law or agreed to in writing, software
    +   distributed under the License is distributed on an "AS IS" BASIS,
    +   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +   See the License for the specific language governing permissions and
    +   limitations under the License.
    +
    +
    +
  • +
  • +

    Apache License 2.0

    +

    Used by:

    + +
                                  Apache License
    +                        Version 2.0 January 2004
    +                     http://www.apache.org/licenses/
    +
    +TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
    +
    +1. Definitions.
    +
    +   "License" shall mean the terms and conditions for use, reproduction,
    +   and distribution as defined by Sections 1 through 9 of this document.
    +
    +   "Licensor" shall mean the copyright owner or entity authorized by
    +   the copyright owner that is granting the License.
    +
    +   "Legal Entity" shall mean the union of the acting entity and all
    +   other entities that control, are controlled by, or are under common
    +   control with that entity. For the purposes of this definition,
    +   "control" means (i) the power, direct or indirect, to cause the
    +   direction or management of such entity, whether by contract or
    +   otherwise, or (ii) ownership of fifty percent (50%) or more of the
    +   outstanding shares, or (iii) beneficial ownership of such entity.
    +
    +   "You" (or "Your") shall mean an individual or Legal Entity
    +   exercising permissions granted by this License.
    +
    +   "Source" form shall mean the preferred form for making modifications,
    +   including but not limited to software source code, documentation
    +   source, and configuration files.
    +
    +   "Object" form shall mean any form resulting from mechanical
    +   transformation or translation of a Source form, including but
    +   not limited to compiled object code, generated documentation,
    +   and conversions to other media types.
    +
    +   "Work" shall mean the work of authorship, whether in Source or
    +   Object form, made available under the License, as indicated by a
    +   copyright notice that is included in or attached to the work
    +   (an example is provided in the Appendix below).
    +
    +   "Derivative Works" shall mean any work, whether in Source or Object
    +   form, that is based on (or derived from) the Work and for which the
    +   editorial revisions, annotations, elaborations, or other modifications
    +   represent, as a whole, an original work of authorship. For the purposes
    +   of this License, Derivative Works shall not include works that remain
    +   separable from, or merely link (or bind by name) to the interfaces of,
    +   the Work and Derivative Works thereof.
    +
    +   "Contribution" shall mean any work of authorship, including
    +   the original version of the Work and any modifications or additions
    +   to that Work or Derivative Works thereof, that is intentionally
    +   submitted to Licensor for inclusion in the Work by the copyright owner
    +   or by an individual or Legal Entity authorized to submit on behalf of
    +   the copyright owner. For the purposes of this definition, "submitted"
    +   means any form of electronic, verbal, or written communication sent
    +   to the Licensor or its representatives, including but not limited to
    +   communication on electronic mailing lists, source code control systems,
    +   and issue tracking systems that are managed by, or on behalf of, the
    +   Licensor for the purpose of discussing and improving the Work, but
    +   excluding communication that is conspicuously marked or otherwise
    +   designated in writing by the copyright owner as "Not a Contribution."
    +
    +   "Contributor" shall mean Licensor and any individual or Legal Entity
    +   on behalf of whom a Contribution has been received by Licensor and
    +   subsequently incorporated within the Work.
    +
    +2. Grant of Copyright License. Subject to the terms and conditions of
    +   this License, each Contributor hereby grants to You a perpetual,
    +   worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +   copyright license to reproduce, prepare Derivative Works of,
    +   publicly display, publicly perform, sublicense, and distribute the
    +   Work and such Derivative Works in Source or Object form.
    +
    +3. Grant of Patent License. Subject to the terms and conditions of
    +   this License, each Contributor hereby grants to You a perpetual,
    +   worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +   (except as stated in this section) patent license to make, have made,
    +   use, offer to sell, sell, import, and otherwise transfer the Work,
    +   where such license applies only to those patent claims licensable
    +   by such Contributor that are necessarily infringed by their
    +   Contribution(s) alone or by combination of their Contribution(s)
    +   with the Work to which such Contribution(s) was submitted. If You
    +   institute patent litigation against any entity (including a
    +   cross-claim or counterclaim in a lawsuit) alleging that the Work
    +   or a Contribution incorporated within the Work constitutes direct
    +   or contributory patent infringement, then any patent licenses
    +   granted to You under this License for that Work shall terminate
    +   as of the date such litigation is filed.
    +
    +4. Redistribution. You may reproduce and distribute copies of the
    +   Work or Derivative Works thereof in any medium, with or without
    +   modifications, and in Source or Object form, provided that You
    +   meet the following conditions:
    +
    +   (a) You must give any other recipients of the Work or
    +       Derivative Works a copy of this License; and
    +
    +   (b) You must cause any modified files to carry prominent notices
    +       stating that You changed the files; and
    +
    +   (c) You must retain, in the Source form of any Derivative Works
    +       that You distribute, all copyright, patent, trademark, and
    +       attribution notices from the Source form of the Work,
    +       excluding those notices that do not pertain to any part of
    +       the Derivative Works; and
    +
    +   (d) If the Work includes a "NOTICE" text file as part of its
    +       distribution, then any Derivative Works that You distribute must
    +       include a readable copy of the attribution notices contained
    +       within such NOTICE file, excluding those notices that do not
    +       pertain to any part of the Derivative Works, in at least one
    +       of the following places: within a NOTICE text file distributed
    +       as part of the Derivative Works; within the Source form or
    +       documentation, if provided along with the Derivative Works; or,
    +       within a display generated by the Derivative Works, if and
    +       wherever such third-party notices normally appear. The contents
    +       of the NOTICE file are for informational purposes only and
    +       do not modify the License. You may add Your own attribution
    +       notices within Derivative Works that You distribute, alongside
    +       or as an addendum to the NOTICE text from the Work, provided
    +       that such additional attribution notices cannot be construed
    +       as modifying the License.
    +
    +   You may add Your own copyright statement to Your modifications and
    +   may provide additional or different license terms and conditions
    +   for use, reproduction, or distribution of Your modifications, or
    +   for any such Derivative Works as a whole, provided Your use,
    +   reproduction, and distribution of the Work otherwise complies with
    +   the conditions stated in this License.
    +
    +5. Submission of Contributions. Unless You explicitly state otherwise,
    +   any Contribution intentionally submitted for inclusion in the Work
    +   by You to the Licensor shall be under the terms and conditions of
    +   this License, without any additional terms or conditions.
    +   Notwithstanding the above, nothing herein shall supersede or modify
    +   the terms of any separate license agreement you may have executed
    +   with Licensor regarding such Contributions.
    +
    +6. Trademarks. This License does not grant permission to use the trade
    +   names, trademarks, service marks, or product names of the Licensor,
    +   except as required for reasonable and customary use in describing the
    +   origin of the Work and reproducing the content of the NOTICE file.
    +
    +7. Disclaimer of Warranty. Unless required by applicable law or
    +   agreed to in writing, Licensor provides the Work (and each
    +   Contributor provides its Contributions) on an "AS IS" BASIS,
    +   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
    +   implied, including, without limitation, any warranties or conditions
    +   of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
    +   PARTICULAR PURPOSE. You are solely responsible for determining the
    +   appropriateness of using or redistributing the Work and assume any
    +   risks associated with Your exercise of permissions under this License.
    +
    +8. Limitation of Liability. In no event and under no legal theory,
    +   whether in tort (including negligence), contract, or otherwise,
    +   unless required by applicable law (such as deliberate and grossly
    +   negligent acts) or agreed to in writing, shall any Contributor be
    +   liable to You for damages, including any direct, indirect, special,
    +   incidental, or consequential damages of any character arising as a
    +   result of this License or out of the use or inability to use the
    +   Work (including but not limited to damages for loss of goodwill,
    +   work stoppage, computer failure or malfunction, or any and all
    +   other commercial damages or losses), even if such Contributor
    +   has been advised of the possibility of such damages.
    +
    +9. Accepting Warranty or Additional Liability. While redistributing
    +   the Work or Derivative Works thereof, You may choose to offer,
    +   and charge a fee for, acceptance of support, warranty, indemnity,
    +   or other liability obligations and/or rights consistent with this
    +   License. However, in accepting such obligations, You may act only
    +   on Your own behalf and on Your sole responsibility, not on behalf
    +   of any other Contributor, and only if You agree to indemnify,
    +   defend, and hold each Contributor harmless for any liability
    +   incurred by, or claims asserted against, such Contributor by reason
    +   of your accepting any such warranty or additional liability.
    +
    +END OF TERMS AND CONDITIONS
    +
    +APPENDIX: How to apply the Apache License to your work.
    +
    +   To apply the Apache License to your work, attach the following
    +   boilerplate notice, with the fields enclosed by brackets "[]"
    +   replaced with your own identifying information. (Don't include
    +   the brackets!)  The text should be enclosed in the appropriate
    +   comment syntax for the file format. We also recommend that a
    +   file or class name and description of purpose be included on the
    +   same "printed page" as the copyright notice for easier
    +   identification within third-party archives.
    +
    +Copyright [yyyy] [name of copyright owner]
     
    -   Unless required by applicable law or agreed to in writing, software
    -   distributed under the License is distributed on an "AS IS" BASIS,
    -   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    -   See the License for the specific language governing permissions and
    -   limitations under the License.
    +Licensed under the Apache License, Version 2.0 (the "License");
    +you may not use this file except in compliance with the License.
    +You may obtain a copy of the License at
    +
    +	http://www.apache.org/licenses/LICENSE-2.0
     
    +Unless required by applicable law or agreed to in writing, software
    +distributed under the License is distributed on an "AS IS" BASIS,
    +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +See the License for the specific language governing permissions and
    +limitations under the License.
     
  • Apache License 2.0

    Used by:

                                  Apache License
    -                        Version 2.0 January 2004
    +                        Version 2.0, January 2004
                          http://www.apache.org/licenses/
     
     TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
    @@ -2381,7 +3020,8 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright [yyyy] [name of copyright owner] +Copyright (c) 2016 Alex Crichton +Copyright (c) 2017 The Tokio Authors Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -2400,15 +3040,7 @@

    Used by:

    Apache License 2.0

    Used by:

                                  Apache License
                             Version 2.0, January 2004
    @@ -2598,8 +3230,7 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright (c) 2016 Alex Crichton -Copyright (c) 2017 The Tokio Authors +Copyright (c) 2019 Matthias Einwag Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -2618,7 +3249,7 @@

    Used by:

    Apache License 2.0

    Used by:

                                  Apache License
                             Version 2.0, January 2004
    @@ -2808,7 +3439,7 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright (c) 2019 Matthias Einwag +Copyright 2014 Paho Lurie-Gregg Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -2820,14 +3451,13 @@

    Used by:

    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and -limitations under the License. -
    +limitations under the License.
  • Apache License 2.0

    Used by:

                                  Apache License
                             Version 2.0, January 2004
    @@ -3017,7 +3647,7 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright 2014 Paho Lurie-Gregg +Copyright 2017 The Rust Project Developers Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -3029,13 +3659,15 @@

    Used by:

    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and -limitations under the License.
    +limitations under the License. +
  • Apache License 2.0

    Used by:

                                  Apache License
                             Version 2.0, January 2004
    @@ -3225,7 +3857,7 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright 2017 The Rust Project Developers +Copyright 2017 http-rs authors Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -3244,7 +3876,7 @@

    Used by:

    Apache License 2.0

    Used by:

                                  Apache License
                             Version 2.0, January 2004
    @@ -3434,7 +4066,7 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright 2017 http-rs authors +Copyright 2017 quininer kel Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -3453,7 +4085,7 @@

    Used by:

    Apache License 2.0

    Used by:

                                  Apache License
                             Version 2.0, January 2004
    @@ -3643,7 +4275,7 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright 2017 quininer kel +Copyright 2018 The pin-utils authors Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -4509,6 +5141,7 @@

    Used by:

  • base64 0.21.7
  • base64 0.22.1
  • bitflags 2.11.1
  • +
  • bytes-utils 0.1.4
  • cc 1.2.62
  • cfg-if 1.0.4
  • cmake 0.1.58
  • @@ -4528,15 +5161,18 @@

    Used by:

  • errno 0.3.14
  • event-listener-strategy 0.5.4
  • event-listener 5.4.2
  • +
  • fastrand 2.5.0
  • find-msvc-tools 0.1.9
  • flate2 1.1.9
  • fnv 1.0.7
  • form_urlencoded 1.2.2
  • hashbrown 0.14.5
  • hashbrown 0.15.5
  • +
  • hashbrown 0.16.1
  • hashbrown 0.17.1
  • heck 0.5.0
  • httparse 1.10.1
  • +
  • hyper-rustls 0.27.10
  • idna 1.1.0
  • idna_adapter 1.2.2
  • indexmap 2.14.0
  • @@ -4549,6 +5185,7 @@

    Used by:

  • num-integer 0.1.46
  • num-traits 0.2.19
  • once_cell 1.21.4
  • +
  • openssl-probe 0.2.1
  • parking 2.2.1
  • parking_lot 0.12.5
  • parking_lot_core 0.9.12
  • @@ -4559,10 +5196,12 @@

    Used by:

  • pest_meta 2.8.6
  • pkg-config 0.3.33
  • regex-automata 0.4.14
  • +
  • regex-lite 0.1.9
  • regex-syntax 0.8.10
  • ring 0.17.14
  • ron 0.8.1
  • rustc_version 0.4.1
  • +
  • rustls-native-certs 0.8.4
  • rustls 0.23.46
  • scopeguard 1.2.0
  • serde_with 3.21.0
  • @@ -4583,6 +5222,7 @@

    Used by:

  • url 2.5.8
  • uuid 1.23.1
  • version_check 0.9.5
  • +
  • xmlparser 0.13.6
  • yasna 0.6.0
  •                               Apache License
    @@ -5443,6 +6083,7 @@ 

    Used by:

  • opaque-debug 0.3.1
  • pbkdf2 0.13.0
  • polyval 0.6.2
  • +
  • sha1 0.10.6
  • sha1 0.11.0
  • sha2 0.10.9
  • sha2 0.11.0
  • @@ -7116,6 +7757,7 @@

    Used by:

    +
  • +
  • +

    MIT License

    +

    Used by:

    + +
    Copyright (c) 2018-2019 Sean McArthur
    +
    +Permission is hereby granted, free of charge, to any person obtaining a copy
    +of this software and associated documentation files (the "Software"), to deal
    +in the Software without restriction, including without limitation the rights
    +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    +copies of the Software, and to permit persons to whom the Software is
    +furnished to do so, subject to the following conditions:
    +
    +The above copyright notice and this permission notice shall be included in
    +all copies or substantial portions of the Software.
    +
    +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
    +THE SOFTWARE.
    +
    +
    +
  • +
  • +

    MIT License

    +

    Used by:

    + +
    Copyright (c) 2018-2023 Sean McArthur
    +Copyright (c) 2016 Alex Crichton
    +
    +Permission is hereby granted, free of charge, to any person obtaining a copy
    +of this software and associated documentation files (the "Software"), to deal
    +in the Software without restriction, including without limitation the rights
    +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    +copies of the Software, and to permit persons to whom the Software is
    +furnished to do so, subject to the following conditions:
    +
    +The above copyright notice and this permission notice shall be included in
    +all copies or substantial portions of the Software.
    +
    +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
    +THE SOFTWARE.
    +
     
  • @@ -7941,6 +8657,39 @@

    Used by:

    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +
  • +
  • +

    MIT License

    +

    Used by:

    + +
    Copyright (c) 2019 Hyper Contributors
    +
    +Permission is hereby granted, free of charge, to any
    +person obtaining a copy of this software and associated
    +documentation files (the "Software"), to deal in the
    +Software without restriction, including without
    +limitation the rights to use, copy, modify, merge,
    +publish, distribute, sublicense, and/or sell copies of
    +the Software, and to permit persons to whom the Software
    +is furnished to do so, subject to the following
    +conditions:
    +
    +The above copyright notice and this permission notice
    +shall be included in all copies or substantial portions
    +of the Software.
    +
    +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF
    +ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED
    +TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A
    +PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT
    +SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
    +CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
    +OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR
    +IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
    +DEALINGS IN THE SOFTWARE.
     
  • @@ -8285,6 +9034,34 @@

    Used by:

    DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. +
  • +
  • +

    MIT License

    +

    Used by:

    + +
    MIT License
    +
    +Copyright (c) 2016 Jerome Froelich
    +
    +Permission is hereby granted, free of charge, to any person obtaining a copy
    +of this software and associated documentation files (the "Software"), to deal
    +in the Software without restriction, including without limitation the rights
    +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    +copies of the Software, and to permit persons to whom the Software is
    +furnished to do so, subject to the following conditions:
    +
    +The above copyright notice and this permission notice shall be included in all
    +copies or substantial portions of the Software.
    +
    +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
    +SOFTWARE.
  • MIT License

    @@ -8671,6 +9448,34 @@

    Used by:

    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. +
  • +
  • +

    MIT License

    +

    Used by:

    + +
    MIT License
    +
    +Copyright (c) 2022 Nugine
    +
    +Permission is hereby granted, free of charge, to any person obtaining a copy
    +of this software and associated documentation files (the "Software"), to deal
    +in the Software without restriction, including without limitation the rights
    +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    +copies of the Software, and to permit persons to whom the Software is
    +furnished to do so, subject to the following conditions:
    +
    +The above copyright notice and this permission notice shall be included in all
    +copies or substantial portions of the Software.
    +
    +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
    +SOFTWARE.
  • MIT License

    @@ -8704,7 +9509,9 @@

    Used by:

    MIT License

    Used by:

    MIT License
     
    @@ -8843,6 +9650,34 @@ 

    Used by:

    OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
    +
  • +
  • +

    MIT License

    +

    Used by:

    + +
    The MIT License (MIT)
    +
    +Copyright (c) 2014 Mathijs van de Nes
    +
    +Permission is hereby granted, free of charge, to any person obtaining a copy
    +of this software and associated documentation files (the "Software"), to deal
    +in the Software without restriction, including without limitation the rights
    +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    +copies of the Software, and to permit persons to whom the Software is
    +furnished to do so, subject to the following conditions:
    +
    +The above copyright notice and this permission notice shall be included in all
    +copies or substantial portions of the Software.
    +
    +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
    +SOFTWARE.
  • MIT License

    @@ -9244,6 +10079,34 @@

    Used by:

    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +
  • +
  • +

    MIT License

    +

    Used by:

    + +
    © 2016 Bertram Truong
    +© 2021 Kornel Lesiński
    +
    +Permission is hereby granted, free of charge, to any person obtaining a copy
    +of this software and associated documentation files (the "Software"), to deal
    +in the Software without restriction, including without limitation the rights
    +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    +copies of the Software, and to permit persons to whom the Software is
    +furnished to do so, subject to the following conditions:
    +
    +The above copyright notice and this permission notice shall be included in
    +all copies or substantial portions of the Software.
    +
    +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
    +THE SOFTWARE.
     
  • @@ -9369,6 +10232,7 @@

    zlib License

    Used by:

    Copyright (c) 2024 Orson Peters
     
    diff --git a/SOFTWARE-LICENSE-NOTICES.html b/SOFTWARE-LICENSE-NOTICES.html
    index 05b0fe365..4234d8bef 100644
    --- a/SOFTWARE-LICENSE-NOTICES.html
    +++ b/SOFTWARE-LICENSE-NOTICES.html
    @@ -49,14 +49,14 @@ 

    ExtendDB Software License Notices

    Overview of licenses:

    All license text:

    @@ -1538,25 +1538,216 @@

    Used by:

    Apache License 2.0

    Used by:

    +
                                     Apache License
    +                           Version 2.0, January 2004
    +                        http://www.apache.org/licenses/
    +
    +   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
    +
    +   1. Definitions.
    +
    +      "License" shall mean the terms and conditions for use, reproduction,
    +      and distribution as defined by Sections 1 through 9 of this document.
    +
    +      "Licensor" shall mean the copyright owner or entity authorized by
    +      the copyright owner that is granting the License.
    +
    +      "Legal Entity" shall mean the union of the acting entity and all
    +      other entities that control, are controlled by, or are under common
    +      control with that entity. For the purposes of this definition,
    +      "control" means (i) the power, direct or indirect, to cause the
    +      direction or management of such entity, whether by contract or
    +      otherwise, or (ii) ownership of fifty percent (50%) or more of the
    +      outstanding shares, or (iii) beneficial ownership of such entity.
    +
    +      "You" (or "Your") shall mean an individual or Legal Entity
    +      exercising permissions granted by this License.
    +
    +      "Source" form shall mean the preferred form for making modifications,
    +      including but not limited to software source code, documentation
    +      source, and configuration files.
    +
    +      "Object" form shall mean any form resulting from mechanical
    +      transformation or translation of a Source form, including but
    +      not limited to compiled object code, generated documentation,
    +      and conversions to other media types.
    +
    +      "Work" shall mean the work of authorship, whether in Source or
    +      Object form, made available under the License, as indicated by a
    +      copyright notice that is included in or attached to the work
    +      (an example is provided in the Appendix below).
    +
    +      "Derivative Works" shall mean any work, whether in Source or Object
    +      form, that is based on (or derived from) the Work and for which the
    +      editorial revisions, annotations, elaborations, or other modifications
    +      represent, as a whole, an original work of authorship. For the purposes
    +      of this License, Derivative Works shall not include works that remain
    +      separable from, or merely link (or bind by name) to the interfaces of,
    +      the Work and Derivative Works thereof.
    +
    +      "Contribution" shall mean any work of authorship, including
    +      the original version of the Work and any modifications or additions
    +      to that Work or Derivative Works thereof, that is intentionally
    +      submitted to Licensor for inclusion in the Work by the copyright owner
    +      or by an individual or Legal Entity authorized to submit on behalf of
    +      the copyright owner. For the purposes of this definition, "submitted"
    +      means any form of electronic, verbal, or written communication sent
    +      to the Licensor or its representatives, including but not limited to
    +      communication on electronic mailing lists, source code control systems,
    +      and issue tracking systems that are managed by, or on behalf of, the
    +      Licensor for the purpose of discussing and improving the Work, but
    +      excluding communication that is conspicuously marked or otherwise
    +      designated in writing by the copyright owner as "Not a Contribution."
    +
    +      "Contributor" shall mean Licensor and any individual or Legal Entity
    +      on behalf of whom a Contribution has been received by Licensor and
    +      subsequently incorporated within the Work.
    +
    +   2. Grant of Copyright License. Subject to the terms and conditions of
    +      this License, each Contributor hereby grants to You a perpetual,
    +      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +      copyright license to reproduce, prepare Derivative Works of,
    +      publicly display, publicly perform, sublicense, and distribute the
    +      Work and such Derivative Works in Source or Object form.
    +
    +   3. Grant of Patent License. Subject to the terms and conditions of
    +      this License, each Contributor hereby grants to You a perpetual,
    +      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +      (except as stated in this section) patent license to make, have made,
    +      use, offer to sell, sell, import, and otherwise transfer the Work,
    +      where such license applies only to those patent claims licensable
    +      by such Contributor that are necessarily infringed by their
    +      Contribution(s) alone or by combination of their Contribution(s)
    +      with the Work to which such Contribution(s) was submitted. If You
    +      institute patent litigation against any entity (including a
    +      cross-claim or counterclaim in a lawsuit) alleging that the Work
    +      or a Contribution incorporated within the Work constitutes direct
    +      or contributory patent infringement, then any patent licenses
    +      granted to You under this License for that Work shall terminate
    +      as of the date such litigation is filed.
    +
    +   4. Redistribution. You may reproduce and distribute copies of the
    +      Work or Derivative Works thereof in any medium, with or without
    +      modifications, and in Source or Object form, provided that You
    +      meet the following conditions:
    +
    +      (a) You must give any other recipients of the Work or
    +          Derivative Works a copy of this License; and
    +
    +      (b) You must cause any modified files to carry prominent notices
    +          stating that You changed the files; and
    +
    +      (c) You must retain, in the Source form of any Derivative Works
    +          that You distribute, all copyright, patent, trademark, and
    +          attribution notices from the Source form of the Work,
    +          excluding those notices that do not pertain to any part of
    +          the Derivative Works; and
    +
    +      (d) If the Work includes a "NOTICE" text file as part of its
    +          distribution, then any Derivative Works that You distribute must
    +          include a readable copy of the attribution notices contained
    +          within such NOTICE file, excluding those notices that do not
    +          pertain to any part of the Derivative Works, in at least one
    +          of the following places: within a NOTICE text file distributed
    +          as part of the Derivative Works; within the Source form or
    +          documentation, if provided along with the Derivative Works; or,
    +          within a display generated by the Derivative Works, if and
    +          wherever such third-party notices normally appear. The contents
    +          of the NOTICE file are for informational purposes only and
    +          do not modify the License. You may add Your own attribution
    +          notices within Derivative Works that You distribute, alongside
    +          or as an addendum to the NOTICE text from the Work, provided
    +          that such additional attribution notices cannot be construed
    +          as modifying the License.
    +
    +      You may add Your own copyright statement to Your modifications and
    +      may provide additional or different license terms and conditions
    +      for use, reproduction, or distribution of Your modifications, or
    +      for any such Derivative Works as a whole, provided Your use,
    +      reproduction, and distribution of the Work otherwise complies with
    +      the conditions stated in this License.
    +
    +   5. Submission of Contributions. Unless You explicitly state otherwise,
    +      any Contribution intentionally submitted for inclusion in the Work
    +      by You to the Licensor shall be under the terms and conditions of
    +      this License, without any additional terms or conditions.
    +      Notwithstanding the above, nothing herein shall supersede or modify
    +      the terms of any separate license agreement you may have executed
    +      with Licensor regarding such Contributions.
    +
    +   6. Trademarks. This License does not grant permission to use the trade
    +      names, trademarks, service marks, or product names of the Licensor,
    +      except as required for reasonable and customary use in describing the
    +      origin of the Work and reproducing the content of the NOTICE file.
    +
    +   7. Disclaimer of Warranty. Unless required by applicable law or
    +      agreed to in writing, Licensor provides the Work (and each
    +      Contributor provides its Contributions) on an "AS IS" BASIS,
    +      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
    +      implied, including, without limitation, any warranties or conditions
    +      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
    +      PARTICULAR PURPOSE. You are solely responsible for determining the
    +      appropriateness of using or redistributing the Work and assume any
    +      risks associated with Your exercise of permissions under this License.
    +
    +   8. Limitation of Liability. In no event and under no legal theory,
    +      whether in tort (including negligence), contract, or otherwise,
    +      unless required by applicable law (such as deliberate and grossly
    +      negligent acts) or agreed to in writing, shall any Contributor be
    +      liable to You for damages, including any direct, indirect, special,
    +      incidental, or consequential damages of any character arising as a
    +      result of this License or out of the use or inability to use the
    +      Work (including but not limited to damages for loss of goodwill,
    +      work stoppage, computer failure or malfunction, or any and all
    +      other commercial damages or losses), even if such Contributor
    +      has been advised of the possibility of such damages.
    +
    +   9. Accepting Warranty or Additional Liability. While redistributing
    +      the Work or Derivative Works thereof, You may choose to offer,
    +      and charge a fee for, acceptance of support, warranty, indemnity,
    +      or other liability obligations and/or rights consistent with this
    +      License. However, in accepting such obligations, You may act only
    +      on Your own behalf and on Your sole responsibility, not on behalf
    +      of any other Contributor, and only if You agree to indemnify,
    +      defend, and hold each Contributor harmless for any liability
    +      incurred by, or claims asserted against, such Contributor by reason
    +      of your accepting any such warranty or additional liability.
    +
    +   END OF TERMS AND CONDITIONS
    +
    +   APPENDIX: How to apply the Apache License to your work.
    +
    +      To apply the Apache License to your work, attach the following
    +      boilerplate notice, with the fields enclosed by brackets "[]"
    +      replaced with your own identifying information. (Don't include
    +      the brackets!)  The text should be enclosed in the appropriate
    +      comment syntax for the file format. We also recommend that a
    +      file or class name and description of purpose be included on the
    +      same "printed page" as the copyright notice for easier
    +      identification within third-party archives.
    +
    +   Copyright [yyyy] [name of copyright owner]
    +
    +   Licensed under the Apache License, Version 2.0 (the "License");
    +   you may not use this file except in compliance with the License.
    +   You may obtain a copy of the License at
    +
    +       http://www.apache.org/licenses/LICENSE-2.0
    +
    +   Unless required by applicable law or agreed to in writing, software
    +   distributed under the License is distributed on an "AS IS" BASIS,
    +   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +   See the License for the specific language governing permissions and
    +   limitations under the License.
    +
    +
  • +
  • +

    Apache License 2.0

    +

    Used by:

    +
                                     Apache License
                                Version 2.0, January 2004
    @@ -1746,30 +1937,687 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. - Copyright {yyyy} {name of copyright owner} + Copyright 2017 Juniper Networks, Inc. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +
    +
  • +
  • +

    Apache License 2.0

    +

    Used by:

    + +
                                     Apache License
    +                           Version 2.0, January 2004
    +                        http://www.apache.org/licenses/
    +
    +   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
    +
    +   1. Definitions.
    +
    +      "License" shall mean the terms and conditions for use, reproduction,
    +      and distribution as defined by Sections 1 through 9 of this document.
    +
    +      "Licensor" shall mean the copyright owner or entity authorized by
    +      the copyright owner that is granting the License.
    +
    +      "Legal Entity" shall mean the union of the acting entity and all
    +      other entities that control, are controlled by, or are under common
    +      control with that entity. For the purposes of this definition,
    +      "control" means (i) the power, direct or indirect, to cause the
    +      direction or management of such entity, whether by contract or
    +      otherwise, or (ii) ownership of fifty percent (50%) or more of the
    +      outstanding shares, or (iii) beneficial ownership of such entity.
    +
    +      "You" (or "Your") shall mean an individual or Legal Entity
    +      exercising permissions granted by this License.
    +
    +      "Source" form shall mean the preferred form for making modifications,
    +      including but not limited to software source code, documentation
    +      source, and configuration files.
    +
    +      "Object" form shall mean any form resulting from mechanical
    +      transformation or translation of a Source form, including but
    +      not limited to compiled object code, generated documentation,
    +      and conversions to other media types.
    +
    +      "Work" shall mean the work of authorship, whether in Source or
    +      Object form, made available under the License, as indicated by a
    +      copyright notice that is included in or attached to the work
    +      (an example is provided in the Appendix below).
    +
    +      "Derivative Works" shall mean any work, whether in Source or Object
    +      form, that is based on (or derived from) the Work and for which the
    +      editorial revisions, annotations, elaborations, or other modifications
    +      represent, as a whole, an original work of authorship. For the purposes
    +      of this License, Derivative Works shall not include works that remain
    +      separable from, or merely link (or bind by name) to the interfaces of,
    +      the Work and Derivative Works thereof.
    +
    +      "Contribution" shall mean any work of authorship, including
    +      the original version of the Work and any modifications or additions
    +      to that Work or Derivative Works thereof, that is intentionally
    +      submitted to Licensor for inclusion in the Work by the copyright owner
    +      or by an individual or Legal Entity authorized to submit on behalf of
    +      the copyright owner. For the purposes of this definition, "submitted"
    +      means any form of electronic, verbal, or written communication sent
    +      to the Licensor or its representatives, including but not limited to
    +      communication on electronic mailing lists, source code control systems,
    +      and issue tracking systems that are managed by, or on behalf of, the
    +      Licensor for the purpose of discussing and improving the Work, but
    +      excluding communication that is conspicuously marked or otherwise
    +      designated in writing by the copyright owner as "Not a Contribution."
    +
    +      "Contributor" shall mean Licensor and any individual or Legal Entity
    +      on behalf of whom a Contribution has been received by Licensor and
    +      subsequently incorporated within the Work.
    +
    +   2. Grant of Copyright License. Subject to the terms and conditions of
    +      this License, each Contributor hereby grants to You a perpetual,
    +      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +      copyright license to reproduce, prepare Derivative Works of,
    +      publicly display, publicly perform, sublicense, and distribute the
    +      Work and such Derivative Works in Source or Object form.
    +
    +   3. Grant of Patent License. Subject to the terms and conditions of
    +      this License, each Contributor hereby grants to You a perpetual,
    +      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +      (except as stated in this section) patent license to make, have made,
    +      use, offer to sell, sell, import, and otherwise transfer the Work,
    +      where such license applies only to those patent claims licensable
    +      by such Contributor that are necessarily infringed by their
    +      Contribution(s) alone or by combination of their Contribution(s)
    +      with the Work to which such Contribution(s) was submitted. If You
    +      institute patent litigation against any entity (including a
    +      cross-claim or counterclaim in a lawsuit) alleging that the Work
    +      or a Contribution incorporated within the Work constitutes direct
    +      or contributory patent infringement, then any patent licenses
    +      granted to You under this License for that Work shall terminate
    +      as of the date such litigation is filed.
    +
    +   4. Redistribution. You may reproduce and distribute copies of the
    +      Work or Derivative Works thereof in any medium, with or without
    +      modifications, and in Source or Object form, provided that You
    +      meet the following conditions:
    +
    +      (a) You must give any other recipients of the Work or
    +          Derivative Works a copy of this License; and
    +
    +      (b) You must cause any modified files to carry prominent notices
    +          stating that You changed the files; and
    +
    +      (c) You must retain, in the Source form of any Derivative Works
    +          that You distribute, all copyright, patent, trademark, and
    +          attribution notices from the Source form of the Work,
    +          excluding those notices that do not pertain to any part of
    +          the Derivative Works; and
    +
    +      (d) If the Work includes a "NOTICE" text file as part of its
    +          distribution, then any Derivative Works that You distribute must
    +          include a readable copy of the attribution notices contained
    +          within such NOTICE file, excluding those notices that do not
    +          pertain to any part of the Derivative Works, in at least one
    +          of the following places: within a NOTICE text file distributed
    +          as part of the Derivative Works; within the Source form or
    +          documentation, if provided along with the Derivative Works; or,
    +          within a display generated by the Derivative Works, if and
    +          wherever such third-party notices normally appear. The contents
    +          of the NOTICE file are for informational purposes only and
    +          do not modify the License. You may add Your own attribution
    +          notices within Derivative Works that You distribute, alongside
    +          or as an addendum to the NOTICE text from the Work, provided
    +          that such additional attribution notices cannot be construed
    +          as modifying the License.
    +
    +      You may add Your own copyright statement to Your modifications and
    +      may provide additional or different license terms and conditions
    +      for use, reproduction, or distribution of Your modifications, or
    +      for any such Derivative Works as a whole, provided Your use,
    +      reproduction, and distribution of the Work otherwise complies with
    +      the conditions stated in this License.
    +
    +   5. Submission of Contributions. Unless You explicitly state otherwise,
    +      any Contribution intentionally submitted for inclusion in the Work
    +      by You to the Licensor shall be under the terms and conditions of
    +      this License, without any additional terms or conditions.
    +      Notwithstanding the above, nothing herein shall supersede or modify
    +      the terms of any separate license agreement you may have executed
    +      with Licensor regarding such Contributions.
    +
    +   6. Trademarks. This License does not grant permission to use the trade
    +      names, trademarks, service marks, or product names of the Licensor,
    +      except as required for reasonable and customary use in describing the
    +      origin of the Work and reproducing the content of the NOTICE file.
    +
    +   7. Disclaimer of Warranty. Unless required by applicable law or
    +      agreed to in writing, Licensor provides the Work (and each
    +      Contributor provides its Contributions) on an "AS IS" BASIS,
    +      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
    +      implied, including, without limitation, any warranties or conditions
    +      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
    +      PARTICULAR PURPOSE. You are solely responsible for determining the
    +      appropriateness of using or redistributing the Work and assume any
    +      risks associated with Your exercise of permissions under this License.
    +
    +   8. Limitation of Liability. In no event and under no legal theory,
    +      whether in tort (including negligence), contract, or otherwise,
    +      unless required by applicable law (such as deliberate and grossly
    +      negligent acts) or agreed to in writing, shall any Contributor be
    +      liable to You for damages, including any direct, indirect, special,
    +      incidental, or consequential damages of any character arising as a
    +      result of this License or out of the use or inability to use the
    +      Work (including but not limited to damages for loss of goodwill,
    +      work stoppage, computer failure or malfunction, or any and all
    +      other commercial damages or losses), even if such Contributor
    +      has been advised of the possibility of such damages.
    +
    +   9. Accepting Warranty or Additional Liability. While redistributing
    +      the Work or Derivative Works thereof, You may choose to offer,
    +      and charge a fee for, acceptance of support, warranty, indemnity,
    +      or other liability obligations and/or rights consistent with this
    +      License. However, in accepting such obligations, You may act only
    +      on Your own behalf and on Your sole responsibility, not on behalf
    +      of any other Contributor, and only if You agree to indemnify,
    +      defend, and hold each Contributor harmless for any liability
    +      incurred by, or claims asserted against, such Contributor by reason
    +      of your accepting any such warranty or additional liability.
    +
    +   END OF TERMS AND CONDITIONS
    +
    +   APPENDIX: How to apply the Apache License to your work.
    +
    +      To apply the Apache License to your work, attach the following
    +      boilerplate notice, with the fields enclosed by brackets "{}"
    +      replaced with your own identifying information. (Don't include
    +      the brackets!)  The text should be enclosed in the appropriate
    +      comment syntax for the file format. We also recommend that a
    +      file or class name and description of purpose be included on the
    +      same "printed page" as the copyright notice for easier
    +      identification within third-party archives.
    +
    +   Copyright {yyyy} {name of copyright owner}
    +
    +   Licensed under the Apache License, Version 2.0 (the "License");
    +   you may not use this file except in compliance with the License.
    +   You may obtain a copy of the License at
    +
    +       http://www.apache.org/licenses/LICENSE-2.0
    +
    +   Unless required by applicable law or agreed to in writing, software
    +   distributed under the License is distributed on an "AS IS" BASIS,
    +   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +   See the License for the specific language governing permissions and
    +   limitations under the License.
    +
    +
    +
  • +
  • +

    Apache License 2.0

    +

    Used by:

    + +
                                    Apache License
    +                           Version 2.0, January 2004
    +                        http://www.apache.org/licenses/
    +
    +   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
    +
    +   1. Definitions.
    +
    +      "License" shall mean the terms and conditions for use, reproduction,
    +      and distribution as defined by Sections 1 through 9 of this document.
    +
    +      "Licensor" shall mean the copyright owner or entity authorized by
    +      the copyright owner that is granting the License.
    +
    +      "Legal Entity" shall mean the union of the acting entity and all
    +      other entities that control, are controlled by, or are under common
    +      control with that entity. For the purposes of this definition,
    +      "control" means (i) the power, direct or indirect, to cause the
    +      direction or management of such entity, whether by contract or
    +      otherwise, or (ii) ownership of fifty percent (50%) or more of the
    +      outstanding shares, or (iii) beneficial ownership of such entity.
    +
    +      "You" (or "Your") shall mean an individual or Legal Entity
    +      exercising permissions granted by this License.
    +
    +      "Source" form shall mean the preferred form for making modifications,
    +      including but not limited to software source code, documentation
    +      source, and configuration files.
    +
    +      "Object" form shall mean any form resulting from mechanical
    +      transformation or translation of a Source form, including but
    +      not limited to compiled object code, generated documentation,
    +      and conversions to other media types.
    +
    +      "Work" shall mean the work of authorship, whether in Source or
    +      Object form, made available under the License, as indicated by a
    +      copyright notice that is included in or attached to the work
    +      (an example is provided in the Appendix below).
    +
    +      "Derivative Works" shall mean any work, whether in Source or Object
    +      form, that is based on (or derived from) the Work and for which the
    +      editorial revisions, annotations, elaborations, or other modifications
    +      represent, as a whole, an original work of authorship. For the purposes
    +      of this License, Derivative Works shall not include works that remain
    +      separable from, or merely link (or bind by name) to the interfaces of,
    +      the Work and Derivative Works thereof.
    +
    +      "Contribution" shall mean any work of authorship, including
    +      the original version of the Work and any modifications or additions
    +      to that Work or Derivative Works thereof, that is intentionally
    +      submitted to Licensor for inclusion in the Work by the copyright owner
    +      or by an individual or Legal Entity authorized to submit on behalf of
    +      the copyright owner. For the purposes of this definition, "submitted"
    +      means any form of electronic, verbal, or written communication sent
    +      to the Licensor or its representatives, including but not limited to
    +      communication on electronic mailing lists, source code control systems,
    +      and issue tracking systems that are managed by, or on behalf of, the
    +      Licensor for the purpose of discussing and improving the Work, but
    +      excluding communication that is conspicuously marked or otherwise
    +      designated in writing by the copyright owner as "Not a Contribution."
    +
    +      "Contributor" shall mean Licensor and any individual or Legal Entity
    +      on behalf of whom a Contribution has been received by Licensor and
    +      subsequently incorporated within the Work.
    +
    +   2. Grant of Copyright License. Subject to the terms and conditions of
    +      this License, each Contributor hereby grants to You a perpetual,
    +      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +      copyright license to reproduce, prepare Derivative Works of,
    +      publicly display, publicly perform, sublicense, and distribute the
    +      Work and such Derivative Works in Source or Object form.
    +
    +   3. Grant of Patent License. Subject to the terms and conditions of
    +      this License, each Contributor hereby grants to You a perpetual,
    +      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +      (except as stated in this section) patent license to make, have made,
    +      use, offer to sell, sell, import, and otherwise transfer the Work,
    +      where such license applies only to those patent claims licensable
    +      by such Contributor that are necessarily infringed by their
    +      Contribution(s) alone or by combination of their Contribution(s)
    +      with the Work to which such Contribution(s) was submitted. If You
    +      institute patent litigation against any entity (including a
    +      cross-claim or counterclaim in a lawsuit) alleging that the Work
    +      or a Contribution incorporated within the Work constitutes direct
    +      or contributory patent infringement, then any patent licenses
    +      granted to You under this License for that Work shall terminate
    +      as of the date such litigation is filed.
    +
    +   4. Redistribution. You may reproduce and distribute copies of the
    +      Work or Derivative Works thereof in any medium, with or without
    +      modifications, and in Source or Object form, provided that You
    +      meet the following conditions:
    +
    +      (a) You must give any other recipients of the Work or
    +          Derivative Works a copy of this License; and
    +
    +      (b) You must cause any modified files to carry prominent notices
    +          stating that You changed the files; and
    +
    +      (c) You must retain, in the Source form of any Derivative Works
    +          that You distribute, all copyright, patent, trademark, and
    +          attribution notices from the Source form of the Work,
    +          excluding those notices that do not pertain to any part of
    +          the Derivative Works; and
    +
    +      (d) If the Work includes a "NOTICE" text file as part of its
    +          distribution, then any Derivative Works that You distribute must
    +          include a readable copy of the attribution notices contained
    +          within such NOTICE file, excluding those notices that do not
    +          pertain to any part of the Derivative Works, in at least one
    +          of the following places: within a NOTICE text file distributed
    +          as part of the Derivative Works; within the Source form or
    +          documentation, if provided along with the Derivative Works; or,
    +          within a display generated by the Derivative Works, if and
    +          wherever such third-party notices normally appear. The contents
    +          of the NOTICE file are for informational purposes only and
    +          do not modify the License. You may add Your own attribution
    +          notices within Derivative Works that You distribute, alongside
    +          or as an addendum to the NOTICE text from the Work, provided
    +          that such additional attribution notices cannot be construed
    +          as modifying the License.
    +
    +      You may add Your own copyright statement to Your modifications and
    +      may provide additional or different license terms and conditions
    +      for use, reproduction, or distribution of Your modifications, or
    +      for any such Derivative Works as a whole, provided Your use,
    +      reproduction, and distribution of the Work otherwise complies with
    +      the conditions stated in this License.
    +
    +   5. Submission of Contributions. Unless You explicitly state otherwise,
    +      any Contribution intentionally submitted for inclusion in the Work
    +      by You to the Licensor shall be under the terms and conditions of
    +      this License, without any additional terms or conditions.
    +      Notwithstanding the above, nothing herein shall supersede or modify
    +      the terms of any separate license agreement you may have executed
    +      with Licensor regarding such Contributions.
    +
    +   6. Trademarks. This License does not grant permission to use the trade
    +      names, trademarks, service marks, or product names of the Licensor,
    +      except as required for reasonable and customary use in describing the
    +      origin of the Work and reproducing the content of the NOTICE file.
    +
    +   7. Disclaimer of Warranty. Unless required by applicable law or
    +      agreed to in writing, Licensor provides the Work (and each
    +      Contributor provides its Contributions) on an "AS IS" BASIS,
    +      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
    +      implied, including, without limitation, any warranties or conditions
    +      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
    +      PARTICULAR PURPOSE. You are solely responsible for determining the
    +      appropriateness of using or redistributing the Work and assume any
    +      risks associated with Your exercise of permissions under this License.
    +
    +   8. Limitation of Liability. In no event and under no legal theory,
    +      whether in tort (including negligence), contract, or otherwise,
    +      unless required by applicable law (such as deliberate and grossly
    +      negligent acts) or agreed to in writing, shall any Contributor be
    +      liable to You for damages, including any direct, indirect, special,
    +      incidental, or consequential damages of any character arising as a
    +      result of this License or out of the use or inability to use the
    +      Work (including but not limited to damages for loss of goodwill,
    +      work stoppage, computer failure or malfunction, or any and all
    +      other commercial damages or losses), even if such Contributor
    +      has been advised of the possibility of such damages.
    +
    +   9. Accepting Warranty or Additional Liability. While redistributing
    +      the Work or Derivative Works thereof, You may choose to offer,
    +      and charge a fee for, acceptance of support, warranty, indemnity,
    +      or other liability obligations and/or rights consistent with this
    +      License. However, in accepting such obligations, You may act only
    +      on Your own behalf and on Your sole responsibility, not on behalf
    +      of any other Contributor, and only if You agree to indemnify,
    +      defend, and hold each Contributor harmless for any liability
    +      incurred by, or claims asserted against, such Contributor by reason
    +      of your accepting any such warranty or additional liability.
    +
    +   END OF TERMS AND CONDITIONS
    +
    +   APPENDIX: How to apply the Apache License to your work.
    +
    +      To apply the Apache License to your work, attach the following
    +      boilerplate notice, with the fields enclosed by brackets "{}"
    +      replaced with your own identifying information. (Don't include
    +      the brackets!)  The text should be enclosed in the appropriate
    +      comment syntax for the file format. We also recommend that a
    +      file or class name and description of purpose be included on the
    +      same "printed page" as the copyright notice for easier
    +      identification within third-party archives.
    +
    +   Copyright 2018-2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.
    +
    +   Licensed under the Apache License, Version 2.0 (the "License");
    +   you may not use this file except in compliance with the License.
    +   You may obtain a copy of the License at
    +
    +       http://www.apache.org/licenses/LICENSE-2.0
    +
    +   Unless required by applicable law or agreed to in writing, software
    +   distributed under the License is distributed on an "AS IS" BASIS,
    +   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +   See the License for the specific language governing permissions and
    +   limitations under the License.
    +
    +
    +
  • +
  • +

    Apache License 2.0

    +

    Used by:

    + +
                                  Apache License
    +                        Version 2.0 January 2004
    +                     http://www.apache.org/licenses/
    +
    +TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
    +
    +1. Definitions.
    +
    +   "License" shall mean the terms and conditions for use, reproduction,
    +   and distribution as defined by Sections 1 through 9 of this document.
    +
    +   "Licensor" shall mean the copyright owner or entity authorized by
    +   the copyright owner that is granting the License.
    +
    +   "Legal Entity" shall mean the union of the acting entity and all
    +   other entities that control, are controlled by, or are under common
    +   control with that entity. For the purposes of this definition,
    +   "control" means (i) the power, direct or indirect, to cause the
    +   direction or management of such entity, whether by contract or
    +   otherwise, or (ii) ownership of fifty percent (50%) or more of the
    +   outstanding shares, or (iii) beneficial ownership of such entity.
    +
    +   "You" (or "Your") shall mean an individual or Legal Entity
    +   exercising permissions granted by this License.
    +
    +   "Source" form shall mean the preferred form for making modifications,
    +   including but not limited to software source code, documentation
    +   source, and configuration files.
    +
    +   "Object" form shall mean any form resulting from mechanical
    +   transformation or translation of a Source form, including but
    +   not limited to compiled object code, generated documentation,
    +   and conversions to other media types.
    +
    +   "Work" shall mean the work of authorship, whether in Source or
    +   Object form, made available under the License, as indicated by a
    +   copyright notice that is included in or attached to the work
    +   (an example is provided in the Appendix below).
    +
    +   "Derivative Works" shall mean any work, whether in Source or Object
    +   form, that is based on (or derived from) the Work and for which the
    +   editorial revisions, annotations, elaborations, or other modifications
    +   represent, as a whole, an original work of authorship. For the purposes
    +   of this License, Derivative Works shall not include works that remain
    +   separable from, or merely link (or bind by name) to the interfaces of,
    +   the Work and Derivative Works thereof.
    +
    +   "Contribution" shall mean any work of authorship, including
    +   the original version of the Work and any modifications or additions
    +   to that Work or Derivative Works thereof, that is intentionally
    +   submitted to Licensor for inclusion in the Work by the copyright owner
    +   or by an individual or Legal Entity authorized to submit on behalf of
    +   the copyright owner. For the purposes of this definition, "submitted"
    +   means any form of electronic, verbal, or written communication sent
    +   to the Licensor or its representatives, including but not limited to
    +   communication on electronic mailing lists, source code control systems,
    +   and issue tracking systems that are managed by, or on behalf of, the
    +   Licensor for the purpose of discussing and improving the Work, but
    +   excluding communication that is conspicuously marked or otherwise
    +   designated in writing by the copyright owner as "Not a Contribution."
    +
    +   "Contributor" shall mean Licensor and any individual or Legal Entity
    +   on behalf of whom a Contribution has been received by Licensor and
    +   subsequently incorporated within the Work.
    +
    +2. Grant of Copyright License. Subject to the terms and conditions of
    +   this License, each Contributor hereby grants to You a perpetual,
    +   worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +   copyright license to reproduce, prepare Derivative Works of,
    +   publicly display, publicly perform, sublicense, and distribute the
    +   Work and such Derivative Works in Source or Object form.
    +
    +3. Grant of Patent License. Subject to the terms and conditions of
    +   this License, each Contributor hereby grants to You a perpetual,
    +   worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +   (except as stated in this section) patent license to make, have made,
    +   use, offer to sell, sell, import, and otherwise transfer the Work,
    +   where such license applies only to those patent claims licensable
    +   by such Contributor that are necessarily infringed by their
    +   Contribution(s) alone or by combination of their Contribution(s)
    +   with the Work to which such Contribution(s) was submitted. If You
    +   institute patent litigation against any entity (including a
    +   cross-claim or counterclaim in a lawsuit) alleging that the Work
    +   or a Contribution incorporated within the Work constitutes direct
    +   or contributory patent infringement, then any patent licenses
    +   granted to You under this License for that Work shall terminate
    +   as of the date such litigation is filed.
    +
    +4. Redistribution. You may reproduce and distribute copies of the
    +   Work or Derivative Works thereof in any medium, with or without
    +   modifications, and in Source or Object form, provided that You
    +   meet the following conditions:
    +
    +   (a) You must give any other recipients of the Work or
    +       Derivative Works a copy of this License; and
    +
    +   (b) You must cause any modified files to carry prominent notices
    +       stating that You changed the files; and
    +
    +   (c) You must retain, in the Source form of any Derivative Works
    +       that You distribute, all copyright, patent, trademark, and
    +       attribution notices from the Source form of the Work,
    +       excluding those notices that do not pertain to any part of
    +       the Derivative Works; and
    +
    +   (d) If the Work includes a "NOTICE" text file as part of its
    +       distribution, then any Derivative Works that You distribute must
    +       include a readable copy of the attribution notices contained
    +       within such NOTICE file, excluding those notices that do not
    +       pertain to any part of the Derivative Works, in at least one
    +       of the following places: within a NOTICE text file distributed
    +       as part of the Derivative Works; within the Source form or
    +       documentation, if provided along with the Derivative Works; or,
    +       within a display generated by the Derivative Works, if and
    +       wherever such third-party notices normally appear. The contents
    +       of the NOTICE file are for informational purposes only and
    +       do not modify the License. You may add Your own attribution
    +       notices within Derivative Works that You distribute, alongside
    +       or as an addendum to the NOTICE text from the Work, provided
    +       that such additional attribution notices cannot be construed
    +       as modifying the License.
    +
    +   You may add Your own copyright statement to Your modifications and
    +   may provide additional or different license terms and conditions
    +   for use, reproduction, or distribution of Your modifications, or
    +   for any such Derivative Works as a whole, provided Your use,
    +   reproduction, and distribution of the Work otherwise complies with
    +   the conditions stated in this License.
    +
    +5. Submission of Contributions. Unless You explicitly state otherwise,
    +   any Contribution intentionally submitted for inclusion in the Work
    +   by You to the Licensor shall be under the terms and conditions of
    +   this License, without any additional terms or conditions.
    +   Notwithstanding the above, nothing herein shall supersede or modify
    +   the terms of any separate license agreement you may have executed
    +   with Licensor regarding such Contributions.
    +
    +6. Trademarks. This License does not grant permission to use the trade
    +   names, trademarks, service marks, or product names of the Licensor,
    +   except as required for reasonable and customary use in describing the
    +   origin of the Work and reproducing the content of the NOTICE file.
    +
    +7. Disclaimer of Warranty. Unless required by applicable law or
    +   agreed to in writing, Licensor provides the Work (and each
    +   Contributor provides its Contributions) on an "AS IS" BASIS,
    +   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
    +   implied, including, without limitation, any warranties or conditions
    +   of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
    +   PARTICULAR PURPOSE. You are solely responsible for determining the
    +   appropriateness of using or redistributing the Work and assume any
    +   risks associated with Your exercise of permissions under this License.
    +
    +8. Limitation of Liability. In no event and under no legal theory,
    +   whether in tort (including negligence), contract, or otherwise,
    +   unless required by applicable law (such as deliberate and grossly
    +   negligent acts) or agreed to in writing, shall any Contributor be
    +   liable to You for damages, including any direct, indirect, special,
    +   incidental, or consequential damages of any character arising as a
    +   result of this License or out of the use or inability to use the
    +   Work (including but not limited to damages for loss of goodwill,
    +   work stoppage, computer failure or malfunction, or any and all
    +   other commercial damages or losses), even if such Contributor
    +   has been advised of the possibility of such damages.
    +
    +9. Accepting Warranty or Additional Liability. While redistributing
    +   the Work or Derivative Works thereof, You may choose to offer,
    +   and charge a fee for, acceptance of support, warranty, indemnity,
    +   or other liability obligations and/or rights consistent with this
    +   License. However, in accepting such obligations, You may act only
    +   on Your own behalf and on Your sole responsibility, not on behalf
    +   of any other Contributor, and only if You agree to indemnify,
    +   defend, and hold each Contributor harmless for any liability
    +   incurred by, or claims asserted against, such Contributor by reason
    +   of your accepting any such warranty or additional liability.
    +
    +END OF TERMS AND CONDITIONS
    +
    +APPENDIX: How to apply the Apache License to your work.
    +
    +   To apply the Apache License to your work, attach the following
    +   boilerplate notice, with the fields enclosed by brackets "[]"
    +   replaced with your own identifying information. (Don't include
    +   the brackets!)  The text should be enclosed in the appropriate
    +   comment syntax for the file format. We also recommend that a
    +   file or class name and description of purpose be included on the
    +   same "printed page" as the copyright notice for easier
    +   identification within third-party archives.
     
    -   Licensed under the Apache License, Version 2.0 (the "License");
    -   you may not use this file except in compliance with the License.
    -   You may obtain a copy of the License at
    +Copyright [yyyy] [name of copyright owner]
     
    -       http://www.apache.org/licenses/LICENSE-2.0
    +Licensed under the Apache License, Version 2.0 (the "License");
    +you may not use this file except in compliance with the License.
    +You may obtain a copy of the License at
     
    -   Unless required by applicable law or agreed to in writing, software
    -   distributed under the License is distributed on an "AS IS" BASIS,
    -   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    -   See the License for the specific language governing permissions and
    -   limitations under the License.
    +	http://www.apache.org/licenses/LICENSE-2.0
     
    +Unless required by applicable law or agreed to in writing, software
    +distributed under the License is distributed on an "AS IS" BASIS,
    +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +See the License for the specific language governing permissions and
    +limitations under the License.
     
  • Apache License 2.0

    Used by:

                                  Apache License
    -                        Version 2.0 January 2004
    +                        Version 2.0, January 2004
                          http://www.apache.org/licenses/
     
     TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
    @@ -1956,7 +2804,8 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright [yyyy] [name of copyright owner] +Copyright (c) 2016 Alex Crichton +Copyright (c) 2017 The Tokio Authors Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -1975,15 +2824,7 @@

    Used by:

    Apache License 2.0

    Used by:

                                  Apache License
                             Version 2.0, January 2004
    @@ -2173,8 +3014,7 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright (c) 2016 Alex Crichton -Copyright (c) 2017 The Tokio Authors +Copyright (c) 2019 Matthias Einwag Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -2193,7 +3033,7 @@

    Used by:

    Apache License 2.0

    Used by:

                                  Apache License
                             Version 2.0, January 2004
    @@ -2383,7 +3223,7 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright (c) 2019 Matthias Einwag +Copyright 2014 Paho Lurie-Gregg Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -2395,14 +3235,13 @@

    Used by:

    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and -limitations under the License. -
    +limitations under the License.
  • Apache License 2.0

    Used by:

                                  Apache License
                             Version 2.0, January 2004
    @@ -2592,7 +3431,7 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright 2014 Paho Lurie-Gregg +Copyright 2017 The Rust Project Developers Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -2604,13 +3443,15 @@

    Used by:

    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and -limitations under the License.
    +limitations under the License. +
  • Apache License 2.0

    Used by:

                                  Apache License
                             Version 2.0, January 2004
    @@ -2800,7 +3641,7 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright 2017 The Rust Project Developers +Copyright 2017 http-rs authors Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -2819,7 +3660,7 @@

    Used by:

    Apache License 2.0

    Used by:

                                  Apache License
                             Version 2.0, January 2004
    @@ -3009,7 +3850,7 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright 2017 http-rs authors +Copyright 2017 quininer kel Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -3028,7 +3869,7 @@

    Used by:

    Apache License 2.0

    Used by:

                                  Apache License
                             Version 2.0, January 2004
    @@ -3218,7 +4059,7 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright 2017 quininer kel +Copyright 2018 The pin-utils authors Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -4084,6 +4925,7 @@

    Used by:

  • base64 0.21.7
  • base64 0.22.1
  • bitflags 2.11.1
  • +
  • bytes-utils 0.1.4
  • cc 1.2.62
  • cfg-if 1.0.4
  • cmake 0.1.58
  • @@ -4101,15 +4943,18 @@

    Used by:

  • errno 0.3.14
  • event-listener-strategy 0.5.4
  • event-listener 5.4.2
  • +
  • fastrand 2.5.0
  • find-msvc-tools 0.1.9
  • flate2 1.1.9
  • fnv 1.0.7
  • form_urlencoded 1.2.2
  • hashbrown 0.14.5
  • hashbrown 0.15.5
  • +
  • hashbrown 0.16.1
  • hashbrown 0.17.1
  • heck 0.5.0
  • httparse 1.10.1
  • +
  • hyper-rustls 0.27.10
  • idna 1.1.0
  • idna_adapter 1.2.2
  • indexmap 2.14.0
  • @@ -4122,6 +4967,7 @@

    Used by:

  • num-integer 0.1.46
  • num-traits 0.2.19
  • once_cell 1.21.4
  • +
  • openssl-probe 0.2.1
  • parking 2.2.1
  • parking_lot 0.12.5
  • parking_lot_core 0.9.12
  • @@ -4132,8 +4978,12 @@

    Used by:

  • pest_meta 2.8.6
  • pkg-config 0.3.33
  • regex-automata 0.4.14
  • +
  • regex-lite 0.1.9
  • regex-syntax 0.8.10
  • +
  • ring 0.17.14
  • ron 0.8.1
  • +
  • rustc_version 0.4.1
  • +
  • rustls-native-certs 0.8.4
  • rustls 0.23.46
  • scopeguard 1.2.0
  • signal-hook-registry 1.4.8
  • @@ -4149,6 +4999,7 @@

    Used by:

  • url 2.5.8
  • uuid 1.23.1
  • version_check 0.9.5
  • +
  • xmlparser 0.13.6
  • yasna 0.6.0
  •                               Apache License
    @@ -4999,6 +5850,7 @@ 

    Used by:

  • md-5 0.10.6
  • opaque-debug 0.3.1
  • polyval 0.6.2
  • +
  • sha1 0.10.6
  • sha2 0.10.9
  • universal-hash 0.5.1
  • @@ -6669,6 +7521,7 @@

    Used by:

    +
  • +
  • +

    ISC License

    +

    Used by:

    + +
    Copyright 2015-2025 Brian Smith.
    +
    +Permission to use, copy, modify, and/or distribute this software for any
    +purpose with or without fee is hereby granted, provided that the above
    +copyright notice and this permission notice appear in all copies.
    +
    +THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
    +WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
    +MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY
    +SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
    +WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION
    +OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN
    +CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
     
  • @@ -7380,6 +8272,63 @@

    Used by:

    OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +
  • +
  • +

    MIT License

    +

    Used by:

    + +
    Copyright (c) 2018-2019 Sean McArthur
    +
    +Permission is hereby granted, free of charge, to any person obtaining a copy
    +of this software and associated documentation files (the "Software"), to deal
    +in the Software without restriction, including without limitation the rights
    +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    +copies of the Software, and to permit persons to whom the Software is
    +furnished to do so, subject to the following conditions:
    +
    +The above copyright notice and this permission notice shall be included in
    +all copies or substantial portions of the Software.
    +
    +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
    +THE SOFTWARE.
    +
    +
    +
  • +
  • +

    MIT License

    +

    Used by:

    + +
    Copyright (c) 2018-2023 Sean McArthur
    +Copyright (c) 2016 Alex Crichton
    +
    +Permission is hereby granted, free of charge, to any person obtaining a copy
    +of this software and associated documentation files (the "Software"), to deal
    +in the Software without restriction, including without limitation the rights
    +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    +copies of the Software, and to permit persons to whom the Software is
    +furnished to do so, subject to the following conditions:
    +
    +The above copyright notice and this permission notice shall be included in
    +all copies or substantial portions of the Software.
    +
    +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
    +THE SOFTWARE.
    +
     
  • @@ -7467,6 +8416,39 @@

    Used by:

    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +
  • +
  • +

    MIT License

    +

    Used by:

    + +
    Copyright (c) 2019 Hyper Contributors
    +
    +Permission is hereby granted, free of charge, to any
    +person obtaining a copy of this software and associated
    +documentation files (the "Software"), to deal in the
    +Software without restriction, including without
    +limitation the rights to use, copy, modify, merge,
    +publish, distribute, sublicense, and/or sell copies of
    +the Software, and to permit persons to whom the Software
    +is furnished to do so, subject to the following
    +conditions:
    +
    +The above copyright notice and this permission notice
    +shall be included in all copies or substantial portions
    +of the Software.
    +
    +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF
    +ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED
    +TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A
    +PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT
    +SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
    +CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
    +OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR
    +IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
    +DEALINGS IN THE SOFTWARE.
     
  • @@ -7784,6 +8766,34 @@

    Used by:

    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. +
  • +
  • +

    MIT License

    +

    Used by:

    + +
    MIT License
    +
    +Copyright (c) 2016 Jerome Froelich
    +
    +Permission is hereby granted, free of charge, to any person obtaining a copy
    +of this software and associated documentation files (the "Software"), to deal
    +in the Software without restriction, including without limitation the rights
    +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    +copies of the Software, and to permit persons to whom the Software is
    +furnished to do so, subject to the following conditions:
    +
    +The above copyright notice and this permission notice shall be included in all
    +copies or substantial portions of the Software.
    +
    +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
    +SOFTWARE.
  • MIT License

    @@ -7998,7 +9008,37 @@

    Used by:

    MIT License

    Used by:

    +
    MIT License
    +
    +Copyright (c) 2022 Nugine
    +
    +Permission is hereby granted, free of charge, to any person obtaining a copy
    +of this software and associated documentation files (the "Software"), to deal
    +in the Software without restriction, including without limitation the rights
    +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    +copies of the Software, and to permit persons to whom the Software is
    +furnished to do so, subject to the following conditions:
    +
    +The above copyright notice and this permission notice shall be included in all
    +copies or substantial portions of the Software.
    +
    +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
    +SOFTWARE.
    +
  • +
  • +

    MIT License

    +

    Used by:

    +
    MIT License
     
    @@ -8137,6 +9177,34 @@ 

    Used by:

    OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
    +
  • +
  • +

    MIT License

    +

    Used by:

    + +
    The MIT License (MIT)
    +
    +Copyright (c) 2014 Mathijs van de Nes
    +
    +Permission is hereby granted, free of charge, to any person obtaining a copy
    +of this software and associated documentation files (the "Software"), to deal
    +in the Software without restriction, including without limitation the rights
    +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    +copies of the Software, and to permit persons to whom the Software is
    +furnished to do so, subject to the following conditions:
    +
    +The above copyright notice and this permission notice shall be included in all
    +copies or substantial portions of the Software.
    +
    +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
    +SOFTWARE.
  • MIT License

    @@ -8542,6 +9610,7 @@

    zlib License

    Used by:

    Copyright (c) 2024 Orson Peters
     
    diff --git a/crates/app/Cargo.toml b/crates/app/Cargo.toml
    index a1e66490f..099ea21cf 100644
    --- a/crates/app/Cargo.toml
    +++ b/crates/app/Cargo.toml
    @@ -21,6 +21,8 @@ extenddb-engine = { workspace = true }
     extenddb-storage = { workspace = true }
     extenddb-config = { workspace = true }
     extenddb-server = { workspace = true }
    +extenddb-backup-store = { workspace = true }
    +futures = { workspace = true }
     tokio = { workspace = true }
     anyhow = { workspace = true }
     clap = { workspace = true }
    diff --git a/crates/app/src/cmd_backup.rs b/crates/app/src/cmd_backup.rs
    new file mode 100644
    index 000000000..5d70b2a20
    --- /dev/null
    +++ b/crates/app/src/cmd_backup.rs
    @@ -0,0 +1,709 @@
    +// Copyright 2026 ExtendDB contributors
    +// SPDX-License-Identifier: Apache-2.0
    +
    +//! `extenddb backup`: the operator's commands over the configured backup
    +//! store.
    +//!
    +//! `backup list` prints the catalog's rows for the configured store's kind
    +//! and root, every status and every store id the rows carry, with `--failed`
    +//! narrowing to the FAILED tombstones the reconciler keeps for the operator;
    +//! each line carries the `store_root` and the `store_id` so an operator sees
    +//! which root and which id a row names. It is a read: it opens the catalog
    +//! beside a running server (on SQLite without the serve lock and without the
    +//! startup recovery a server runs), reads the root's identity file without
    +//! creating one, and runs on an unhealthy store, printing the store's
    +//! problem in its header, since an operator whose store is unhealthy is the
    +//! one who most wants to see which rows name the root.
    +//!
    +//! `backup purge --confirm` deletes every backup of the configured store: the
    +//! prefix of each row (`///`, derived from the
    +//! ARN, never a whole account prefix, so a prefix with no row survives) and
    +//! the row itself, outright, leaving no tombstone. It leaves
    +//! `extenddb-store.json`, which is under no account prefix, so the root
    +//! keeps its identity, and the `exports/` component. It needs the store
    +//! healthy, its identity known: a purge over an unhealthy store would act on
    +//! rows it cannot trust. Before it deletes it takes the catalog-level
    +//! maintenance claim, which it heartbeats on a timer for the claim's whole
    +//! life, at the period the jobs heartbeat their rows at
    +//! (`extenddb_storage::backup_catalog::JOB_HEARTBEAT_INTERVAL`, so the
    +//! `stale_job_after_secs` floor that keeps a live job from being declared
    +//! dead keeps this claim held too), so every reader of the claim sees it
    +//! held from the take to the release, through a deletion phase of any
    +//! length; a heartbeat that finds the claim lost, or fails, stops the
    +//! deletion loop before its next row. It then waits, up to
    +//! `2 * stale_job_after_secs`, until no CREATING or FAILED backup row of the
    +//! store has a live owner (a heartbeat within `stale_job_after_secs`; a
    +//! FAILED row's heartbeat may still be refreshed by a job that has not yet
    +//! seen its claim lost, and its prefix must not be deleted under a manifest
    +//! it may still write) and no restore job is RUNNING; it refuses and
    +//! releases the claim if they have not gone by then, printing the wait. A
    +//! CREATING row the wait saw no live owner for is deleted too: the row
    +//! first (`remove_dead_creating_backup_row`, on the catalog's clock, which
    +//! leaves a row whose heartbeat is young after all, and its objects with
    +//! it), then its prefix. A DELETING row is not
    +//! waited on: nothing heartbeats one, the reconciler finishes any it finds,
    +//! and the deletion is the same idempotent sequence purge runs on every row
    +//! (two deleters of one prefix do not fail each other, and the row's moves
    +//! are conditional), so purge finishes it itself; for the same reason a row
    +//! a DeleteBackup moved to DELETING or DELETED, or removed, between the
    +//! command's read and its turn is finished and counted as deleted, not left
    +//! (`extenddb_engine::maintenance::purge_row` states each state's step).
    +//! Every backup and restore
    +//! job checks the claim before it starts and refuses while it is held
    +//! (`LimitExceededException`; the engine's backup module states where in
    +//! the job's order the check runs). The wait ends early, with the claim
    +//! released and nothing deleted, when a heartbeat finds the claim lost, when
    +//! the rows cannot be read, or on `SIGINT` (Ctrl-C) or `SIGTERM`; the same
    +//! signal during the deletion ends the loop before its next row, never
    +//! inside one, so a stopped command leaves no half-deleted row and no claim
    +//! for the next command or server to wait out. Without `--confirm` the
    +//! command is a
    +//! read like `list` (no serve lock, nothing created): it prints what it
    +//! would delete, changes nothing and exits 0, with no store configured as
    +//! with one. Each deletion is logged under
    +//! the `extenddb::audit` target with the account and the ARN.
    +//!
    +//! On SQLite purge opens the database the way `serve` does, which takes the
    +//! serve lock: it refuses while a server runs on the file, and a server
    +//! refuses to start while it runs. Under that lock every job owner is dead
    +//! (one process serves the file, and this is it), and no reconciler runs
    +//! while the command holds the file, so before the wait the command runs
    +//! the reconciler's duties itself with a zero stale window: a legacy restore
    +//! the stopped server left RUNNING is failed and its target removed, a
    +//! target a FAILED restore left CREATING is removed, an in-catalog CREATING
    +//! backup it left is removed, and a DELETING row is finished; the wait and
    +//! the CREATING-row removal then also use a zero window, so a CREATING or
    +//! FAILED row whose heartbeat is young (the stopped server's job wrote it)
    +//! is neither waited on nor left, and the wait has nothing to wait for. The
    +//! server's own startup under the same lock clears a maintenance claim a
    +//! stopped command left, for the same reason. On the other backends
    +//! purge runs beside live servers; the maintenance claim is the exclusion.
    +//!
    +//! `extenddb destroy` removes the databases and leaves the store untouched:
    +//! the store is the copy that survives destroy, and this command is the one
    +//! that deletes it.
    +
    +use std::sync::Arc;
    +use std::time::{Duration, Instant};
    +
    +use clap::{Args, Subcommand};
    +use extenddb_config as config;
    +use extenddb_engine::maintenance::{Interrupt, Wait, WaitOutcome, purge_row, wait_for_quiet};
    +use extenddb_engine::{BackupRuntime, ConfiguredStore, HeldClaim, StoreHealth, StoreProblem};
    +use extenddb_storage::StorageEngine;
    +use extenddb_storage::backup_catalog::{
    +    BackupRow, BackupStatus, JOB_HEARTBEAT_INTERVAL, StoreIdentity,
    +};
    +use extenddb_storage::error::StorageError;
    +use futures::TryStreamExt;
    +
    +#[derive(Args)]
    +#[allow(clippy::doc_markdown)] // Clap help text, not rustdoc
    +pub struct BackupArgs {
    +    /// Path to configuration file (accepted before or after the subcommand)
    +    #[arg(short, long, default_value = "extenddb.toml", global = true)]
    +    config: String,
    +
    +    #[command(subcommand)]
    +    command: BackupCommand,
    +}
    +
    +#[derive(Subcommand)]
    +#[allow(clippy::doc_markdown)] // Clap help text, not rustdoc
    +enum BackupCommand {
    +    /// List the catalog's backup rows for the configured backup store. A read that runs
    +    /// beside a server, on an unhealthy store too.
    +    List {
    +        /// Only the FAILED rows (backups whose creation failed, kept for the operator)
    +        #[arg(long)]
    +        failed: bool,
    +    },
    +    /// Delete every backup of the configured backup store: each backup's objects and its
    +    /// catalog row. Leaves extenddb-store.json, exports/ and any prefix with no row.
    +    /// Takes the maintenance claim first and waits for running jobs to finish.
    +    Purge {
    +        /// Confirm the deletion (required; without it the command prints what it would delete
    +        /// and exits 0)
    +        #[arg(long)]
    +        confirm: bool,
    +    },
    +}
    +
    +/// How often the waiting purge re-reads the rows.
    +const WAIT_POLL: Duration = Duration::from_secs(2);
    +
    +pub async fn run(args: BackupArgs, release: &'static str) -> anyhow::Result<()> {
    +    if !std::path::Path::new(&args.config).exists() {
    +        anyhow::bail!(
    +            "Config file '{}' not found; use --config  to specify its location.",
    +            args.config,
    +        );
    +    }
    +    let app_config = config::load(&args.config)?;
    +    match args.command {
    +        BackupCommand::List { failed } => match open(&app_config, release, Mode::Read).await? {
    +            Opening::Store(opened) => list(&opened, failed).await,
    +            Opening::NoStore(reason) => {
    +                println!("=== backups in the configured store: none ({reason}) ===");
    +                println!("  (none)");
    +                Ok(())
    +            }
    +        },
    +        BackupCommand::Purge { confirm: false } => {
    +            // The dry run is a read, as `list` is: no serve lock, nothing
    +            // created, the plan from the rows of the root; with no store,
    +            // the plan is empty and says why, as `list` does.
    +            match open(&app_config, release, Mode::Read).await? {
    +                Opening::Store(opened) => dry_run(&opened).await,
    +                Opening::NoStore(reason) => {
    +                    println!("=== extenddb backup purge ===");
    +                    println!("Store:  none ({reason})");
    +                    println!();
    +                    println!("--- Backups to delete (each backup's objects and its catalog row):");
    +                    println!("  (none)");
    +                    println!();
    +                    println!("Dry run: nothing was deleted.");
    +                    Ok(())
    +                }
    +            }
    +        }
    +        BackupCommand::Purge { confirm: true } => {
    +            match open(&app_config, release, Mode::Write).await? {
    +                Opening::Store(opened) => purge(&opened, &app_config).await,
    +                Opening::NoStore(reason) => anyhow::bail!("{reason}"),
    +            }
    +        }
    +    }
    +}
    +
    +/// What `open` found: the configured store, or why there is none.
    +enum Opening {
    +    Store(Opened),
    +    NoStore(String),
    +}
    +
    +/// How a command opens the catalog.
    +#[derive(Debug, Clone, Copy, PartialEq, Eq)]
    +enum Mode {
    +    /// A read beside a running server: no serve lock, no startup recovery,
    +    /// the identity file read and never written.
    +    Read,
    +    /// As `serve` opens it (on SQLite under the serve lock); the store's
    +    /// check may write the identity file of an empty root.
    +    Write,
    +}
    +
    +/// The storage engine, the backup runtime with the configured store, and
    +/// the store's identity as the command read it.
    +struct Opened {
    +    storage: Arc,
    +    runtime: Arc,
    +    backend: String,
    +    identity: Identity,
    +}
    +
    +/// The store's identity as the command read it.
    +enum Identity {
    +    /// The check passed, or the file was read and agrees with the rows.
    +    Known(StoreIdentity),
    +    /// The root has no identity file and the catalog no rows for it: a
    +    /// fresh root nothing has written to.
    +    Empty,
    +    /// The check failed (a listing still runs; a purge refuses).
    +    Problem(StoreProblem),
    +}
    +
    +impl Opened {
    +    fn store(&self) -> &ConfiguredStore {
    +        self.runtime
    +            .configured_store()
    +            .expect("open returns only with a configured store")
    +    }
    +}
    +
    +async fn open(
    +    app_config: &config::AppConfig,
    +    release: &'static str,
    +    mode: Mode,
    +) -> anyhow::Result {
    +    let mut options = extenddb_storage::server_components::ServerComponentsOptions::default();
    +    options.release = release;
    +    options.max_concurrent_jobs = app_config.backup.max_concurrent_jobs;
    +    options.beside_server = mode == Mode::Read;
    +    let mut storage_config = app_config.storage.clone();
    +    storage_config.set_instance_id(&format!(
    +        "{}:backup-command",
    +        extenddb_server::backup_store::hostname()
    +    ));
    +    let components = extenddb_storage::create_server_components(
    +        storage_config.as_trait(),
    +        &app_config.server.region,
    +        options,
    +    )
    +    .await
    +    .map_err(|e| anyhow::anyhow!("cannot open the catalog: {e}"))?;
    +    let storage = components.engine;
    +    let backend = app_config.storage.backend.clone();
    +    let setup = extenddb_server::backup_store::configure_described(
    +        &app_config.backup,
    +        app_config.storage.database_file().as_deref(),
    +        &backend,
    +        &*storage,
    +        match mode {
    +            Mode::Read => extenddb_server::backup_store::Caller::ReadOnlyCommand,
    +            Mode::Write => extenddb_server::backup_store::Caller::Command,
    +        },
    +    )
    +    .await?;
    +    let runtime = Arc::new(setup.runtime);
    +    let Some(store) = runtime.configured_store() else {
    +        // Why there is none, as `configure_described` found it: the section
    +        // absent or disabled, the default root not yet created, or the
    +        // default root failed its check with no rows for it.
    +        let why = setup
    +            .not_configured
    +            .expect("a runtime without a store carries the reason");
    +        return Ok(Opening::NoStore(why.to_string()));
    +    };
    +    // The server's startup leaves an S3 store's check to its reconciler;
    +    // the command runs it here. A read leaves the root as it finds it: the
    +    // identity file is read, never created, under the store's check bound as
    +    // the server's check is, so an endpoint that accepts and never answers
    +    // or a mount that has stopped answering costs the command the bound and
    +    // not the store's retries.
    +    let identity = match (mode, store.health()) {
    +        (Mode::Read, _) => match extenddb_engine::read_identity_bounded(store, &*storage).await {
    +            Ok(Some(identity)) => Identity::Known(identity),
    +            Ok(None) => Identity::Empty,
    +            Err(problem) => Identity::Problem(problem),
    +        },
    +        (Mode::Write, StoreHealth::Checking) => {
    +            let outcome = extenddb_engine::check_configured_store(store, &*storage, &backend).await;
    +            store.record_check(outcome.clone());
    +            match outcome {
    +                Ok(identity) => Identity::Known(identity),
    +                Err(problem) => Identity::Problem(problem),
    +            }
    +        }
    +        (Mode::Write, StoreHealth::Healthy) => match store.identity() {
    +            Some(identity) => Identity::Known(identity),
    +            None => Identity::Problem(StoreProblem::Catalog(
    +                "the store's id is unknown".to_owned(),
    +            )),
    +        },
    +        (Mode::Write, StoreHealth::Unhealthy(problem)) => Identity::Problem(problem),
    +    };
    +    Ok(Opening::Store(Opened {
    +        storage,
    +        runtime,
    +        backend,
    +        identity,
    +    }))
    +}
    +
    +async fn rows_of(
    +    storage: &dyn StorageEngine,
    +    identity: &StoreIdentity,
    +    status: Option,
    +) -> anyhow::Result> {
    +    storage
    +        .list_backup_rows(status, identity)
    +        .try_collect()
    +        .await
    +        .map_err(|e| anyhow::anyhow!("cannot list the catalog's backup rows: {e}"))
    +}
    +
    +fn print_row(row: &BackupRow) {
    +    println!(
    +        "{:<9} {:<12} {:<32} {:>12} {:>10}  {}  store_root={} store_id={}{}",
    +        row.status,
    +        row.account_id,
    +        row.table_name,
    +        row.size_bytes,
    +        row.item_count,
    +        row.backup_arn,
    +        row.store.store_root,
    +        row.store.store_id,
    +        row.failure_reason
    +            .as_deref()
    +            .map(|r| format!("  failure_reason={r:?}"))
    +            .unwrap_or_default()
    +    );
    +}
    +
    +/// Every row of the store's kind and root, whatever id it carries: the ids
    +/// the rows record, plus the file's when the check read one.
    +async fn rows_of_root(
    +    opened: &Opened,
    +    status: Option,
    +) -> anyhow::Result<(Vec, Vec)> {
    +    let store = opened.store();
    +    let mut ids = opened
    +        .storage
    +        .read_store_ids(store.kind, &store.root)
    +        .await
    +        .map_err(|e| anyhow::anyhow!("cannot read the catalog's store ids: {e}"))?;
    +    if let Identity::Known(identity) = &opened.identity
    +        && !ids.contains(&identity.store_id)
    +    {
    +        ids.push(identity.store_id.clone());
    +    }
    +    let mut rows = Vec::new();
    +    for id in &ids {
    +        let identity = StoreIdentity {
    +            store_kind: store.kind,
    +            store_root: store.root.clone(),
    +            store_id: id.clone(),
    +        };
    +        rows.extend(rows_of(&*opened.storage, &identity, status).await?);
    +    }
    +    Ok((ids, rows))
    +}
    +
    +async fn list(opened: &Opened, failed: bool) -> anyhow::Result<()> {
    +    let status = failed.then_some(BackupStatus::Failed);
    +    let store = opened.store();
    +    let (ids, rows) = rows_of_root(opened, status).await?;
    +    match &opened.identity {
    +        Identity::Known(identity) => println!(
    +            "=== backups in the {} store {} (store id {}) ===",
    +            store.kind, store.root, identity.store_id
    +        ),
    +        Identity::Empty => println!(
    +            "=== backups in the {} store {} (the root has no {} yet; a server's first check \
    +             creates it) ===",
    +            store.kind,
    +            store.root,
    +            extenddb_engine::STORE_IDENTITY_KEY
    +        ),
    +        Identity::Problem(problem) => println!(
    +            "=== backups in the {} store {} (the store is unhealthy: {problem}; rows of every \
    +             id the catalog records for the root are listed) ===",
    +            store.kind, store.root
    +        ),
    +    }
    +    if ids.len() > 1 {
    +        println!(
    +            "  the catalog's rows for this root carry {} store ids: {}",
    +            ids.len(),
    +            ids.join(", ")
    +        );
    +    }
    +    if rows.is_empty() {
    +        println!("  (none)");
    +        return Ok(());
    +    }
    +    for row in &rows {
    +        print_row(row);
    +    }
    +    println!("{} row(s)", rows.len());
    +    Ok(())
    +}
    +
    +/// The plan a purge prints before it deletes, and the dry run's whole
    +/// output: the store, its rows, and what stays.
    +async fn print_plan(opened: &Opened) -> anyhow::Result<()> {
    +    let configured = opened.store();
    +    println!("=== extenddb backup purge ===");
    +    match &opened.identity {
    +        Identity::Known(identity) => println!(
    +            "Store:  {} {} (store id {})",
    +            identity.store_kind, identity.store_root, identity.store_id
    +        ),
    +        Identity::Empty => println!(
    +            "Store:  {} {} (no {} yet)",
    +            configured.kind,
    +            configured.root,
    +            extenddb_engine::STORE_IDENTITY_KEY
    +        ),
    +        Identity::Problem(problem) => println!(
    +            "Store:  {} {} (unhealthy: {problem})",
    +            configured.kind, configured.root
    +        ),
    +    }
    +    let (_, planned) = rows_of_root(opened, None).await?;
    +    println!();
    +    println!("--- Backups to delete (each backup's objects and its catalog row):");
    +    if planned.is_empty() {
    +        println!("  (none)");
    +    }
    +    for row in &planned {
    +        print_row(row);
    +    }
    +    println!();
    +    println!(
    +        "Left in place: {}, exports/, and any prefix the catalog has no row for.",
    +        extenddb_engine::STORE_IDENTITY_KEY
    +    );
    +    println!();
    +    Ok(())
    +}
    +
    +/// `backup purge` without `--confirm`: the plan, nothing changed, exit 0.
    +async fn dry_run(opened: &Opened) -> anyhow::Result<()> {
    +    print_plan(opened).await?;
    +    println!(
    +        "Dry run: nothing was deleted. Run again with --confirm to permanently delete every \
    +         backup of the configured store listed above."
    +    );
    +    Ok(())
    +}
    +
    +async fn purge(opened: &Opened, app_config: &config::AppConfig) -> anyhow::Result<()> {
    +    let backup = &app_config.backup;
    +    let storage = &*opened.storage;
    +    let configured = opened.store();
    +    let identity = match &opened.identity {
    +        Identity::Known(identity) => identity,
    +        Identity::Empty => anyhow::bail!(
    +            "the {} backup store {} has no {} and the catalog has no rows for it: nothing to purge",
    +            configured.kind,
    +            configured.root,
    +            extenddb_engine::STORE_IDENTITY_KEY
    +        ),
    +        Identity::Problem(problem) => anyhow::bail!(
    +            "the {} backup store {} is unhealthy: {problem}; fix that before purging its backups",
    +            configured.kind,
    +            configured.root
    +        ),
    +    };
    +    let store = &*configured.store;
    +    print_plan(opened).await?;
    +
    +    let stale_after = Duration::from_secs(backup.stale_job_after_secs);
    +    let holder = format!(
    +        "extenddb backup purge on {} (pid {})",
    +        extenddb_server::backup_store::hostname(),
    +        std::process::id()
    +    );
    +    // The signal listener is up before the claim is taken, so a signal
    +    // from here on ends the command with the claim released: at once during
    +    // the wait, before the next row during the deletion.
    +    let mut interrupt = listen_for_signals();
    +    let claim = storage
    +        .maintenance_claim(&holder, stale_after)
    +        .await
    +        .map_err(|e| match e {
    +            StorageError::AlreadyExists(m) => anyhow::anyhow!(
    +                "{m}; one maintenance command runs at a time, and a claim whose holder died \
    +                 expires after stale_job_after_secs ({} s)",
    +                backup.stale_job_after_secs
    +            ),
    +            other => anyhow::anyhow!("cannot take the maintenance claim: {other}"),
    +        })?;
    +    println!("Took the maintenance claim as {holder:?}.");
    +    let claim = HeldClaim::start(claim, JOB_HEARTBEAT_INTERVAL);
    +
    +    // On SQLite the command holds the file alone: every job owner is dead
    +    // and no reconciler runs, so their resolution is this command's, and the
    +    // owner of any row is dead whatever its heartbeat says, so the window a
    +    // live owner is told apart by is zero. On the other backends purge runs
    +    // beside live servers, whose jobs heartbeat within `stale_after`.
    +    let owner_window = if opened.backend == "sqlite" {
    +        resolve_dead_owners(opened).await;
    +        Duration::ZERO
    +    } else {
    +        stale_after
    +    };
    +
    +    // Wait for the jobs the claim excludes from starting to finish.
    +    let wait_limit = stale_after * 2;
    +    let started = Instant::now();
    +    let outcome = wait_for_quiet(
    +        storage,
    +        identity,
    +        &claim,
    +        Wait {
    +            owner_window,
    +            limit: wait_limit,
    +            poll: WAIT_POLL,
    +        },
    +        &mut interrupt,
    +        |remaining| {
    +            println!(
    +                "Waiting for running jobs to finish: {}",
    +                remaining.join("; ")
    +            )
    +        },
    +    )
    +    .await;
    +    match outcome {
    +        WaitOutcome::Quiet => {}
    +        WaitOutcome::Busy(remaining) => {
    +            release(claim).await;
    +            anyhow::bail!(
    +                "jobs were still running after waiting {} s (2 * stale_job_after_secs): {}. The \
    +                 maintenance claim is released; retry once they finish",
    +                started.elapsed().as_secs(),
    +                remaining.join("; ")
    +            );
    +        }
    +        WaitOutcome::Lost(reason) => {
    +            // The release is conditioned on the generation: another holder's
    +            // claim is left to it.
    +            release(claim).await;
    +            anyhow::bail!("stopped before any deletion: {reason}; retry once the claim is free");
    +        }
    +        WaitOutcome::CatalogError(e) => {
    +            release(claim).await;
    +            anyhow::bail!(
    +                "stopped before any deletion: {e}. The maintenance claim is released; retry once \
    +                 the catalog answers"
    +            );
    +        }
    +        WaitOutcome::Interrupted(signal) => {
    +            release(claim).await;
    +            anyhow::bail!(
    +                "stopped before any deletion: {signal} received while waiting. The maintenance \
    +                 claim is released"
    +            );
    +        }
    +    }
    +
    +    // Re-read under the claim: the plan printed before it may be stale.
    +    let rows = match rows_of(storage, identity, None).await {
    +        Ok(rows) => rows,
    +        Err(e) => {
    +            release(claim).await;
    +            return Err(e);
    +        }
    +    };
    +    let mut deleted = 0usize;
    +    let mut skipped = Vec::new();
    +    let mut stopped = None;
    +    for row in &rows {
    +        if let Some(reason) = claim.lost() {
    +            stopped = Some(format!("{reason}; retry once the claim is free"));
    +            break;
    +        }
    +        if let Some(signal) = interrupt.signalled() {
    +            stopped = Some(format!(
    +                "{signal} received. The maintenance claim is released; run the command again \
    +                 for the rest"
    +            ));
    +            break;
    +        }
    +        if let Err(e) = purge_row(storage, store, row, owner_window).await {
    +            skipped.push(format!("{}: {e}", row.backup_arn));
    +            continue;
    +        }
    +        tracing::info!(
    +            target: "extenddb::audit",
    +            action = "backup purge",
    +            account_id = %row.account_id,
    +            backup_arn = %row.backup_arn,
    +            store_root = %row.store.store_root,
    +            "deleted backup objects and catalog row"
    +        );
    +        println!("deleted {} ({})", row.backup_arn, row.status);
    +        deleted += 1;
    +    }
    +    release(claim).await;
    +    println!();
    +    println!(
    +        "=== extenddb backup purge complete: {deleted} backup(s) deleted, {} left ===",
    +        rows.len() - deleted
    +    );
    +    if let Some(reason) = stopped {
    +        anyhow::bail!("stopped before the remaining rows: {reason}");
    +    }
    +    if !skipped.is_empty() {
    +        for line in &skipped {
    +            println!("  left: {line}");
    +        }
    +        anyhow::bail!(
    +            "{} backup(s) could not be deleted; see above",
    +            skipped.len()
    +        );
    +    }
    +    Ok(())
    +}
    +
    +/// The handle the wait and the deletion loop read a signal from, fed by a
    +/// task that waits for `SIGINT` (Ctrl-C) or `SIGTERM`. The first signal
    +/// stops the command at its next safe point (the wait at once, the loop
    +/// before its next row) with the claim released; a second one exits the
    +/// process at once, the claim then expiring by the stale rule (on SQLite
    +/// the next server start clears it).
    +fn listen_for_signals() -> Interrupt {
    +    let (tx, interrupt) = Interrupt::new();
    +    tokio::spawn(async move {
    +        let name = signal_name().await;
    +        println!(
    +            "{name} received; finishing the current step and releasing the maintenance claim (a \
    +             second {name} exits at once)."
    +        );
    +        let _ = tx.send(Some(name));
    +        let again = signal_name().await;
    +        println!(
    +            "{again} received again; exiting now. The maintenance claim is left to expire by the \
    +             stale rule."
    +        );
    +        std::process::exit(130);
    +    });
    +    interrupt
    +}
    +
    +#[cfg(unix)]
    +async fn signal_name() -> &'static str {
    +    use tokio::signal::unix::{SignalKind, signal};
    +    let ctrl_c = tokio::signal::ctrl_c();
    +    let Ok(mut sigterm) = signal(SignalKind::terminate()) else {
    +        let _ = ctrl_c.await;
    +        return "SIGINT";
    +    };
    +    tokio::select! {
    +        _ = ctrl_c => "SIGINT",
    +        _ = sigterm.recv() => "SIGTERM",
    +    }
    +}
    +
    +#[cfg(not(unix))]
    +async fn signal_name() -> &'static str {
    +    let _ = tokio::signal::ctrl_c().await;
    +    "Ctrl-C"
    +}
    +
    +/// The reconciler's duties, run once with a zero stale window: on SQLite
    +/// under the serve lock every owner is dead. What was resolved is printed;
    +/// a failure is printed and the wait decides.
    +async fn resolve_dead_owners(opened: &Opened) {
    +    match extenddb_engine::reconciler::cycle(
    +        &*opened.storage,
    +        &opened.runtime,
    +        Duration::ZERO,
    +        &opened.backend,
    +        &mut extenddb_engine::reconciler::ReconcilerState::default(),
    +    )
    +    .await
    +    {
    +        Ok(report) => {
    +            for table in &report.legacy_restores_removed {
    +                println!("removed the restore target {table}: its restore did not finish");
    +            }
    +            for table in &report.failed_targets_removed {
    +                println!("removed the restore target {table}: its restore had failed");
    +            }
    +            for table in &report.targets_left_to_owner {
    +                println!(
    +                    "left the table {table}: it is no longer the restore target its record names"
    +                );
    +            }
    +            for arn in &report.legacy_backups_removed {
    +                println!("removed the in-catalog backup {arn}: its create did not finish");
    +            }
    +            for arn in &report.deletions_finished {
    +                println!("finished the deletion of {arn}");
    +            }
    +        }
    +        Err(e) => println!("could not resolve the jobs the stopped server left: {e}"),
    +    }
    +}
    +
    +/// Release the claim; a failure is printed, since the claim then expires by
    +/// the stale rule and the command's own outcome is what the operator needs.
    +async fn release(claim: HeldClaim) {
    +    if let Err(e) = claim.release().await {
    +        println!("could not release the maintenance claim ({e}); it expires on its own");
    +    }
    +}
    diff --git a/crates/app/src/cmd_destroy.rs b/crates/app/src/cmd_destroy.rs
    index a9f95e023..3966d929b 100755
    --- a/crates/app/src/cmd_destroy.rs
    +++ b/crates/app/src/cmd_destroy.rs
    @@ -3,7 +3,10 @@
     
     //! `extenddb destroy` — tear down a extenddb deployment (REQ-CAT-012).
     //!
    -//! Reads config, enumerates tables, requires `--yes` to confirm, drops both databases.
    +//! Reads config, enumerates tables, requires `--yes` to confirm, drops both
    +//! databases. The backup store (`[backup.store]`) is not touched: it is the
    +//! copy that survives destroy, and `extenddb backup purge --confirm` is the
    +//! command that deletes it.
     
     use clap::Args;
     
    @@ -25,7 +28,8 @@ pub struct DestroyArgs {
         #[arg(long)]
         pg_pass: Option,
     
    -    /// Confirm destruction (required, no interactive prompt)
    +    /// Confirm destruction (required, no interactive prompt). The backup store is left
    +    /// untouched; use `extenddb backup purge --confirm` to delete its backups.
         #[arg(long)]
         yes: bool,
     }
    @@ -90,6 +94,12 @@ pub async fn run(args: DestroyArgs) -> anyhow::Result<()> {
     
         println!();
         println!("WARNING: This will permanently destroy ALL data in both databases.");
    +    println!(
    +        "The backup store ([backup.store]) is left untouched; `extenddb backup purge --confirm` \
    +         deletes its backups. Its extenddb-store.json names this catalog: a catalog re-created \
    +         with `init` needs that file removed (or a new root), or the store is reported owned \
    +         by another catalog."
    +    );
         println!();
     
         if !args.yes {
    diff --git a/crates/app/src/cmd_serve.rs b/crates/app/src/cmd_serve.rs
    index 60ae99fd9..d03750712 100755
    --- a/crates/app/src/cmd_serve.rs
    +++ b/crates/app/src/cmd_serve.rs
    @@ -174,6 +174,13 @@ pub fn run(args: &ServeArgs, build: BuildInfo) -> anyhow::Result<()> {
             backend,
             config::redact_password(app_config.storage.connection_config()),
         );
    +    let banner_line3 = format!(
    +        "  {}",
    +        extenddb_server::backup_store::banner_line(
    +            &app_config.backup,
    +            app_config.storage.database_file().as_deref(),
    +        )
    +    );
         let mut warnings = app_config.storage.startup_warnings();
         warnings.extend(app_config.server.startup_warnings());
         let startup_warnings = render_startup_warnings(&warnings);
    @@ -181,11 +188,13 @@ pub fn run(args: &ServeArgs, build: BuildInfo) -> anyhow::Result<()> {
             eprint!("{wordmark}");
             eprintln!("{banner_line1}");
             eprintln!("{banner_line2}");
    +        eprintln!("{banner_line3}");
             eprint!("{startup_warnings}");
         } else {
             print!("{wordmark}");
             println!("{banner_line1}");
             println!("{banner_line2}");
    +        println!("{banner_line3}");
             print!("{startup_warnings}");
         }
         if cfg!(feature = "dev-mode") {
    @@ -275,10 +284,14 @@ pub fn run(args: &ServeArgs, build: BuildInfo) -> anyhow::Result<()> {
         if config_exists {
             params = params.with_config_path(&args.config);
         }
    -    let result = tokio::runtime::Builder::new_multi_thread()
    +    // The runtime is ended with a bound (`run_then_shut_down`): a backup
    +    // store check abandoned on a mount that has stopped answering leaves a
    +    // blocking thread parked there, and a plain drop of the runtime would
    +    // join it, holding the process open after the drain until SIGKILL.
    +    let runtime = tokio::runtime::Builder::new_multi_thread()
             .enable_all()
    -        .build()?
    -        .block_on(extenddb_server::serve(params));
    +        .build()?;
    +    let result = crate::run_then_shut_down(runtime, extenddb_server::serve(params));
         // `serve` has printed the refusal; end with its code, which `migrate`
         // and `verify` use for the same status.
         if let Err(e) = &result
    diff --git a/crates/app/src/init_helpers.rs b/crates/app/src/init_helpers.rs
    index 15e700f3f..8be891e82 100755
    --- a/crates/app/src/init_helpers.rs
    +++ b/crates/app/src/init_helpers.rs
    @@ -312,7 +312,30 @@ key_path = "{tls_key}"
     # failed_row_retention_secs = 604800   # FAILED tombstones kept for the operator (7 days)
     # reconcile_interval_secs = 60         # Reconciler cycle
     # max_concurrent_jobs = 4              # Backup and restore jobs running at once on this instance
    -# spool_path = "/var/lib/extenddb/spool" # Restore download area for an S3 store
    +# spool_path = "/var/lib/extenddb/spool" # Restore download area for an S3 store; required
    +#                                        # with an S3 store except on file-backed SQLite, where
    +#                                        # it defaults to backup-spool/ beside the database file
    +
    +# [backup.store]
    +# Where CreateBackup's backups will live, from a later release (this release's
    +# CreateBackup writes in-catalog backups with any store, and the store serves
    +# restores and deletes of the backups it holds). Absent on file-backed SQLite: a filesystem
    +# store at backups/ beside the database file. Absent on PostgreSQL, MongoDB,
    +# Cassandra and an in-memory SQLite database: no store, and CreateBackup
    +# writes in-catalog backups with a deprecation warning. kind = "disabled"
    +# opts out on any backend. Instances that share a catalog must share one
    +# store root (an S3 bucket and prefix, or a shared mount).
    +# kind = "filesystem"                  # "filesystem" | "s3" | "disabled"
    +# [backup.store.filesystem]
    +# path = "/var/lib/extenddb/backups"   # Must exist, owned by the extenddb user, mode 0700
    +# [backup.store.s3]
    +# bucket = "my-extenddb-backups"
    +# prefix = "extenddb/"                 # Optional key prefix
    +# region = "us-east-1"                 # Optional; the AWS SDK's chain otherwise
    +# endpoint = "http://127.0.0.1:9000"   # Optional, for an S3-compatible server
    +# force_path_style = true              # Required with an http:// endpoint
    +# upload_part_size_bytes = 8388608     # 5 MiB to 5 GiB
    +# max_concurrent_uploads = 4
     
     # [import]
     # paths = []                     # Allowed directories for import operations
    diff --git a/crates/app/src/lib.rs b/crates/app/src/lib.rs
    index 153470867..492d842ee 100644
    --- a/crates/app/src/lib.rs
    +++ b/crates/app/src/lib.rs
    @@ -3,9 +3,9 @@
     
     //! ExtendDB application/CLI library.
     //!
    -//! Owns the command-line interface (`serve`, `init`, `destroy`, `verify`,
    -//! `migrate`, `status`, `stop`, `settings`, `manage`, `catalog-check`,
    -//! `healthcheck`) and the
    +//! Owns the command-line interface (`serve`, `init`, `destroy`, `backup`,
    +//! `verify`, `migrate`, `status`, `stop`, `settings`, `manage`,
    +//! `catalog-check`, `healthcheck`) and the
     //! subcommand dispatch. It is backend-agnostic: a backend's thin `main`
     //! installs its backend with
     //! [`set_backend`](extenddb_storage::set_backend) and then calls [`run`]:
    @@ -22,6 +22,7 @@
     //! }
     //! ```
     
    +mod cmd_backup;
     mod cmd_catalog_check;
     mod cmd_destroy;
     mod cmd_healthcheck;
    @@ -66,8 +67,10 @@ enum Command {
         Serve(cmd_serve::ServeArgs),
         /// Initialize a new extenddb deployment
         Init(cmd_init::InitArgs),
    -    /// Tear down a extenddb deployment
    +    /// Tear down a extenddb deployment (the databases; the backup store is left untouched)
         Destroy(cmd_destroy::DestroyArgs),
    +    /// List or purge the backups of the configured backup store
    +    Backup(cmd_backup::BackupArgs),
         /// Validate a extenddb deployment
         Verify(cmd_verify::VerifyArgs),
         /// Apply catalog migrations (exit 0 up to date or applied, 1 error, 3 pending without --yes,
    @@ -116,6 +119,7 @@ pub fn run(build: BuildInfo) -> anyhow::Result<()> {
                 Ok(())
             }
             Command::Destroy(args) => run_interactive(cmd_destroy::run(args)),
    +        Command::Backup(args) => run_interactive(cmd_backup::run(args, build.version)),
             Command::Verify(args) => {
                 let code = run_interactive(cmd_verify::run(args, build.version))?;
                 if code != 0 {
    @@ -192,8 +196,75 @@ fn run_interactive(
         tracing_subscriber::fmt()
             .try_init()
             .unwrap_or_else(|e| eprintln!("Warning: logging init failed: {e}"));
    -    tokio::runtime::Builder::new_current_thread()
    +    let runtime = tokio::runtime::Builder::new_current_thread()
             .enable_all()
    -        .build()?
    -        .block_on(future)
    +        .build()?;
    +    run_then_shut_down(runtime, future)
    +}
    +
    +/// How long the process waits at exit for the runtime's blocking threads
    +/// after the command's future returns. A thread parked in a filesystem call
    +/// on a mount that has stopped answering (a backup store check the engine
    +/// abandoned at its bound) returns only when the mount does; past this bound
    +/// the thread is detached and the process exits without it.
    +pub(crate) const RUNTIME_SHUTDOWN_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(5);
    +
    +/// Run `future` to completion on `runtime`, then end the runtime within
    +/// [`RUNTIME_SHUTDOWN_TIMEOUT`]. Dropping the runtime instead joins every
    +/// blocking thread, so one parked syscall held the process open past its
    +/// shutdown until SIGKILL.
    +pub(crate) fn run_then_shut_down(
    +    runtime: tokio::runtime::Runtime,
    +    future: impl std::future::Future,
    +) -> T {
    +    let output = runtime.block_on(future);
    +    runtime.shutdown_timeout(RUNTIME_SHUTDOWN_TIMEOUT);
    +    output
    +}
    +
    +#[cfg(test)]
    +mod tests {
    +    use super::{RUNTIME_SHUTDOWN_TIMEOUT, run_then_shut_down};
    +
    +    /// A blocking task that never returns (a filesystem call on a mount that
    +    /// has stopped answering) whose handle the caller dropped does not hold
    +    /// the runtime's end past the bound, on both runtimes the binary builds
    +    /// (`serve`'s multi-thread one and the commands' current-thread one).
    +    #[test]
    +    fn a_parked_blocking_thread_does_not_hold_the_runtime_past_its_bound() {
    +        let builders = [
    +            tokio::runtime::Builder::new_multi_thread(),
    +            tokio::runtime::Builder::new_current_thread(),
    +        ];
    +        for mut builder in builders {
    +            let runtime = builder.enable_all().build().unwrap();
    +            let (done, finished) = std::sync::mpsc::channel();
    +            let started = std::time::Instant::now();
    +            std::thread::spawn(move || {
    +                run_then_shut_down(runtime, async {
    +                    // Parked for good; the thread is leaked on purpose and
    +                    // dies with the test process.
    +                    drop(tokio::task::spawn_blocking(|| {
    +                        loop {
    +                            std::thread::park();
    +                        }
    +                    }));
    +                    // Let the blocking pool pick the task up before the
    +                    // runtime ends, so a thread is parked when it does.
    +                    tokio::time::sleep(std::time::Duration::from_millis(50)).await;
    +                });
    +                let _ = done.send(());
    +            });
    +            // The test's own guard, well past the bound: a shutdown that
    +            // joins the parked thread fails here instead of hanging.
    +            finished
    +                .recv_timeout(RUNTIME_SHUTDOWN_TIMEOUT * 6)
    +                .expect("the runtime ends within its shutdown bound");
    +            let elapsed = started.elapsed();
    +            assert!(
    +                elapsed < RUNTIME_SHUTDOWN_TIMEOUT * 3,
    +                "the runtime's end took {elapsed:?}"
    +            );
    +        }
    +    }
     }
    diff --git a/crates/backup-store/README.md b/crates/backup-store/README.md
    index dd6251534..ae741ecfa 100644
    --- a/crates/backup-store/README.md
    +++ b/crates/backup-store/README.md
    @@ -77,6 +77,26 @@ store's callers honour; this README is the record of what the store does.
     - `put` returns the stored object's metadata (bytes written and, on S3, the
       ETag with quotes stripped; a multipart ETag is not an MD5) so a manifest
       can be written from the put alone.
    +- `put_if_absent` creates the object only if no object is under the key and
    +  returns `None`, leaving the existing object untouched, if one is; of two
    +  concurrent calls on one key exactly one stores its body. The filesystem
    +  store writes and syncs its temporary as `put` does (created
    +  `O_CREAT | O_EXCL`) and then `link`s it to the key, which fails if
    +  anything is there, so the object appears complete or not at all; the root
    +  must therefore be on a filesystem that supports hard links (some CIFS and
    +  FUSE mounts do not, and the call fails there with `EPERM`, which the
    +  server reports as the store having failed validation, with the `link`
    +  error in the log line). The S3
    +  store sends `PutObject` (or `CompleteMultipartUpload` for a body over one
    +  part) with `If-None-Match: *`: a 412 is the key present, and a 409
    +  `ConditionalRequestConflict` from a concurrent conditional write is
    +  re-sent after a short back-off, up to eight times. An S3-compatible
    +  server that ignores `If-None-Match` on `PutObject` turns the call into a
    +  plain put, last writer wins; moto 5.2 honours it (the conformance suite's
    +  race cell checks that exactly one of eight concurrent callers stores its
    +  body), other servers have not been run. The caller that depends on the
    +  guarantee re-reads the object after its call and compares, which is what
    +  catches a server that lost the race silently.
     - The S3 store takes credentials from the AWS SDK's default chain (the
       `s3` module doc names its members, among them `credential_process`,
       which runs a command named in the shared config file, and the SSO
    diff --git a/crates/backup-store/src/config.rs b/crates/backup-store/src/config.rs
    index 002a43e59..927329603 100644
    --- a/crates/backup-store/src/config.rs
    +++ b/crates/backup-store/src/config.rs
    @@ -11,7 +11,9 @@
     //!
     //! ```toml
     //! [backup.store]
    -//! kind = "filesystem"                   # "filesystem" | "s3"; filesystem when absent
    +//! kind = "filesystem"                   # "filesystem" | "s3" | "disabled"; filesystem when absent
    +//!                                       # ("disabled" configures no store on purpose; the server
    +//!                                       # then creates in-catalog backups and logs that)
     //!
     //! [backup.store.filesystem]
     //! path = "/backups"           # required when kind = "filesystem"; the server defaults it
    @@ -167,6 +169,21 @@ pub enum BackupStoreKind {
         Filesystem,
         /// S3 bucket store.
         S3,
    +    /// No store, on purpose: the opt-out of the server's per-backend default
    +    /// root. [`crate::open`] refuses it; a caller checks the kind first.
    +    Disabled,
    +}
    +
    +impl BackupStoreKind {
    +    /// The value `kind` takes in the configuration.
    +    #[must_use]
    +    pub const fn as_str(self) -> &'static str {
    +        match self {
    +            Self::Filesystem => "filesystem",
    +            Self::S3 => "s3",
    +            Self::Disabled => "disabled",
    +        }
    +    }
     }
     
     /// The `[backup.store]` table.
    @@ -175,7 +192,7 @@ pub enum BackupStoreKind {
     /// survive `serde(flatten)`, so the server nests this struct under its own
     /// key (`store`) in its `[backup]` model rather than flattening it; that
     /// keeps one definition of these fields and unknown-field rejection on both.
    -#[derive(Debug, Clone, Deserialize)]
    +#[derive(Debug, Clone, PartialEq, Eq, Deserialize)]
     #[serde(deny_unknown_fields)]
     pub struct BackupStoreConfig {
         /// Selected store kind; `filesystem` when absent.
    @@ -190,7 +207,7 @@ pub struct BackupStoreConfig {
     }
     
     /// `[backup.store.filesystem]` settings.
    -#[derive(Debug, Clone, Deserialize)]
    +#[derive(Debug, Clone, PartialEq, Eq, Deserialize)]
     #[serde(deny_unknown_fields)]
     pub struct FilesystemStoreConfig {
         /// Root directory for backup objects. Must exist at startup.
    @@ -201,7 +218,7 @@ pub struct FilesystemStoreConfig {
     /// from the standard AWS SDK chain. `Debug` prints the endpoint without its
     /// user information, which `open` refuses but a log of the configuration
     /// written before `open` would otherwise carry.
    -#[derive(Clone, Deserialize)]
    +#[derive(Clone, PartialEq, Eq, Deserialize)]
     #[serde(deny_unknown_fields)]
     pub struct S3StoreConfig {
         /// Destination bucket.
    @@ -375,6 +392,15 @@ mod tests {
             assert!(toml::from_str::("kind = \"gcs\"").is_err());
         }
     
    +    #[tokio::test]
    +    async fn disabled_parses_and_does_not_open() {
    +        let config: BackupStoreConfig = toml::from_str("kind = \"disabled\"").unwrap();
    +        assert_eq!(config.kind, BackupStoreKind::Disabled);
    +        assert_eq!(BackupStoreKind::Disabled.as_str(), "disabled");
    +        let err = open_err(&config).await;
    +        assert!(err.to_string().contains("disabled"), "{err}");
    +    }
    +
         #[tokio::test]
         async fn open_requires_the_matching_section() {
             let missing_fs = BackupStoreConfig {
    diff --git a/crates/backup-store/src/fs.rs b/crates/backup-store/src/fs.rs
    index 48bcb5895..948ecc093 100644
    --- a/crates/backup-store/src/fs.rs
    +++ b/crates/backup-store/src/fs.rs
    @@ -80,6 +80,16 @@
     //! concurrent put into the same new directory, and could not cover a
     //! failure inside the creation walk itself.)
     //!
    +//! `put_if_absent` writes and syncs its temporary the same way, created
    +//! `O_CREAT | O_EXCL` as every temporary is, and then `link`s it to the key
    +//! instead of renaming: `link(2)` fails with `EEXIST` when anything is at
    +//! the new path, so of two concurrent calls on one key exactly one links and
    +//! the other reads `None`, the first body stays, and the object appears
    +//! under its key complete or not at all. The temporary is unlinked after the
    +//! link on every path (the two names are one inode, so the object stays). A
    +//! regular file already at the destination is reported `None` before the
    +//! body is read.
    +//!
     //! Where this store narrows the shared key space (temporary names, a file
     //! and a directory at one path, the filesystem's own name limit and case
     //! folding) is listed in the `key` module doc.
    @@ -142,10 +152,15 @@ impl FilesystemStore {
         ///
         /// # Errors
         ///
    -    /// Returns an error when the root does not exist, cannot be resolved, or
    -    /// is not a directory.
    +    /// Returns an error when the root path is empty, the root does not exist,
    +    /// cannot be resolved, or is not a directory.
         pub async fn open(root: impl AsRef) -> Result {
             let root = root.as_ref();
    +        if root.as_os_str().is_empty() {
    +            return Err(StoreError::Other(
    +                "the backup root path is empty; name the directory".to_owned(),
    +            ));
    +        }
             let canonical = tokio::fs::canonicalize(root).await.map_err(|e| {
                 StoreError::io(format!("backup root {} is not usable", root.display()), e)
             })?;
    @@ -165,6 +180,14 @@ impl FilesystemStore {
             })
         }
     
    +    /// The root label a catalog row records for this store: the canonical
    +    /// root directory. Two servers configured for one directory (by any
    +    /// path that resolves to it) spell the same label.
    +    #[must_use]
    +    pub fn store_root(&self) -> String {
    +        self.root.display().to_string()
    +    }
    +
         /// Run `hook` on the blocking thread of every `delete_prefix` walk
         /// right before each removal (a file, a symlink or a directory), with
         /// the number of removals that walk has made so far. The crate's own
    @@ -313,7 +336,14 @@ impl FilesystemStore {
             Ok(current)
         }
     
    -    async fn put_impl(&self, key: &str, mut body: ByteStream) -> Result {
    +    /// Resolve `key` to the directory its object is written into and the
    +    /// object's final path, creating the directories (`prepare_parent`) and
    +    /// refusing a symlink or a directory at the destination: a put never
    +    /// resolves through a link, and a directory there is a key-space
    +    /// collision (`a/b` exists, `a` is put), refused before the body is read
    +    /// rather than after the rename or link fails. Reports whether a regular
    +    /// file is already at the destination, which `put_if_absent` reads.
    +    async fn destination(&self, key: &str) -> Result {
             validate_key(key)?;
             let components: Vec<&str> = key.split('/').collect();
             let (file_name, parent_components) = components
    @@ -321,12 +351,7 @@ impl FilesystemStore {
                 .expect("a validated key has at least one component");
             let parent = self.prepare_parent(parent_components).await?;
             let final_path = parent.join(file_name);
    -
    -        // A symlink at the destination, in-root or escaping, is refused: a
    -        // put never resolves through a link. A directory there is a key-space
    -        // collision (`a/b` exists, `a` is put), refused before the body is
    -        // read rather than after the rename fails.
    -        match tokio::fs::symlink_metadata(&final_path).await {
    +        let exists = match tokio::fs::symlink_metadata(&final_path).await {
                 Ok(meta) if meta.is_symlink() => {
                     return Err(symlink_refused(&final_path));
                 }
    @@ -339,13 +364,101 @@ impl FilesystemStore {
                         ),
                     ));
                 }
    -            Ok(_) | Err(_) => {}
    -        }
    +            Ok(_) => true,
    +            Err(_) => false,
    +        };
    +        Ok(Destination {
    +            parent,
    +            final_path,
    +            exists,
    +        })
    +    }
     
    +    /// Write `body` to a new temporary sibling of `final_path`, flushed and
    +    /// synced, and return its path, the guard that removes it, and the
    +    /// number of bytes written. The temporary is created `O_CREAT | O_EXCL`
    +    /// with mode `0600` (`create_private_file`).
    +    ///
    +    /// The temporary and its guard come back together. The guard removes
    +    /// the temporary on every exit the caller does not disarm it for: an
    +    /// error, the put's future dropped (a caller's timeout, a cancelled
    +    /// job), including while the open itself is in flight on the blocking
    +    /// pool, or a body that panics.
    +    async fn write_temporary(
    +        parent: &Path,
    +        mut body: ByteStream,
    +    ) -> Result<(PathBuf, RemoveOnDrop, u64), StoreError> {
             let temp_path = parent.join(format!(
                 "{TEMP_PREFIX}{}{TEMP_SUFFIX}",
                 uuid::Uuid::new_v4()
             ));
    +        let (mut file, temp_guard) = create_private_file(&temp_path).await?;
    +        let mut size: u64 = 0;
    +        while let Some(chunk) = body.next().await {
    +            let chunk = chunk?;
    +            file.write_all(&chunk)
    +                .await
    +                .map_err(|e| StoreError::io(format!("write {}", temp_path.display()), e))?;
    +            size += chunk.len() as u64;
    +        }
    +        // tokio's `File` hands each write to a background task and
    +        // reports its error on the next operation. `sync_all` waits for
    +        // the last background write but does not return its error (tokio
    +        // keeps only its kind for a later write or flush), so without a
    +        // flush before the drop a short file would be renamed into place
    +        // on a full disk (or past RLIMIT_FSIZE). `flush` returns it.
    +        file.flush()
    +            .await
    +            .map_err(|e| StoreError::io(format!("write {}", temp_path.display()), e))?;
    +        // Defence in depth, no test: the fsync of the temporary matters
    +        // only across a crash or power loss, which the suite cannot
    +        // inject.
    +        file.sync_all()
    +            .await
    +            .map_err(|e| StoreError::io(format!("sync {}", temp_path.display()), e))?;
    +        drop(file);
    +        Ok((temp_path, temp_guard, size))
    +    }
    +
    +    /// Sync `parent` and every directory between it and the root, so the
    +    /// entry a rename or link just made, and the directories a put created
    +    /// on the way, are durable. Syncing every ancestor is a few extra
    +    /// fsyncs per put (one per path component) against a bounded depth;
    +    /// cheaper than tracking which ones this put created, and correct when a
    +    /// concurrent put created them. Defence in depth, no test: it matters
    +    /// only across a crash.
    +    async fn sync_parent_chain(&self, parent: &Path) -> Result<(), StoreError> {
    +        sync_dir(parent).await?;
    +        let mut dir = parent.to_path_buf();
    +        while let Some(p) = dir.parent() {
    +            if !p.starts_with(&self.root) || p == self.root {
    +                sync_dir(&self.root).await?;
    +                break;
    +            }
    +            sync_dir(p).await?;
    +            dir = p.to_path_buf();
    +        }
    +        Ok(())
    +    }
    +
    +    /// The metadata of the object now at `final_path`.
    +    async fn stored_meta(key: &str, final_path: &Path, size: u64) -> ObjectMeta {
    +        let last_modified = tokio::fs::symlink_metadata(final_path)
    +            .await
    +            .ok()
    +            .and_then(|m| m.modified().ok());
    +        ObjectMeta {
    +            key: key.to_owned(),
    +            size,
    +            last_modified,
    +            etag: None,
    +        }
    +    }
    +
    +    async fn put_impl(&self, key: &str, body: ByteStream) -> Result {
    +        let Destination {
    +            parent, final_path, ..
    +        } = self.destination(key).await?;
     
             // Durability contract: the temp file is synced before the rename and
             // the parent directory is synced after it, so an Ok return means both
    @@ -353,37 +466,9 @@ impl FilesystemStore {
             // directory sync the rename itself could be lost on power failure
             // even though the file data had been written.
             async {
    -            // The temporary and its guard come back together. The guard
    -            // removes it on every exit but the rename: an error (`?` below),
    -            // the put's future dropped (a caller's timeout, a cancelled
    -            // job), including while the open itself is in flight on the
    -            // blocking pool, or a body that panics. It lives inside this
    -            // block so dropping the block's future drops it.
    -            let (mut file, mut temp_guard) = create_private_file(&temp_path).await?;
    -            let mut size: u64 = 0;
    -            while let Some(chunk) = body.next().await {
    -                let chunk = chunk?;
    -                file.write_all(&chunk)
    -                    .await
    -                    .map_err(|e| StoreError::io(format!("write {}", temp_path.display()), e))?;
    -                size += chunk.len() as u64;
    -            }
    -            // tokio's `File` hands each write to a background task and
    -            // reports its error on the next operation. `sync_all` waits for
    -            // the last background write but does not return its error (tokio
    -            // keeps only its kind for a later write or flush), so without a
    -            // flush before the drop a short file would be renamed into place
    -            // on a full disk (or past RLIMIT_FSIZE). `flush` returns it.
    -            file.flush()
    -                .await
    -                .map_err(|e| StoreError::io(format!("write {}", temp_path.display()), e))?;
    -            // Defence in depth, no test: the fsync of the temporary matters
    -            // only across a crash or power loss, which the suite cannot
    -            // inject.
    -            file.sync_all()
    -                .await
    -                .map_err(|e| StoreError::io(format!("sync {}", temp_path.display()), e))?;
    -            drop(file);
    +            // The guard lives inside this block so dropping the block's
    +            // future drops it (`write_temporary`).
    +            let (temp_path, mut temp_guard, size) = Self::write_temporary(&parent, body).await?;
                 tokio::fs::rename(&temp_path, &final_path)
                     .await
                     .map_err(|e| {
    @@ -400,32 +485,56 @@ impl FilesystemStore {
                 // before the put into one that holds nothing, which is worse
                 // than either complete object (the trait contract is "no partial
                 // object", not "no object").
    -            sync_dir(&parent).await?;
    -            // The directories between the root and the parent are durable
    -            // only once their parents' entries are synced too. Syncing every
    -            // ancestor is a few extra fsyncs per put (one per path component)
    -            // against a bounded depth; cheaper than tracking which ones this
    -            // put created, and correct when a concurrent put created them.
    -            // Defence in depth, no test: it matters only across a crash.
    -            let mut dir = parent.clone();
    -            while let Some(p) = dir.parent() {
    -                if !p.starts_with(&self.root) || p == self.root {
    -                    sync_dir(&self.root).await?;
    -                    break;
    +            self.sync_parent_chain(&parent).await?;
    +            Ok(Self::stored_meta(key, &final_path, size).await)
    +        }
    +        .await
    +    }
    +
    +    /// `put_if_absent`: the temporary is written and synced as `put` writes
    +    /// it, then `link`ed to the key. `link(2)` fails with `EEXIST` when
    +    /// anything is at the new path (a file, a directory, a symlink, whatever
    +    /// it points at), so the object appears under its key complete or not at
    +    /// all, and an object that was there first is untouched. A regular file
    +    /// already at the destination skips the body: the link would fail, and
    +    /// writing the body first would only cost disk. The temporary is removed
    +    /// after the link on every path; the link and the temporary are one
    +    /// inode, so the object stays.
    +    async fn put_if_absent_impl(
    +        &self,
    +        key: &str,
    +        body: ByteStream,
    +    ) -> Result, StoreError> {
    +        let Destination {
    +            parent,
    +            final_path,
    +            exists,
    +        } = self.destination(key).await?;
    +        if exists {
    +            return Ok(None);
    +        }
    +        async {
    +            let (temp_path, temp_guard, size) = Self::write_temporary(&parent, body).await?;
    +            let linked = tokio::fs::hard_link(&temp_path, &final_path).await;
    +            // The temporary's name is surplus from here on whether the
    +            // link landed or not; the guard removes it when it drops at the
    +            // end of this block (and on every early return below).
    +            match linked {
    +                Ok(()) => {}
    +                Err(e) if e.kind() == std::io::ErrorKind::AlreadyExists => {
    +                    drop(temp_guard);
    +                    return Ok(None);
    +                }
    +                Err(e) => {
    +                    return Err(StoreError::io(
    +                        format!("link {} to {}", temp_path.display(), final_path.display()),
    +                        e,
    +                    ));
                     }
    -                sync_dir(p).await?;
    -                dir = p.to_path_buf();
                 }
    -            let last_modified = tokio::fs::symlink_metadata(&final_path)
    -                .await
    -                .ok()
    -                .and_then(|m| m.modified().ok());
    -            Ok(ObjectMeta {
    -                key: key.to_owned(),
    -                size,
    -                last_modified,
    -                etag: None,
    -            })
    +            drop(temp_guard);
    +            self.sync_parent_chain(&parent).await?;
    +            Ok(Some(Self::stored_meta(key, &final_path, size).await))
             }
             .await
         }
    @@ -512,15 +621,22 @@ impl FilesystemStore {
         }
     
         async fn validate_impl(&self) -> Result<(), StoreError> {
    -        let meta = tokio::fs::symlink_metadata(&self.root).await.map_err(|e| {
    -            StoreError::io(
    -                format!("backup root {} is not usable", self.root.display()),
    -                e,
    -            )
    -        })?;
    -        check_root_is_directory(&self.root, &meta)?;
    -        check_root_location(&self.root)?;
    -        check_root_ownership(&self.root, &meta)?;
    +        // The root's own checks are synchronous filesystem calls (`lstat` of
    +        // the root and each ancestor, `realpath`); they run on the blocking
    +        // pool so a root on a mount that has stopped answering parks a
    +        // blocking thread and not the runtime, and a caller's timeout on
    +        // this future can fire.
    +        let root = self.root.clone();
    +        tokio::task::spawn_blocking(move || {
    +            let meta = std::fs::symlink_metadata(&root).map_err(|e| {
    +                StoreError::io(format!("backup root {} is not usable", root.display()), e)
    +            })?;
    +            check_root_is_directory(&root, &meta)?;
    +            check_root_location(&root)?;
    +            check_root_ownership(&root, &meta)
    +        })
    +        .await
    +        .map_err(|e| StoreError::Other(format!("root check task failed: {e}")))??;
             let probe = self.root.join(format!(
                 "{TEMP_PREFIX}validate-{}{TEMP_SUFFIX}",
                 uuid::Uuid::new_v4()
    @@ -574,6 +690,15 @@ impl FilesystemStore {
         }
     }
     
    +/// Where a put writes: the directory the object goes into (created and
    +/// checked by `prepare_parent`), the object's final path, and whether a
    +/// regular file is already there.
    +struct Destination {
    +    parent: PathBuf,
    +    final_path: PathBuf,
    +    exists: bool,
    +}
    +
     impl BackupStore for FilesystemStore {
         fn put<'a>(
             &'a self,
    @@ -583,6 +708,14 @@ impl BackupStore for FilesystemStore {
             self.put_impl(key, body).boxed()
         }
     
    +    fn put_if_absent<'a>(
    +        &'a self,
    +        key: &'a str,
    +        body: ByteStream,
    +    ) -> BoxFuture<'a, Result, StoreError>> {
    +        self.put_if_absent_impl(key, body).boxed()
    +    }
    +
         fn get<'a>(&'a self, key: &'a str) -> BoxFuture<'a, Result> {
             self.get_impl(key).boxed()
         }
    @@ -624,6 +757,28 @@ impl BackupStore for FilesystemStore {
     /// cannot detect. Messages name the path and the fix. No check is made on
     /// other platforms.
     fn check_root_ownership(root: &Path, meta: &std::fs::Metadata) -> Result<(), StoreError> {
    +    check_private_tree(root, meta, "backup root")
    +}
    +
    +/// The owner-only rule of a directory the server keeps private, by its
    +/// canonical path: [`check_root_ownership`]'s rule (the directory owned by
    +/// the effective user with no group or other permission bits; every ancestor
    +/// a directory owned by the effective user or by root, not writable by group
    +/// or others unless sticky) for a directory that is not a store root, named
    +/// `what` in the messages (the server's restore spool). No check is made on
    +/// other platforms.
    +///
    +/// # Errors
    +///
    +/// `PermissionDenied` naming the directory or ancestor and the mode it
    +/// needs; `Io` when a directory on the way cannot be read.
    +pub fn check_private_directory(dir: &Path, what: &str) -> Result<(), StoreError> {
    +    let meta = std::fs::symlink_metadata(dir)
    +        .map_err(|e| StoreError::io(format!("stat {}", dir.display()), e))?;
    +    check_private_tree(dir, &meta, what)
    +}
    +
    +fn check_private_tree(root: &Path, meta: &std::fs::Metadata, what: &str) -> Result<(), StoreError> {
         #[cfg(unix)]
         {
             use std::os::unix::fs::MetadataExt;
    @@ -637,8 +792,8 @@ fn check_root_ownership(root: &Path, meta: &std::fs::Metadata) -> Result<(), Sto
                 // only a swap between the two reaches this (module doc).
                 if m.is_symlink() {
                     return Err(StoreError::PermissionDenied(format!(
    -                    "{} (an ancestor of the backup root) is a symlink; configure the backup \
    -                     root by its resolved path",
    +                    "{} (an ancestor of the {what}) is a symlink; configure the {what} by its \
    +                     resolved path",
                         dir.display()
                     )));
                 }
    @@ -646,17 +801,17 @@ fn check_root_ownership(root: &Path, meta: &std::fs::Metadata) -> Result<(), Sto
                 let sticky = mode & 0o1000 != 0;
                 if m.uid() != euid && m.uid() != 0 {
                     return Err(StoreError::PermissionDenied(format!(
    -                    "{} (an ancestor of the backup root) is owned by uid {}; the backup root \
    -                     must sit under directories owned by root or by the extenddb user",
    +                    "{} (an ancestor of the {what}) is owned by uid {}; the {what} must sit \
    +                     under directories owned by root or by the extenddb user",
                         dir.display(),
                         m.uid()
                     )));
                 }
                 if mode & 0o022 != 0 && !sticky {
                     return Err(StoreError::PermissionDenied(format!(
    -                    "{} (an ancestor of the backup root) has mode {mode:04o} and is writable by \
    -                     group or others without the sticky bit; another user could replace the \
    -                     root. chmod it to remove group and world write, or move the backup root",
    +                    "{} (an ancestor of the {what}) has mode {mode:04o} and is writable by group \
    +                     or others without the sticky bit; another user could replace the {what}. \
    +                     chmod it to remove group and world write, or move the {what}",
                         dir.display()
                     )));
                 }
    @@ -665,7 +820,7 @@ fn check_root_ownership(root: &Path, meta: &std::fs::Metadata) -> Result<(), Sto
         }
         #[cfg(not(unix))]
         {
    -        let _ = (root, meta);
    +        let _ = (root, meta, what);
         }
         Ok(())
     }
    diff --git a/crates/backup-store/src/lib.rs b/crates/backup-store/src/lib.rs
    index 5793dce10..23e7526ad 100644
    --- a/crates/backup-store/src/lib.rs
    +++ b/crates/backup-store/src/lib.rs
    @@ -33,8 +33,11 @@ mod s3;
     
     pub use config::{BackupStoreConfig, BackupStoreKind, FilesystemStoreConfig, S3StoreConfig};
     pub use error::StoreError;
    -pub use fs::FilesystemStore;
    -pub use s3::{MAX_PART_SIZE_BYTES, MAX_STORE_PREFIX_LEN, MIN_PART_SIZE_BYTES, S3Store};
    +pub use fs::{FilesystemStore, check_private_directory};
    +pub use s3::{
    +    ATTEMPT_TIMEOUT as S3_ATTEMPT_TIMEOUT, MAX_PART_SIZE_BYTES, MAX_STORE_PREFIX_LEN,
    +    MIN_PART_SIZE_BYTES, S3Store,
    +};
     
     /// Byte stream flowing into and out of a store.
     ///
    @@ -106,6 +109,31 @@ pub trait BackupStore: Send + Sync {
             body: ByteStream,
         ) -> BoxFuture<'a, Result>;
     
    +    /// Store `body` under `key` only if no object is there, and return the
    +    /// stored object's metadata; return `None`, leaving the existing object
    +    /// untouched and reading none of `body` the store did not already read,
    +    /// if one is. Of two concurrent calls on one key exactly one stores its
    +    /// body. The object appears under its key complete or not at all, as
    +    /// with [`BackupStore::put`], and a call that is dropped or fails before
    +    /// the object is in place leaves nothing (the same guards as `put`); a
    +    /// filesystem call whose `link` succeeded and whose parent-directory
    +    /// sync then fails returns `Err` with the object in place, as `put`
    +    /// does after its `rename`. Written for the store identity file; the
    +    /// mechanics are the stores': S3 sends `PutObject` (or, for a body over
    +    /// one part, `CompleteMultipartUpload`) with `If-None-Match: *`, where a
    +    /// 412 means the key is present and a 409 `ConditionalRequestConflict`
    +    /// from a concurrent conditional write is retried; the filesystem store
    +    /// writes and syncs its temporary as `put` does, created
    +    /// `O_CREAT | O_EXCL`, then `link`s it to the key, which fails if the
    +    /// key exists. An S3-compatible server that ignores `If-None-Match` on
    +    /// `PutObject` makes this a plain put, last writer wins; the README
    +    /// names it, and the caller's re-read is what catches it.
    +    fn put_if_absent<'a>(
    +        &'a self,
    +        key: &'a str,
    +        body: ByteStream,
    +    ) -> BoxFuture<'a, Result, StoreError>>;
    +
         /// Stream the object stored under `key`.
         ///
         /// Returns [`StoreError::NotFound`] when no object exists.
    @@ -183,13 +211,37 @@ pub trait BackupStore: Send + Sync {
     ///
     /// # Errors
     ///
    -/// Returns an error when the section matching the selected store kind is
    +/// Returns an error when `kind` is `disabled` (no store to open; the caller
    +/// checks the kind first), when the section matching the selected store kind
    +/// is
     /// absent, a value fails validation (an empty bucket, an S3 Express bucket
     /// name, a part size outside 5 MiB to 5 GiB, zero concurrency, a malformed
     /// prefix, an endpoint setting that breaks the rules in the `config` module
     /// doc, the SDK's own and its credential providers' and IMDS client's
     /// included), or the filesystem root is unusable.
     pub async fn open(config: &BackupStoreConfig) -> Result, StoreError> {
    +    open_described(config).await.map(|opened| opened.store)
    +}
    +
    +/// A store [`open_described`] built, with the two facts a catalog row records
    +/// about it.
    +pub struct OpenedStore {
    +    /// The store.
    +    pub store: Arc,
    +    /// Which kind it is (never [`BackupStoreKind::Disabled`]).
    +    pub kind: BackupStoreKind,
    +    /// The root label: the canonical directory of a filesystem store
    +    /// ([`FilesystemStore::store_root`]), `s3:///` with an
    +    /// optional `?endpoint=` for an S3 store ([`S3Store::store_root`]).
    +    pub root: String,
    +}
    +
    +/// [`open`], also returning the store's kind and root label.
    +///
    +/// # Errors
    +///
    +/// As [`open`].
    +pub async fn open_described(config: &BackupStoreConfig) -> Result {
         match config.kind {
             BackupStoreKind::Filesystem => {
                 let fs_config = config.filesystem.as_ref().ok_or_else(|| {
    @@ -198,7 +250,13 @@ pub async fn open(config: &BackupStoreConfig) -> Result, St
                             .to_owned(),
                     )
                 })?;
    -            Ok(Arc::new(FilesystemStore::open(&fs_config.path).await?))
    +            let store = FilesystemStore::open(&fs_config.path).await?;
    +            let root = store.store_root();
    +            Ok(OpenedStore {
    +                store: Arc::new(store),
    +                kind: BackupStoreKind::Filesystem,
    +                root,
    +            })
             }
             BackupStoreKind::S3 => {
                 let s3_config = config.s3.as_ref().ok_or_else(|| {
    @@ -206,7 +264,16 @@ pub async fn open(config: &BackupStoreConfig) -> Result, St
                         "[backup.store.s3] with a bucket is required when kind = \"s3\"".to_owned(),
                     )
                 })?;
    -            Ok(Arc::new(S3Store::open(s3_config).await?))
    +            let store = S3Store::open(s3_config).await?;
    +            let root = store.store_root().to_owned();
    +            Ok(OpenedStore {
    +                store: Arc::new(store),
    +                kind: BackupStoreKind::S3,
    +                root,
    +            })
             }
    +        BackupStoreKind::Disabled => Err(StoreError::Other(
    +            "[backup.store] kind = \"disabled\" configures no store; nothing to open".to_owned(),
    +        )),
         }
     }
    diff --git a/crates/backup-store/src/s3.rs b/crates/backup-store/src/s3.rs
    index 64c7e2a9d..da125ee52 100644
    --- a/crates/backup-store/src/s3.rs
    +++ b/crates/backup-store/src/s3.rs
    @@ -166,7 +166,10 @@ const _: () = {
     /// would otherwise hang a put for as long as the server keeps the socket.
     /// The attempt timeout includes sending the request body, so each part must
     /// upload within 60 s (see `upload_part_size_bytes` in the config module).
    -const ATTEMPT_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(60);
    +/// Exported as the crate's `S3_ATTEMPT_TIMEOUT`: a caller that bounds a
    +/// sequence of small requests (the engine's store check) stays under one
    +/// attempt, so its bound fires before the SDK starts a retry.
    +pub const ATTEMPT_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(60);
     const OPERATION_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(300);
     
     /// Longest wait for the next chunk of a `GetObject` body. The two timeouts
    @@ -232,6 +235,8 @@ pub struct S3Store {
         /// Longest wait for the next chunk of a `GetObject` body;
         /// [`BODY_IDLE_TIMEOUT`] except in the crate's own tests.
         body_idle_timeout: std::time::Duration,
    +    /// The root label catalog rows record for this store ([`Self::store_root`]).
    +    root: String,
     }
     
     impl S3Store {
    @@ -436,6 +441,14 @@ impl S3Store {
                 // this keeps a client that reaches one (the test-only `open`)
                 // from sending the request at all.
                 .disable_s3_express_session_auth(true);
    +        // The label names the endpoint the client uses, whichever source
    +        // supplied it, so two servers reaching one bucket through one
    +        // endpoint spell one label however each was told about it.
    +        let root = store_root_label(
    +            &config.bucket,
    +            &prefix,
    +            endpoint.as_ref().map(|(endpoint, _)| endpoint.as_str()),
    +        );
             builder.set_endpoint_url(endpoint.map(|(endpoint, _)| endpoint));
             // The SDK's default client sends through the proxy named in
             // `HTTP_PROXY`, `http_proxy`, `ALL_PROXY` or `all_proxy`, which
    @@ -472,9 +485,24 @@ impl S3Store {
                 max_parts: MAX_PARTS,
                 page_size: None,
                 body_idle_timeout: BODY_IDLE_TIMEOUT,
    +            root,
             })
         }
     
    +    /// The root label a catalog row records for this store:
    +    /// `s3:///`, with `?endpoint=` appended when the
    +    /// client uses an endpoint other than the SDK's default (from `endpoint`
    +    /// or from the SDK's environment and shared config), the URL in one
    +    /// spelling: lowercase scheme and host, no default port, no `/` for an
    +    /// empty path, no user information, query or fragment. Two servers
    +    /// reaching the same bucket and prefix through the same endpoint spell
    +    /// the same label however each was told about the endpoint, and so
    +    /// address the same rows.
    +    #[must_use]
    +    pub fn store_root(&self) -> &str {
    +        &self.root
    +    }
    +
         /// Bucket-side key for a store key. S3 bounds the whole object key at
         /// [`crate::key::MAX_KEY_LEN`] bytes, prefix included, so a key that fits
         /// the shared rule can still be too long under a long prefix; refuse it
    @@ -493,6 +521,36 @@ impl S3Store {
         }
     
         async fn put_impl(&self, key: &str, body: ByteStream) -> Result {
    +        // `put_inner` answers `None` only to a conditional put.
    +        self.put_inner(key, body, false)
    +            .await?
    +            .ok_or_else(|| StoreError::Other(format!("s3 PutObject of {key} returned no object")))
    +    }
    +
    +    async fn put_if_absent_impl(
    +        &self,
    +        key: &str,
    +        body: ByteStream,
    +    ) -> Result, StoreError> {
    +        self.put_inner(key, body, true).await
    +    }
    +
    +    /// The one upload path behind `put` and `put_if_absent`. With
    +    /// `if_none_match`, the request that creates the object (`PutObject` for
    +    /// a body of one part, `CompleteMultipartUpload` otherwise) carries
    +    /// `If-None-Match: *`: a 412 means an object is under the key and `None`
    +    /// is returned (the multipart upload is aborted first, since the parts
    +    /// would otherwise stay); a 409 `ConditionalRequestConflict` means a
    +    /// concurrent conditional write on the key was in flight, and the request
    +    /// is re-sent after a short back-off, up to [`CONDITIONAL_RETRIES`] times
    +    /// (the parts already uploaded are reused), after which the last answer
    +    /// is reported as the error it is.
    +    async fn put_inner(
    +        &self,
    +        key: &str,
    +        body: ByteStream,
    +        if_none_match: bool,
    +    ) -> Result, StoreError> {
             validate_key(key)?;
             let full_key = self.full_key(key)?;
             let mut chunks = std::pin::pin!(chunk_stream(body, self.part_size));
    @@ -506,21 +564,38 @@ impl S3Store {
             let Some(second) = second else {
                 // The whole body fits in one part: a single PutObject.
                 let size = first.len() as u64;
    -            let output = self
    -                .client
    -                .put_object()
    -                .bucket(&self.bucket)
    -                .key(&full_key)
    -                .body(first.into())
    -                .send()
    -                .await
    -                .map_err(|e| map_sdk_error("PutObject", e))?;
    -            return Ok(ObjectMeta {
    -                key: key.to_owned(),
    -                size,
    -                last_modified: None,
    -                etag: output.e_tag().map(strip_etag_quotes),
    -            });
    +            let mut attempt = 0;
    +            loop {
    +                let mut request = self
    +                    .client
    +                    .put_object()
    +                    .bucket(&self.bucket)
    +                    .key(&full_key)
    +                    .body(first.clone().into());
    +                if if_none_match {
    +                    request = request.if_none_match("*");
    +                }
    +                match request.send().await {
    +                    Ok(output) => {
    +                        return Ok(Some(ObjectMeta {
    +                            key: key.to_owned(),
    +                            size,
    +                            last_modified: None,
    +                            etag: output.e_tag().map(strip_etag_quotes),
    +                        }));
    +                    }
    +                    Err(e) => match conditional_outcome(&e) {
    +                        Some(Conditional::Present) if if_none_match => return Ok(None),
    +                        Some(Conditional::Conflict)
    +                            if if_none_match && attempt < CONDITIONAL_RETRIES =>
    +                        {
    +                            attempt += 1;
    +                            tokio::time::sleep(conditional_backoff(attempt)).await;
    +                        }
    +                        _ => return Err(map_sdk_error("PutObject", e)),
    +                    },
    +                }
    +            }
             };
     
             // Every failure after CreateMultipartUpload must reach the abort,
    @@ -537,38 +612,62 @@ impl S3Store {
             // is in flight.
             let mut abort_guard = self.create_upload_guarded(&full_key).await?;
             let upload_id = abort_guard.upload_id.clone();
    -        let outcome: Result = async {
    +        let outcome: Result, StoreError> = async {
                 let (parts, size) = self
                     .upload_parts(&full_key, &upload_id, first, second, chunks)
                     .await?;
    -            let output = self
    -                .client
    -                .complete_multipart_upload()
    -                .bucket(&self.bucket)
    -                .key(&full_key)
    -                .upload_id(&upload_id)
    -                .multipart_upload(
    -                    CompletedMultipartUpload::builder()
    -                        .set_parts(Some(parts))
    -                        .build(),
    -                )
    -                .send()
    -                .await
    -                .map_err(|e| map_sdk_error("CompleteMultipartUpload", e))?;
    -            Ok(ObjectMeta {
    -                key: key.to_owned(),
    -                size,
    -                last_modified: None,
    -                etag: output.e_tag().map(strip_etag_quotes),
    -            })
    +            let mut attempt = 0;
    +            loop {
    +                let mut request = self
    +                    .client
    +                    .complete_multipart_upload()
    +                    .bucket(&self.bucket)
    +                    .key(&full_key)
    +                    .upload_id(&upload_id)
    +                    .multipart_upload(
    +                        CompletedMultipartUpload::builder()
    +                            .set_parts(Some(parts.clone()))
    +                            .build(),
    +                    );
    +                if if_none_match {
    +                    request = request.if_none_match("*");
    +                }
    +                match request.send().await {
    +                    Ok(output) => {
    +                        return Ok(Some(ObjectMeta {
    +                            key: key.to_owned(),
    +                            size,
    +                            last_modified: None,
    +                            etag: output.e_tag().map(strip_etag_quotes),
    +                        }));
    +                    }
    +                    Err(e) => match conditional_outcome(&e) {
    +                        Some(Conditional::Present) if if_none_match => return Ok(None),
    +                        Some(Conditional::Conflict)
    +                            if if_none_match && attempt < CONDITIONAL_RETRIES =>
    +                        {
    +                            attempt += 1;
    +                            tokio::time::sleep(conditional_backoff(attempt)).await;
    +                        }
    +                        _ => return Err(map_sdk_error("CompleteMultipartUpload", e)),
    +                    },
    +                }
    +            }
             }
             .await;
     
             // The guard stays armed until the inline abort has returned: a put
             // dropped while that abort is in flight (a caller's timeout) still
    -        // has the guard abort the upload.
    +        // has the guard abort the upload. A conditional complete that found
    +        // the key present leaves its upload open too, so it is aborted the
    +        // same way.
             let meta = match outcome {
    -            Ok(meta) => meta,
    +            Ok(Some(meta)) => meta,
    +            Ok(None) => {
    +                abort_upload(&self.client, &self.bucket, &full_key, &upload_id).await;
    +                abort_guard.disarm();
    +                return Ok(None);
    +            }
                 Err(e) => {
                     // Best effort: the original failure is what the caller needs.
                     abort_upload(&self.client, &self.bucket, &full_key, &upload_id).await;
    @@ -577,7 +676,7 @@ impl S3Store {
                 }
             };
             abort_guard.disarm();
    -        Ok(meta)
    +        Ok(Some(meta))
         }
     
         /// Send `CreateMultipartUpload` for the bucket-side key `full_key` and
    @@ -1061,10 +1160,7 @@ impl S3Store {
                              ({}; {reason})",
                             self.bucket, PROBE_LIST_OBJECTS.grant
                         )),
    -                    other => StoreError::Other(format!(
    -                        "backup bucket {} failed validation: {other}",
    -                        self.bucket
    -                    )),
    +                    other => validation_failure(&self.bucket, other),
                     }
                 })?;
             self.validate_write_grants().await
    @@ -1080,10 +1176,7 @@ impl S3Store {
                     "the configured credentials lack {grant} on backup bucket {} ({reason}){extra}",
                     self.bucket
                 )),
    -            other => StoreError::Other(format!(
    -                "backup bucket {} failed validation: {other}",
    -                self.bucket
    -            )),
    +            other => validation_failure(&self.bucket, other),
             };
             self.client
                 .delete_object()
    @@ -1183,6 +1276,14 @@ impl BackupStore for S3Store {
             self.put_impl(key, body).boxed()
         }
     
    +    fn put_if_absent<'a>(
    +        &'a self,
    +        key: &'a str,
    +        body: ByteStream,
    +    ) -> BoxFuture<'a, Result, StoreError>> {
    +        self.put_if_absent_impl(key, body).boxed()
    +    }
    +
         fn get<'a>(&'a self, key: &'a str) -> BoxFuture<'a, Result> {
             self.get_impl(key).boxed()
         }
    @@ -1261,6 +1362,59 @@ impl BackupStore for S3Store {
     /// Normalize the configured bucket prefix: strip leading `/`, require a
     /// trailing `/` when non-empty, and hold it to the same component rules as
     /// keys.
    +/// [`S3Store::store_root`]'s rule, over the normalized prefix and the
    +/// endpoint the client uses (the configured one, or the one the SDK's
    +/// environment and shared config supplied). The endpoint is spelled one
    +/// way whatever the operator typed: scheme and host lowercased, a default
    +/// port (`80` for `http`, `443` for `https`) dropped, an empty path spelled
    +/// without its `/`, and (defence in depth for a label built before `open`'s
    +/// checks ran) user information, query and fragment removed. An endpoint
    +/// `http::Uri` accepts but `url` does not parse is spelled as typed, minus
    +/// those three parts.
    +fn store_root_label(bucket: &str, prefix: &str, endpoint: Option<&str>) -> String {
    +    let mut label = format!("s3://{bucket}/{prefix}");
    +    if let Some(endpoint) = endpoint {
    +        label.push_str("?endpoint=");
    +        label.push_str(&normalized_endpoint(endpoint.trim()));
    +    }
    +    label
    +}
    +
    +/// The one spelling of an endpoint URL the root label carries
    +/// (`store_root_label`).
    +fn normalized_endpoint(endpoint: &str) -> String {
    +    if let Ok(url) = url::Url::parse(endpoint)
    +        && let Some(host) = url.host_str()
    +    {
    +        // `Url` lowercases the scheme and the host (a domain, or an IPv6
    +        // address's hex digits), and reports no port when it is the scheme's
    +        // default.
    +        let mut spelled = format!("{}://{host}", url.scheme());
    +        if let Some(port) = url.port() {
    +            spelled.push(':');
    +            spelled.push_str(&port.to_string());
    +        }
    +        if url.path() != "/" {
    +            spelled.push_str(url.path());
    +        }
    +        return spelled;
    +    }
    +    let authority_start = endpoint.find("//").map_or(0, |at| at + 2);
    +    let rest = &endpoint[authority_start..];
    +    let authority_end = rest.find(['/', '?', '#']).unwrap_or(rest.len());
    +    let authority = &rest[..authority_end];
    +    let authority = authority
    +        .rfind('@')
    +        .map_or(authority, |at| &authority[at + 1..]);
    +    let path_end = rest.find(['?', '#']).unwrap_or(rest.len());
    +    let path = &rest[authority_end..path_end];
    +    format!(
    +        "{}{}{path}",
    +        endpoint[..authority_start].to_ascii_lowercase(),
    +        authority.to_ascii_lowercase()
    +    )
    +}
    +
     fn normalize_store_prefix(prefix: &str) -> Result {
         let trimmed = prefix.trim_start_matches('/');
         if trimmed.is_empty() {
    @@ -1955,6 +2109,46 @@ fn is_no_such_upload(err: &SdkError) -> bool {
         matches!(err, SdkError::ServiceError(ctx) if ctx.err().code() == Some("NoSuchUpload"))
     }
     
    +/// Why a request sent with `If-None-Match: *` did not store its object.
    +#[derive(Debug, Clone, Copy, PartialEq, Eq)]
    +enum Conditional {
    +    /// 412 `PreconditionFailed`: an object is under the key.
    +    Present,
    +    /// 409 `ConditionalRequestConflict`: another conditional write on the
    +    /// key was in flight; S3 asks the caller to retry.
    +    Conflict,
    +}
    +
    +/// Classify a conditional write's refusal, or `None` for any other error.
    +/// S3 answers a failed `If-None-Match` with 412 and the code
    +/// `PreconditionFailed`, and a collision of conditional writes with 409 and
    +/// `ConditionalRequestConflict`; either the status or the code identifies
    +/// the case, so a response carrying only one of them still classifies.
    +fn conditional_outcome(err: &SdkError) -> Option {
    +    let SdkError::ServiceError(ctx) = err else {
    +        return None;
    +    };
    +    match (ctx.raw().status().as_u16(), ctx.err().code()) {
    +        (412, _) | (_, Some("PreconditionFailed")) => Some(Conditional::Present),
    +        (409, Some("ConditionalRequestConflict")) | (_, Some("ConditionalRequestConflict")) => {
    +            Some(Conditional::Conflict)
    +        }
    +        _ => None,
    +    }
    +}
    +
    +/// How many times a conditional write answered `ConditionalRequestConflict`
    +/// is re-sent before the conflict is reported. Each retry waits
    +/// [`conditional_backoff`]; the eight waits sum to 7.9 s.
    +const CONDITIONAL_RETRIES: u32 = 8;
    +
    +/// The wait before retry `attempt` (1-based) of a conflicting conditional
    +/// write: 100 ms, doubling each retry (100, 200, 400, 800 ms), capped at
    +/// 1.6 s from the fifth retry on.
    +fn conditional_backoff(attempt: u32) -> std::time::Duration {
    +    std::time::Duration::from_millis(50u64.saturating_mul(1u64 << attempt.min(5)))
    +}
    +
     /// Best-effort abort of a multipart upload, after a put failure or from a
     /// drop guard; the original failure (if any) is what the caller needs, so
     /// an abort error is only logged. `NoSuchUpload` means the upload is no
    @@ -2152,6 +2346,42 @@ where
         }
     }
     
    +/// A `validate` failure that is not a denial, with the bucket named. A
    +/// transport failure keeps its class: the caller tells a store it could not
    +/// reach (which degrades the store) from a configuration it refuses (which
    +/// stops startup), so the bucket goes in as a link of the error chain rather
    +/// than flattening the error into text.
    +fn validation_failure(bucket: &str, error: StoreError) -> StoreError {
    +    match error {
    +        StoreError::Transport { source } => StoreError::Transport {
    +            source: Box::new(ValidationContext {
    +                context: format!("backup bucket {bucket} failed validation"),
    +                source,
    +            }),
    +        },
    +        other => StoreError::Other(format!("backup bucket {bucket} failed validation: {other}")),
    +    }
    +}
    +
    +/// The context link `validation_failure` adds to a transport error's chain.
    +#[derive(Debug)]
    +struct ValidationContext {
    +    context: String,
    +    source: Box,
    +}
    +
    +impl std::fmt::Display for ValidationContext {
    +    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
    +        f.write_str(&self.context)
    +    }
    +}
    +
    +impl std::error::Error for ValidationContext {
    +    fn source(&self) -> Option<&(dyn std::error::Error + 'static)> {
    +        Some(self.source.as_ref())
    +    }
    +}
    +
     /// The SDK's non-service errors display as a bare category ("dispatch
     /// failure"); the cause (no region configured, no credentials, connection
     /// refused) is in the `source()` chain. Join the chain so an operator reads
    @@ -2174,6 +2404,81 @@ pub(crate) fn describe_error_chain(error: &(dyn std::error::Error + 'static)) ->
     mod tests {
         use super::*;
     
    +    /// A transport failure through `validate` keeps its class and names the
    +    /// bucket first in its chain; any other failure is `Other` with the same
    +    /// prefix.
    +    #[test]
    +    fn a_validation_transport_failure_keeps_its_class() {
    +        let transport: SdkError =
    +            SdkError::construction_failure("no region");
    +        let wrapped = validation_failure("b", map_sdk_error("Op", transport));
    +        assert!(
    +            matches!(wrapped, StoreError::Transport { .. }),
    +            "{wrapped:?}"
    +        );
    +        let text = wrapped.to_string();
    +        assert!(
    +            text.starts_with("transport error: backup bucket b failed validation: "),
    +            "{text}"
    +        );
    +        assert!(text.contains("no region"), "{text}");
    +        let other = validation_failure("b", StoreError::Other("x".to_owned()));
    +        assert!(
    +            matches!(&other, StoreError::Other(m) if m == "backup bucket b failed validation: x"),
    +            "{other:?}"
    +        );
    +    }
    +
    +    /// The label spells the endpoint one way: the two spellings an operator
    +    /// may type for one endpoint (a trailing `/`, upper-case letters, the
    +    /// scheme's default port) give one label, and a credential, query or
    +    /// fragment never reaches it.
    +    #[test]
    +    fn the_store_root_label_names_bucket_prefix_and_endpoint() {
    +        assert_eq!(store_root_label("b", "", None), "s3://b/");
    +        assert_eq!(store_root_label("b", "extenddb/", None), "s3://b/extenddb/");
    +        assert_eq!(
    +            store_root_label("b", "p/", Some("http://127.0.0.1:9000")),
    +            "s3://b/p/?endpoint=http://127.0.0.1:9000"
    +        );
    +        assert_eq!(
    +            store_root_label("b", "p/", Some("https://u:S3cretPW@h:1/base?x=1#f")),
    +            "s3://b/p/?endpoint=https://h:1/base"
    +        );
    +        for spelling in [
    +            "http://127.0.0.1:9000",
    +            "http://127.0.0.1:9000/",
    +            "HTTP://127.0.0.1:9000/",
    +            " http://127.0.0.1:9000 ",
    +        ] {
    +            assert_eq!(
    +                store_root_label("b", "p/", Some(spelling)),
    +                "s3://b/p/?endpoint=http://127.0.0.1:9000",
    +                "{spelling:?}"
    +            );
    +        }
    +        for spelling in [
    +            "https://S3.Example.com",
    +            "https://s3.example.com/",
    +            "https://s3.example.com:443",
    +            "HTTPS://s3.example.com:443/",
    +        ] {
    +            assert_eq!(
    +                store_root_label("b", "", Some(spelling)),
    +                "s3://b/?endpoint=https://s3.example.com",
    +                "{spelling:?}"
    +            );
    +        }
    +        assert_eq!(
    +            store_root_label("b", "", Some("http://LocalHost:80/base/")),
    +            "s3://b/?endpoint=http://localhost/base/"
    +        );
    +        assert_eq!(
    +            store_root_label("b", "", Some("http://[::1]:9000")),
    +            "s3://b/?endpoint=http://[::1]:9000"
    +        );
    +    }
    +
         #[test]
         fn store_prefix_normalization() {
             assert_eq!(normalize_store_prefix("").unwrap(), "");
    @@ -2246,6 +2551,47 @@ mod tests {
             )
         }
     
    +    /// The two answers a conditional write classifies, from synthetic
    +    /// service errors, by status alone, by code alone, and by both; every
    +    /// other answer is `None` and takes `map_sdk_error`'s path.
    +    #[test]
    +    fn conditional_outcomes_from_a_reply_document() {
    +        for (status, code, expected) in [
    +            (412, Some("PreconditionFailed"), Some(Conditional::Present)),
    +            (412, None, Some(Conditional::Present)),
    +            (400, Some("PreconditionFailed"), Some(Conditional::Present)),
    +            (
    +                409,
    +                Some("ConditionalRequestConflict"),
    +                Some(Conditional::Conflict),
    +            ),
    +            (
    +                503,
    +                Some("ConditionalRequestConflict"),
    +                Some(Conditional::Conflict),
    +            ),
    +            // A bare 409 is some other conflict (a bucket operation's), not
    +            // a conditional one.
    +            (409, None, None),
    +            (409, Some("BucketAlreadyOwnedByYou"), None),
    +            (403, Some("AccessDenied"), None),
    +            (404, Some("NoSuchKey"), None),
    +            (500, Some("InternalError"), None),
    +        ] {
    +            assert_eq!(
    +                conditional_outcome(&service_error(status, code)),
    +                expected,
    +                "{status} {code:?}"
    +            );
    +        }
    +        let transport: SdkError =
    +            SdkError::construction_failure("no region");
    +        assert_eq!(conditional_outcome(&transport), None);
    +        assert_eq!(conditional_backoff(1).as_millis(), 100);
    +        assert_eq!(conditional_backoff(5).as_millis(), 1600);
    +        assert_eq!(conditional_backoff(9).as_millis(), 1600, "capped");
    +    }
    +
         /// Every cell of `map_sdk_error`'s table, from synthetic service errors,
         /// so the mapping is pinned without an endpoint.
         #[test]
    diff --git a/crates/backup-store/tests/conformance.rs b/crates/backup-store/tests/conformance.rs
    index 1511b6164..30adcdd53 100644
    --- a/crates/backup-store/tests/conformance.rs
    +++ b/crates/backup-store/tests/conformance.rs
    @@ -441,6 +441,146 @@ mod suite {
             );
             assert!(list_keys(store, "aborted").await.is_empty());
         }
    +
    +    /// `put_if_absent` on an empty key stores the body and reports it; on a
    +    /// key that holds an object (its own, or one `put` wrote) it reports
    +    /// `None` and leaves the first body in place, whatever body the second
    +    /// call carried, and the key's metadata is the first object's. The cell
    +    /// over one part runs the multipart path on S3 (a conditional
    +    /// `CompleteMultipartUpload`) and a plain link on the filesystem store;
    +    /// its second call carries a body over one part too, so on S3 a
    +    /// conditional complete meets the present key and has an upload to
    +    /// abort, and its third call a body of one part.
    +    pub async fn put_if_absent_creates_once(store: &dyn BackupStore) {
    +        let small = b"first body".to_vec();
    +        let large = patterned_bytes(usize::try_from(PART_SIZE).unwrap() * 2 + 4096);
    +        let mut other_large = patterned_bytes(usize::try_from(PART_SIZE).unwrap() * 2 + 2048);
    +        other_large.reverse();
    +        for (key, first, seconds) in [
    +            (
    +                "ident/small.json",
    +                small.clone(),
    +                vec![b"second body, longer".to_vec()],
    +            ),
    +            (
    +                "ident/large.bin",
    +                large.clone(),
    +                vec![other_large, small.clone()],
    +            ),
    +        ] {
    +            let stored = store
    +                .put_if_absent(key, chunked_body(first.clone()))
    +                .await
    +                .expect("first put_if_absent succeeds")
    +                .expect("an empty key is created");
    +            assert_eq!(stored.key, key);
    +            assert_eq!(stored.size, first.len() as u64);
    +            for second in seconds {
    +                let again = store
    +                    .put_if_absent(key, chunked_body(second.clone()))
    +                    .await
    +                    .expect("a later put_if_absent succeeds");
    +                assert!(again.is_none(), "{key}: a present key reports None");
    +                let read = read_all(store.get(key).await.unwrap()).await;
    +                assert_eq!(read, first, "{key}: the first body stays");
    +            }
    +            let meta = store.head(key).await.unwrap().expect("object exists");
    +            assert_eq!(meta.size, first.len() as u64);
    +            assert_eq!(list_keys(store, key).await, vec![key.to_owned()]);
    +        }
    +        // A key `put` wrote is present to `put_if_absent` too.
    +        store
    +            .put("ident/put-first", chunked_body(b"by put"))
    +            .await
    +            .unwrap();
    +        assert!(
    +            store
    +                .put_if_absent("ident/put-first", chunked_body(b"by put_if_absent"))
    +                .await
    +                .unwrap()
    +                .is_none()
    +        );
    +        assert_eq!(
    +            read_all(store.get("ident/put-first").await.unwrap()).await,
    +            b"by put"
    +        );
    +        // And a key `put_if_absent` wrote is replaced by `put`, as any object
    +        // is.
    +        store
    +            .put("ident/small.json", chunked_body(b"replaced"))
    +            .await
    +            .unwrap();
    +        assert_eq!(
    +            read_all(store.get("ident/small.json").await.unwrap()).await,
    +            b"replaced"
    +        );
    +        // A body that fails leaves nothing under the key and the key stays
    +        // free for the next call.
    +        let err = store
    +            .put_if_absent("ident/failing", failing_body(b"partial"))
    +            .await
    +            .expect_err("mid-body failure surfaces");
    +        assert!(matches!(&err, StoreError::Other(msg) if msg.contains("injected")));
    +        assert!(store.head("ident/failing").await.unwrap().is_none());
    +        assert!(
    +            store
    +                .put_if_absent("ident/failing", chunked_body(b"ok"))
    +                .await
    +                .unwrap()
    +                .is_some()
    +        );
    +    }
    +
    +    /// Concurrent `put_if_absent` calls on one key: exactly one stores its
    +    /// body, every other reads `None`, and the object under the key is the
    +    /// winner's body. The bodies are distinct so the winner is identifiable;
    +    /// the race is driven by starting every call before any is polled to
    +    /// completion (`join_all`), and repeated so a store whose calls happen to
    +    /// serialise once is still exercised across rounds.
    +    pub async fn concurrent_put_if_absent_leaves_the_first_body(store: &dyn BackupStore) {
    +        const CALLERS: usize = 8;
    +        const ROUNDS: usize = 3;
    +        for round in 0..ROUNDS {
    +            let key = format!("race/round-{round}/extenddb-store.json");
    +            let bodies: Vec> = (0..CALLERS)
    +                .map(|i| format!("{{\"round\":{round},\"caller\":{i}}}").into_bytes())
    +                .collect();
    +            let outcomes = futures::future::join_all(
    +                bodies
    +                    .iter()
    +                    .map(|body| store.put_if_absent(&key, chunked_body(body.clone()))),
    +            )
    +            .await;
    +            let winners: Vec = outcomes
    +                .iter()
    +                .enumerate()
    +                .filter_map(|(i, outcome)| {
    +                    outcome
    +                        .as_ref()
    +                        .expect("every call succeeds")
    +                        .as_ref()
    +                        .map(|_| i)
    +                })
    +                .collect();
    +            assert_eq!(
    +                winners.len(),
    +                1,
    +                "round {round}: exactly one caller stores its body, got {winners:?}"
    +            );
    +            let read = read_all(store.get(&key).await.unwrap()).await;
    +            assert_eq!(
    +                read, bodies[winners[0]],
    +                "round {round}: the object is the winner's body"
    +            );
    +            let meta = store.head(&key).await.unwrap().expect("object exists");
    +            assert_eq!(meta.size, bodies[winners[0]].len() as u64);
    +            assert_eq!(
    +                list_keys(store, &format!("race/round-{round}")).await,
    +                vec![key.clone()],
    +                "round {round}: one object under the prefix"
    +            );
    +        }
    +    }
     }
     
     #[cfg(unix)]
    @@ -557,6 +697,65 @@ mod filesystem {
             }
         }
     
    +    #[tokio::test]
    +    async fn put_if_absent_creates_once() {
    +        let (dir, store) = fs_store().await;
    +        suite::put_if_absent_creates_once(&store).await;
    +        // The link discipline: no temporary stays, whether the call stored
    +        // its object, found the key present, or failed mid-body, and the
    +        // object the winning call linked is private.
    +        assert!(temporaries_under(dir.path()).is_empty());
    +        assert_eq!(
    +            mode_of(&dir.path().join("ident/large.bin")) & 0o777,
    +            0o600,
    +            "a linked object keeps the temporary's mode"
    +        );
    +    }
    +
    +    #[tokio::test(flavor = "multi_thread", worker_threads = 4)]
    +    async fn concurrent_put_if_absent_leaves_the_first_body() {
    +        let (dir, store) = fs_store().await;
    +        suite::concurrent_put_if_absent_leaves_the_first_body(&store).await;
    +        assert!(temporaries_under(dir.path()).is_empty());
    +    }
    +
    +    /// `put_if_absent` applies the destination rules `put` does: a symlink at
    +    /// the key is refused without being followed and its target untouched, a
    +    /// directory at the key is a key-space collision, and an object at an
    +    /// ancestor of the key makes the key unusable. None of the three
    +    /// consumes the body or leaves a temporary.
    +    #[tokio::test]
    +    async fn put_if_absent_refuses_what_put_refuses() {
    +        let (dir, store) = fs_store().await;
    +        let outside = private_tempdir();
    +        let target = outside.path().join("target");
    +        std::fs::write(&target, b"outside").unwrap();
    +        private_dir(&dir.path().join("a"));
    +        std::os::unix::fs::symlink(&target, dir.path().join("a/link")).unwrap();
    +        let err = store
    +            .put_if_absent("a/link", chunked_body(b"x"))
    +            .await
    +            .expect_err("a symlink at the key is refused");
    +        assert!(matches!(err, StoreError::PermissionDenied(_)), "{err}");
    +        assert_eq!(std::fs::read(&target).unwrap(), b"outside");
    +
    +        store
    +            .put("dir/under/obj", chunked_body(b"x"))
    +            .await
    +            .unwrap();
    +        let err = store
    +            .put_if_absent("dir/under", chunked_body(b"x"))
    +            .await
    +            .expect_err("a directory at the key is refused");
    +        assert!(matches!(err, StoreError::Io { .. }), "{err}");
    +        let err = store
    +            .put_if_absent("dir/under/obj/below", chunked_body(b"x"))
    +            .await
    +            .expect_err("an object at an ancestor is refused");
    +        assert!(matches!(err, StoreError::Io { .. }), "{err}");
    +        assert!(temporaries_under(dir.path()).is_empty());
    +    }
    +
         /// Every `.put-*` entry under `root`, at any depth.
         fn temporaries_under(root: &std::path::Path) -> Vec {
             let mut found = Vec::new();
    @@ -1547,6 +1746,14 @@ mod filesystem {
             assert!(FilesystemStore::open(&missing).await.is_err());
         }
     
    +    /// An empty root path is refused by name, not as a stat of "" (whose
    +    /// message would show an empty name between two spaces).
    +    #[tokio::test]
    +    async fn open_refuses_an_empty_root_path() {
    +        let err = FilesystemStore::open("").await.err().expect("refused");
    +        assert!(err.to_string().contains("root path is empty"), "{err}");
    +    }
    +
         #[tokio::test]
         async fn symlinked_directory_escape_is_refused() {
             let outside = tempfile::tempdir().expect("outside dir");
    @@ -2553,6 +2760,41 @@ mod s3 {
             suite::list_order_is_key_bytes(&store).await;
         }
     
    +    /// Both cells of the conditional write: a single `PutObject` and a
    +    /// `CompleteMultipartUpload`, each with `If-None-Match: *`. A present key
    +    /// answers 412 to either; the multipart cell's upload is aborted on that
    +    /// answer, which the listing of open uploads below checks.
    +    #[tokio::test]
    +    async fn put_if_absent_creates_once() {
    +        let Some(endpoint) = endpoint_or_skip() else {
    +            return;
    +        };
    +        let prefix = unique_prefix();
    +        let config = test_config(endpoint, test_bucket(), prefix.clone());
    +        let store = S3Store::open(&config).await.expect("S3Store::open");
    +        ensure_bucket(&config).await;
    +        suite::put_if_absent_creates_once(&store).await;
    +        let open = raw_client(&config)
    +            .await
    +            .list_multipart_uploads()
    +            .bucket(&config.bucket)
    +            .prefix(&prefix)
    +            .send()
    +            .await
    +            .expect("list_multipart_uploads");
    +        assert!(
    +            open.uploads().is_empty(),
    +            "a conditional complete that found the key present aborts its upload: {:?}",
    +            open.uploads()
    +        );
    +    }
    +
    +    #[tokio::test(flavor = "multi_thread", worker_threads = 4)]
    +    async fn concurrent_put_if_absent_leaves_the_first_body() {
    +        let Some(store) = store().await else { return };
    +        suite::concurrent_put_if_absent_leaves_the_first_body(&store).await;
    +    }
    +
         #[tokio::test]
         async fn delete_prefix_removes_and_counts() {
             let Some(store) = store().await else { return };
    @@ -2859,6 +3101,11 @@ mod s3 {
                 .validate()
                 .await
                 .expect_err("nothing listens on port 9");
    +        assert!(
    +            matches!(err, StoreError::Transport { .. }),
    +            "a transport failure keeps its class through validate, so a server can tell it \
    +             from a configuration error: {err:?}"
    +        );
             let text = err.to_string();
             assert!(text.contains("unused"), "names the bucket: {text}");
             assert!(
    @@ -2867,6 +3114,284 @@ mod s3 {
             );
         }
     
    +    /// The 409 `ConditionalRequestConflict` loop of `put_if_absent`, driven
    +    /// by a server of the test's own that answers from a script (moto never
    +    /// answers 409): a `PutObject` answered 409 twice is re-sent and the
    +    /// third answer stands; a `CompleteMultipartUpload` answered 409 and
    +    /// then 412 is "present" and its upload is aborted; a `PutObject`
    +    /// answered 409 nine times is the error the ninth answer is, after
    +    /// eight retries. Runs in a child with the suite's own credentials, as
    +    /// every request needs them. No endpoint is needed.
    +    #[tokio::test]
    +    async fn a_conditional_request_conflict_is_retried_from_a_reply_script() {
    +        const NAME: &str = "s3::a_conditional_request_conflict_is_retried_from_a_reply_script";
    +        if !is_child(NAME) {
    +            run_with_test_credentials(NAME);
    +            return;
    +        }
    +        let server = ScriptedS3::start().await;
    +        let config = test_config(server.endpoint(), "scripted".to_owned(), String::new());
    +        let store = S3Store::open(&config).await.expect("S3Store::open");
    +
    +        // Two conflicts, then created.
    +        server.conflicts_before_success(2);
    +        let meta = store
    +            .put_if_absent("k/one-part", chunked_body(b"body"))
    +            .await
    +            .expect("the third answer is 200")
    +            .expect("created");
    +        assert_eq!(meta.size, 4);
    +        let puts = server.requests("PUT /scripted/k/one-part?x-id=PutObject");
    +        assert_eq!(puts, 3, "one send and two re-sends");
    +
    +        // The multipart cell: the complete answers 409 once, then 412.
    +        server.conflicts_then_present(1);
    +        let large = patterned_bytes(usize::try_from(PART_SIZE).unwrap() + 16);
    +        let again = store
    +            .put_if_absent("k/two-parts", chunked_body(large))
    +            .await
    +            .expect("412 is an answer, not an error");
    +        assert!(again.is_none(), "a present key reports None");
    +        assert_eq!(
    +            server.requests("POST /scripted/k/two-parts?uploadId="),
    +            2,
    +            "the complete was re-sent once"
    +        );
    +        assert_eq!(
    +            server.requests("DELETE /scripted/k/two-parts?x-id=AbortMultipartUpload"),
    +            1,
    +            "the upload the present key refused was aborted"
    +        );
    +
    +        // Conflicts that never end: eight retries, then the error.
    +        server.conflicts_before_success(u32::MAX);
    +        let started = std::time::Instant::now();
    +        let err = store
    +            .put_if_absent("k/never", chunked_body(b"body"))
    +            .await
    +            .expect_err("the ninth 409 is reported");
    +        let text = err.to_string();
    +        assert!(
    +            text.contains("ConditionalRequestConflict"),
    +            "names the conflict: {text}"
    +        );
    +        assert_eq!(server.requests("PUT /scripted/k/never"), 9);
    +        let waited = started.elapsed();
    +        assert!(
    +            waited >= std::time::Duration::from_millis(7_000)
    +                && waited < std::time::Duration::from_secs(20),
    +            "the back-offs sum to about 7.9 s, measured {waited:?}"
    +        );
    +    }
    +
    +    /// An HTTP/1.1 server that plays S3 from a script: every request's
    +    /// method, path and query are recorded; a `PutObject` or
    +    /// `CompleteMultipartUpload` is answered 409 `ConditionalRequestConflict`
    +    /// a scripted number of times, then 200, or 412 `PreconditionFailed`;
    +    /// `CreateMultipartUpload`, `UploadPart` and `AbortMultipartUpload` are
    +    /// answered as S3 answers them.
    +    struct ScriptedS3 {
    +        port: u16,
    +        state: std::sync::Arc>,
    +    }
    +
    +    #[derive(Default)]
    +    struct ScriptState {
    +        /// Conflicts left to answer before the terminal answer.
    +        conflicts: u32,
    +        /// Whether the terminal answer is 412 rather than 200.
    +        then_present: bool,
    +        /// `" "` of every request, in order.
    +        requests: Vec,
    +    }
    +
    +    impl ScriptedS3 {
    +        async fn start() -> Self {
    +            let listener = tokio::net::TcpListener::bind("127.0.0.1:0")
    +                .await
    +                .expect("bind");
    +            let port = listener.local_addr().unwrap().port();
    +            let state = std::sync::Arc::new(std::sync::Mutex::new(ScriptState::default()));
    +            let shared = std::sync::Arc::clone(&state);
    +            tokio::spawn(async move {
    +                loop {
    +                    let Ok((socket, _)) = listener.accept().await else {
    +                        return;
    +                    };
    +                    tokio::spawn(Self::serve(socket, std::sync::Arc::clone(&shared)));
    +                }
    +            });
    +            Self { port, state }
    +        }
    +
    +        fn endpoint(&self) -> String {
    +            format!("http://127.0.0.1:{}", self.port)
    +        }
    +
    +        fn conflicts_before_success(&self, n: u32) {
    +            let mut state = self.state.lock().unwrap();
    +            state.conflicts = n;
    +            state.then_present = false;
    +        }
    +
    +        fn conflicts_then_present(&self, n: u32) {
    +            let mut state = self.state.lock().unwrap();
    +            state.conflicts = n;
    +            state.then_present = true;
    +        }
    +
    +        /// How many recorded requests start with `prefix`.
    +        fn requests(&self, prefix: &str) -> usize {
    +            self.state
    +                .lock()
    +                .unwrap()
    +                .requests
    +                .iter()
    +                .filter(|r| r.starts_with(prefix))
    +                .count()
    +        }
    +
    +        async fn serve(
    +            mut socket: tokio::net::TcpStream,
    +            state: std::sync::Arc>,
    +        ) {
    +            use tokio::io::{AsyncReadExt, AsyncWriteExt};
    +            let mut buffer = Vec::new();
    +            loop {
    +                // Read one request: the head, then the body it announces.
    +                let head_end = loop {
    +                    if let Some(at) = buffer.windows(4).position(|w| w == b"\r\n\r\n") {
    +                        break at + 4;
    +                    }
    +                    let mut chunk = [0u8; 8192];
    +                    match socket.read(&mut chunk).await {
    +                        Ok(0) | Err(_) => return,
    +                        Ok(n) => buffer.extend_from_slice(&chunk[..n]),
    +                    }
    +                };
    +                let head = String::from_utf8_lossy(&buffer[..head_end]).into_owned();
    +                let mut lines = head.split("\r\n");
    +                let request_line = lines.next().unwrap_or_default().to_owned();
    +                let mut content_length = 0usize;
    +                let mut chunked = false;
    +                for line in lines {
    +                    let Some((name, value)) = line.split_once(':') else {
    +                        continue;
    +                    };
    +                    let value = value.trim();
    +                    if name.eq_ignore_ascii_case("content-length") {
    +                        content_length = value.parse().unwrap_or(0);
    +                    } else if name.eq_ignore_ascii_case("transfer-encoding")
    +                        && value.eq_ignore_ascii_case("chunked")
    +                    {
    +                        chunked = true;
    +                    }
    +                }
    +                buffer.drain(..head_end);
    +                if chunked {
    +                    // Read until the terminating chunk.
    +                    loop {
    +                        if buffer.windows(5).any(|w| w == b"0\r\n\r\n") {
    +                            let at = buffer.windows(5).position(|w| w == b"0\r\n\r\n").unwrap();
    +                            buffer.drain(..at + 5);
    +                            break;
    +                        }
    +                        let mut chunk = [0u8; 65536];
    +                        match socket.read(&mut chunk).await {
    +                            Ok(0) | Err(_) => return,
    +                            Ok(n) => buffer.extend_from_slice(&chunk[..n]),
    +                        }
    +                    }
    +                } else {
    +                    while buffer.len() < content_length {
    +                        let mut chunk = [0u8; 65536];
    +                        match socket.read(&mut chunk).await {
    +                            Ok(0) | Err(_) => return,
    +                            Ok(n) => buffer.extend_from_slice(&chunk[..n]),
    +                        }
    +                    }
    +                    buffer.drain(..content_length);
    +                }
    +
    +                let mut parts = request_line.split(' ');
    +                let method = parts.next().unwrap_or_default().to_owned();
    +                let target = parts.next().unwrap_or_default().to_owned();
    +                let (status, body) = {
    +                    let mut state = state.lock().unwrap();
    +                    state.requests.push(format!("{method} {target}"));
    +                    Self::answer(&mut state, &method, &target)
    +                };
    +                let reason = match status {
    +                    200 => "OK",
    +                    204 => "No Content",
    +                    409 => "Conflict",
    +                    412 => "Precondition Failed",
    +                    _ => "Error",
    +                };
    +                let response = format!(
    +                    "HTTP/1.1 {status} {reason}\r\nContent-Type: application/xml\r\n\
    +                     Content-Length: {}\r\nETag: \"etag-1\"\r\nConnection: keep-alive\r\n\r\n{body}",
    +                    body.len()
    +                );
    +                if socket.write_all(response.as_bytes()).await.is_err() {
    +                    return;
    +                }
    +            }
    +        }
    +
    +        /// The scripted answer to one request.
    +        fn answer(state: &mut ScriptState, method: &str, target: &str) -> (u16, String) {
    +            let conflict = || {
    +                (
    +                    409,
    +                    "ConditionalRequestConflictConditional request \
    +                     cannot succeed due to a conflicting operation against this \
    +                     resource."
    +                        .to_owned(),
    +                )
    +            };
    +            let present = || {
    +                (
    +                    412,
    +                    "PreconditionFailedAt least one of the \
    +                     pre-conditions you specified did not hold"
    +                        .to_owned(),
    +                )
    +            };
    +            let conditional = |state: &mut ScriptState, success: String| {
    +                if state.conflicts > 0 {
    +                    state.conflicts = state.conflicts.saturating_sub(1);
    +                    conflict()
    +                } else if state.then_present {
    +                    present()
    +                } else {
    +                    (200, success)
    +                }
    +            };
    +            match method {
    +                "PUT" if target.contains("partNumber=") => (200, String::new()),
    +                "PUT" => conditional(state, String::new()),
    +                "POST" if target.ends_with("?uploads") => (
    +                    200,
    +                    "scriptedk\
    +                     upload-1"
    +                        .to_owned(),
    +                ),
    +                "POST" if target.contains("uploadId=") => conditional(
    +                    state,
    +                    "scriptedk\
    +                     \"etag-1\""
    +                        .to_owned(),
    +                ),
    +                "DELETE" => (204, String::new()),
    +                _ => (
    +                    404,
    +                    "NoSuchKeyno".to_owned(),
    +                ),
    +            }
    +        }
    +    }
    +
         /// A put whose future is dropped mid-upload (a cancelled job) aborts its
         /// multipart upload from the drop guard, and `delete_prefix` aborts any
         /// upload a crashed process left, so no parts accrue storage.
    @@ -5172,6 +5697,16 @@ mod s3 {
             };
             println!("request URI: {uri}");
             assert!(uri.starts_with(expected), "{uri} does not go to {expected}");
    +        // The root label names the endpoint the request went to, in its one
    +        // spelling, whichever source supplied it.
    +        let endpoint = expected.trim_end_matches('/');
    +        let label = store.store_root();
    +        println!("store root: {label}");
    +        assert_eq!(
    +            label,
    +            format!("s3://env-bucket/?endpoint={endpoint}"),
    +            "the label names the effective endpoint"
    +        );
         }
     
         /// Rewrites every request's URI, after the endpoint is resolved and
    diff --git a/crates/config/Cargo.toml b/crates/config/Cargo.toml
    index 1c2c59ebf..f9f086124 100644
    --- a/crates/config/Cargo.toml
    +++ b/crates/config/Cargo.toml
    @@ -15,8 +15,12 @@ dev-mode = []
     [dependencies]
     extenddb-core = { workspace = true }
     extenddb-storage = { workspace = true }
    +extenddb-backup-store = { workspace = true }
     serde = { workspace = true }
     toml = { workspace = true }
     config = { workspace = true }
     anyhow = { workspace = true }
     tracing = { workspace = true }
    +
    +[dev-dependencies]
    +tempfile = "3"
    diff --git a/crates/config/src/backup.rs b/crates/config/src/backup.rs
    index 92339fa4f..83e3a9c83 100644
    --- a/crates/config/src/backup.rs
    +++ b/crates/config/src/backup.rs
    @@ -1,7 +1,8 @@
     // Copyright 2026 ExtendDB contributors
     // SPDX-License-Identifier: Apache-2.0
     
    -//! The `[backup]` section: the knobs backup and restore jobs read.
    +//! The `[backup]` section: the knobs backup and restore jobs read, and the
    +//! store they write to.
     //!
     //! ```toml
     //! [backup]
    @@ -12,6 +13,11 @@
     //! reconcile_interval_secs = 60       # reconciler cycle
     //! max_concurrent_jobs = 4            # backup and restore jobs running at once on this instance
     //! # spool_path = "/var/lib/extenddb/spool"  # restore download area for an S3 store
    +//!
    +//! [backup.store]
    +//! kind = "filesystem"                # "filesystem" | "s3" | "disabled"
    +//! [backup.store.filesystem]
    +//! path = "/var/lib/extenddb/backups"
     //! ```
     //!
     //! `stale_job_after_secs` must be at least three heartbeat periods: a job
    @@ -33,9 +39,56 @@
     //! at load, and the PostgreSQL pool rule makes anything over a few hundred
     //! meaningless anyway.
     //!
    -//! The store itself (`[backup.store]`) is configured separately and is not
    -//! part of this struct.
    +//! The store. `[backup.store]` is the backup store crate's
    +//! [`BackupStoreConfig`], nested under its own key so the one definition of
    +//! the shape stays in that crate and `deny_unknown_fields` holds on both
    +//! tables (`serde(flatten)` would drop it). Its keys, defaults and `open`'s
    +//! refusals are that crate's `config` module doc. What this module adds is
    +//! the per-backend rule for an absent section and the spool
    +//! ([`BackupConfig::resolve_store`]):
    +//!
    +//! - On a backend whose database is one file on this host (file-backed
    +//!   SQLite), a missing `[backup.store]` is `kind = "filesystem"` with
    +//!   `path = "backups/"` beside the database file. That defaulted root is
    +//!   created `0700` if missing, and a failure of its validation is a logged
    +//!   warning that leaves the store unconfigured, not a startup error, so an
    +//!   upgrade never stops a server that never configured backups; an
    +//!   explicitly configured root that fails validation is a startup error.
    +//! - On PostgreSQL, MongoDB, Cassandra and an in-memory SQLite database
    +//!   (no file, so no default root), a missing section leaves the store
    +//!   unconfigured and startup logs that.
    +//! - `kind = "disabled"` leaves the store unconfigured on purpose, on any
    +//!   backend.
    +//! - Without a configured store, CreateBackup writes the in-catalog format
    +//!   and logs a deprecation warning naming `[backup.store]` on each call;
    +//!   backups of either format stay readable, listable, restorable and
    +//!   deletable.
    +//!
    +//! The spool. Restores from an S3 store download to `spool_path`, created
    +//! `0700` with files `0600`. On file-backed SQLite it defaults to
    +//! `backup-spool/` beside the database file; on every other backend an S3
    +//! store needs `spool_path` and startup refuses without it (a filesystem
    +//! store needs no spool: files are verified in place). A `spool_path` that
    +//! contains or is contained in a filesystem store root is refused (the store
    +//! would list and delete it, and a sync would read its first component as an
    +//! account id), and so is one that is or contains the database file's
    +//! directory. A filesystem store root that is or contains the database
    +//! file's directory is refused too: the database and its sidecars would sit
    +//! among the backups. An empty root is compared with nothing (it names no
    +//! directory) and left for `open` to refuse by name. The spool's `.lock`
    +//! another live instance holds is refused at startup by the server, which
    +//! owns the lock file.
    +//!
    +//! A relative `[backup.store.filesystem] path` or `spool_path` is made
    +//! absolute against the working directory at config load
    +//! ([`BackupConfig::make_paths_absolute`]), where `storage.path` is: `serve`
    +//! daemonizes after loading, and the child's working directory is `/`.
     
    +use std::path::{Path, PathBuf};
    +
    +pub use extenddb_backup_store::{
    +    BackupStoreConfig, BackupStoreKind, FilesystemStoreConfig, S3StoreConfig,
    +};
     use extenddb_storage::backup_catalog::JOB_HEARTBEAT_INTERVAL;
     use serde::Deserialize;
     
    @@ -50,6 +103,13 @@ pub const MIN_STALE_HEARTBEATS: u64 = 3;
     /// The largest `max_concurrent_jobs`.
     pub const MAX_CONCURRENT_JOBS: usize = 1024;
     
    +/// The directory name of the defaulted filesystem store root beside a
    +/// database file.
    +pub const DEFAULT_ROOT_DIR_NAME: &str = "backups";
    +
    +/// The directory name of the defaulted restore spool beside a database file.
    +pub const DEFAULT_SPOOL_DIR_NAME: &str = "backup-spool";
    +
     #[derive(Debug, Clone, Deserialize, PartialEq, Eq)]
     #[serde(deny_unknown_fields)]
     pub struct BackupConfig {
    @@ -76,10 +136,14 @@ pub struct BackupConfig {
         #[serde(default = "default_max_concurrent_jobs")]
         pub max_concurrent_jobs: usize,
         /// Restore download area for an S3 store. Defaults beside the database
    -    /// file on SQLite; required on the other backends once a store is
    -    /// configured.
    +    /// file on SQLite; required on the other backends with an S3 store.
         #[serde(default)]
         pub spool_path: Option,
    +    /// `[backup.store]`: the store this instance writes backups to. `None`
    +    /// when the section is absent, which [`BackupConfig::resolve_store`]
    +    /// turns into the per-backend default.
    +    #[serde(default)]
    +    pub store: Option,
     }
     
     impl Default for BackupConfig {
    @@ -92,10 +156,45 @@ impl Default for BackupConfig {
                 reconcile_interval_secs: default_reconcile_interval_secs(),
                 max_concurrent_jobs: default_max_concurrent_jobs(),
                 spool_path: None,
    +            store: None,
             }
         }
     }
     
    +/// Why no store is configured.
    +#[derive(Debug, Clone, Copy, PartialEq, Eq)]
    +pub enum Unconfigured {
    +    /// `[backup.store]` is absent and this backend has no default root (a
    +    /// server database, or an in-memory SQLite database).
    +    Absent,
    +    /// `kind = "disabled"`.
    +    Disabled,
    +}
    +
    +/// The store `[backup.store]` and the backend together select.
    +#[derive(Debug, Clone, PartialEq, Eq)]
    +pub enum EffectiveStore {
    +    /// No store: CreateBackup writes the in-catalog format.
    +    Unconfigured(Unconfigured),
    +    /// A store to open and validate at startup.
    +    Configured {
    +        config: BackupStoreConfig,
    +        /// The section was absent and the root is the default beside the
    +        /// database file: the server creates it if missing and treats a
    +        /// validation failure as a warning that leaves the store
    +        /// unconfigured.
    +        defaulted: bool,
    +    },
    +}
    +
    +/// What [`BackupConfig::resolve_store`] decided.
    +#[derive(Debug, Clone, PartialEq, Eq)]
    +pub struct ResolvedStore {
    +    pub store: EffectiveStore,
    +    /// The restore spool, when the configured store is S3.
    +    pub spool_path: Option,
    +}
    +
     impl BackupConfig {
         /// The period the jobs heartbeat at: the larger of
         /// `heartbeat_interval_secs` and [`JOB_HEARTBEAT_INTERVAL`], which is
    @@ -106,6 +205,36 @@ impl BackupConfig {
                 .max(JOB_HEARTBEAT_INTERVAL.as_secs())
         }
     
    +    /// Make `[backup.store.filesystem] path` and `spool_path` absolute
    +    /// against `cwd` (`~` expanded first), so a relative value written in
    +    /// the config file names the directory it does from the shell that
    +    /// started the server, in the daemonized child too. An absolute value
    +    /// is left as it is, and so is an empty one, which `open` refuses as it
    +    /// did (it would otherwise name the working directory as the root).
    +    pub fn make_paths_absolute(&mut self, cwd: &Path) {
    +        if let Some(fs) = self.store.as_mut().and_then(|s| s.filesystem.as_mut())
    +            && !fs.path.is_absolute()
    +            && !fs.path.as_os_str().is_empty()
    +        {
    +            let expanded = PathBuf::from(crate::expand_tilde(&fs.path.to_string_lossy()));
    +            fs.path = if expanded.is_absolute() {
    +                expanded
    +            } else {
    +                cwd.join(expanded)
    +            };
    +        }
    +        if let Some(spool) = self.spool_path.as_mut() {
    +            let expanded = PathBuf::from(crate::expand_tilde(spool));
    +            *spool = if expanded.is_absolute() {
    +                expanded
    +            } else {
    +                cwd.join(expanded)
    +            }
    +            .to_string_lossy()
    +            .into_owned();
    +        }
    +    }
    +
         /// The cross-field rules, checked at load.
         ///
         /// # Errors
    @@ -148,8 +277,197 @@ impl BackupConfig {
                     self.max_concurrent_jobs
                 ));
             }
    +        if let Some(spool) = &self.spool_path
    +            && spool.trim().is_empty()
    +        {
    +            return Err("backup.spool_path must not be empty".to_owned());
    +        }
             Ok(())
         }
    +
    +    /// The store and spool this configuration selects on a backend whose
    +    /// database is `database_file` (`Some` for file-backed SQLite, `None`
    +    /// for a server database or an in-memory one), by the rules in the
    +    /// module doc.
    +    ///
    +    /// # Errors
    +    ///
    +    /// A message naming the key: an S3 store without `spool_path` on a
    +    /// backend with no default spool; a `spool_path` inside or containing a
    +    /// filesystem store root, or inside or containing the database file's
    +    /// directory; a filesystem root that is or contains the database file's
    +    /// directory.
    +    pub fn resolve_store(&self, database_file: Option<&Path>) -> Result {
    +        let database_dir = database_file.and_then(Path::parent).map(Path::to_path_buf);
    +        let store = match &self.store {
    +            None => match &database_dir {
    +                Some(dir) => EffectiveStore::Configured {
    +                    config: BackupStoreConfig {
    +                        kind: BackupStoreKind::Filesystem,
    +                        filesystem: Some(FilesystemStoreConfig {
    +                            path: dir.join(DEFAULT_ROOT_DIR_NAME),
    +                        }),
    +                        s3: None,
    +                    },
    +                    defaulted: true,
    +                },
    +                None => EffectiveStore::Unconfigured(Unconfigured::Absent),
    +            },
    +            Some(config) if config.kind == BackupStoreKind::Disabled => {
    +                EffectiveStore::Unconfigured(Unconfigured::Disabled)
    +            }
    +            Some(config) => EffectiveStore::Configured {
    +                config: config.clone(),
    +                defaulted: false,
    +            },
    +        };
    +
    +        let explicit_spool = self
    +            .spool_path
    +            .as_deref()
    +            .map(|p| PathBuf::from(crate::expand_tilde(p)));
    +        let filesystem_root = match &store {
    +            EffectiveStore::Configured { config, .. }
    +                if config.kind == BackupStoreKind::Filesystem =>
    +            {
    +                // An empty `path` names no directory: it is compared with
    +                // nothing here (compared as written it would be the working
    +                // directory, and refused by name as the database directory
    +                // when the server starts there), and `open` refuses it.
    +                config
    +                    .filesystem
    +                    .as_ref()
    +                    .map(|f| f.path.clone())
    +                    .filter(|path| !path.as_os_str().is_empty())
    +            }
    +            _ => None,
    +        };
    +        let is_s3 = matches!(
    +            &store,
    +            EffectiveStore::Configured { config, .. } if config.kind == BackupStoreKind::S3
    +        );
    +
    +        // The spool rules apply to every explicit `spool_path`, so a
    +        // configuration that breaks them is refused before the store it
    +        // will one day be used with is selected. Each message spells both
    +        // paths as they were compared (resolved through existing symlinks).
    +        if let Some(spool) = &explicit_spool {
    +            if let Some(root) = &filesystem_root
    +                && paths_overlap(spool, root)
    +            {
    +                return Err(format!(
    +                    "backup.spool_path {} is inside or contains the filesystem backup store \
    +                     root {}; the store would list and delete the spool. Put the spool \
    +                     outside the root",
    +                    comparable_path(spool).display(),
    +                    comparable_path(root).display()
    +                ));
    +            }
    +            if let Some(dir) = &database_dir
    +                && path_is_or_contains(spool, dir)
    +            {
    +                return Err(format!(
    +                    "backup.spool_path {} is or contains the database directory {}; put the \
    +                     spool in a directory of its own",
    +                    comparable_path(spool).display(),
    +                    comparable_path(dir).display()
    +                ));
    +            }
    +        }
    +        // A filesystem root that is the database directory or an ancestor of
    +        // it would hold the database file and its sidecars among the
    +        // backups: a listing would read the directory's entries as account
    +        // ids and a deletion of the root's whole prefix would walk them. The
    +        // default root is a child of the directory and passes.
    +        if let (Some(root), Some(dir)) = (&filesystem_root, &database_dir)
    +            && path_is_or_contains(root, dir)
    +        {
    +            return Err(format!(
    +                "backup.store.filesystem.path {} is or contains the database directory {}; \
    +                 put the backup root in a directory of its own (the default is \
    +                 {DEFAULT_ROOT_DIR_NAME}/ beside the database file)",
    +                comparable_path(root).display(),
    +                comparable_path(dir).display()
    +            ));
    +        }
    +
    +        let spool_path = if is_s3 {
    +            match (explicit_spool, &database_dir) {
    +                (Some(spool), _) => Some(spool),
    +                (None, Some(dir)) => Some(dir.join(DEFAULT_SPOOL_DIR_NAME)),
    +                (None, None) => {
    +                    return Err(
    +                        "backup.spool_path is required with an S3 backup store on this \
    +                         backend: a restore downloads its files there before loading them, \
    +                         and only a file-backed SQLite database has a default beside its file"
    +                            .to_owned(),
    +                    );
    +                }
    +            }
    +        } else {
    +            None
    +        };
    +        Ok(ResolvedStore { store, spool_path })
    +    }
    +}
    +
    +/// Whether one path is the other or an ancestor of it, after each is made
    +/// absolute and lexically normalized (`.` and `..` resolved without
    +/// consulting the filesystem, since neither directory need exist yet) and
    +/// canonicalized when it exists, so a symlinked spelling of the same
    +/// directory compares equal.
    +#[must_use]
    +pub fn paths_overlap(a: &Path, b: &Path) -> bool {
    +    let a = comparable_path(a);
    +    let b = comparable_path(b);
    +    a.starts_with(&b) || b.starts_with(&a)
    +}
    +
    +/// Whether `a` is `b` or an ancestor of it, compared as [`paths_overlap`]
    +/// compares.
    +#[must_use]
    +pub fn path_is_or_contains(a: &Path, b: &Path) -> bool {
    +    comparable_path(b).starts_with(comparable_path(a))
    +}
    +
    +fn comparable_path(path: &Path) -> PathBuf {
    +    let absolute = if path.is_absolute() {
    +        path.to_path_buf()
    +    } else {
    +        std::env::current_dir()
    +            .map(|cwd| cwd.join(path))
    +            .unwrap_or_else(|_| path.to_path_buf())
    +    };
    +    let mut normalized = PathBuf::new();
    +    for component in absolute.components() {
    +        match component {
    +            std::path::Component::CurDir => {}
    +            std::path::Component::ParentDir => {
    +                normalized.pop();
    +            }
    +            other => normalized.push(other.as_os_str()),
    +        }
    +    }
    +    // The deepest existing ancestor decides the canonical spelling; the
    +    // rest is appended as written.
    +    let mut existing = normalized.clone();
    +    let mut tail = Vec::new();
    +    loop {
    +        if let Ok(canonical) = std::fs::canonicalize(&existing) {
    +            let mut out = canonical;
    +            for part in tail.iter().rev() {
    +                out.push(part);
    +            }
    +            return out;
    +        }
    +        match (existing.file_name(), existing.parent()) {
    +            (Some(name), Some(parent)) => {
    +                tail.push(name.to_os_string());
    +                existing = parent.to_path_buf();
    +            }
    +            _ => return normalized,
    +        }
    +    }
     }
     
     fn default_data_file_target_bytes() -> u64 {
    @@ -185,15 +503,97 @@ mod tests {
             assert_eq!(c.reconcile_interval_secs, 60);
             assert_eq!(c.max_concurrent_jobs, 4);
             assert_eq!(c.spool_path, None);
    +        assert_eq!(c.store, None);
             c.validate().unwrap();
             let parsed: BackupConfig = toml::from_str("").unwrap();
             assert_eq!(parsed, c);
         }
     
    +    /// A relative root or spool is made absolute against the directory
    +    /// given (the shell's at load); an absolute one, and `~`, resolve as
    +    /// written.
         #[test]
    -    fn an_unknown_key_is_refused() {
    +    fn relative_store_and_spool_paths_are_made_absolute_at_load() {
    +        let mut c: BackupConfig = toml::from_str(
    +            "spool_path = \"rel-spool\"\n[store]\nkind = \"filesystem\"\n\
    +             [store.filesystem]\npath = \"rel-backups\"\n",
    +        )
    +        .unwrap();
    +        c.make_paths_absolute(Path::new("/srv/extenddb"));
    +        assert_eq!(
    +            c.store.as_ref().unwrap().filesystem.as_ref().unwrap().path,
    +            PathBuf::from("/srv/extenddb/rel-backups")
    +        );
    +        assert_eq!(c.spool_path.as_deref(), Some("/srv/extenddb/rel-spool"));
    +
    +        let mut absolute: BackupConfig = toml::from_str(
    +            "spool_path = \"~/spool\"\n[store]\nkind = \"filesystem\"\n\
    +             [store.filesystem]\npath = \"/var/lib/extenddb/backups\"\n",
    +        )
    +        .unwrap();
    +        absolute.make_paths_absolute(Path::new("/srv/extenddb"));
    +        assert_eq!(
    +            absolute
    +                .store
    +                .as_ref()
    +                .unwrap()
    +                .filesystem
    +                .as_ref()
    +                .unwrap()
    +                .path,
    +            PathBuf::from("/var/lib/extenddb/backups")
    +        );
    +        let home = std::env::var("HOME").unwrap();
    +        assert_eq!(
    +            absolute.spool_path.as_deref(),
    +            Some(format!("{home}/spool").as_str())
    +        );
    +        // An S3 store has no path to resolve; `None` stays `None`.
    +        let mut none = BackupConfig::default();
    +        none.make_paths_absolute(Path::new("/srv"));
    +        assert_eq!(none, BackupConfig::default());
    +        // An empty root stays empty, for `open` to refuse; it must not
    +        // become the working directory.
    +        let mut empty: BackupConfig =
    +            toml::from_str("[store]\nkind = \"filesystem\"\n[store.filesystem]\npath = \"\"\n")
    +                .unwrap();
    +        empty.make_paths_absolute(Path::new("/srv/extenddb"));
    +        assert_eq!(
    +            empty
    +                .store
    +                .as_ref()
    +                .unwrap()
    +                .filesystem
    +                .as_ref()
    +                .unwrap()
    +                .path,
    +            PathBuf::from("")
    +        );
    +    }
    +
    +    #[test]
    +    fn an_unknown_key_is_refused_at_every_level() {
             let e = toml::from_str::("store_kind = \"s3\"\n").unwrap_err();
             assert!(e.to_string().contains("store_kind"), "{e}");
    +        let e = toml::from_str::("[store]\nkid = \"s3\"\n").unwrap_err();
    +        assert!(e.to_string().contains("kid"), "{e}");
    +        let e = toml::from_str::(
    +            "[store]\nkind = \"s3\"\n[store.s3]\nbucket = \"b\"\nbukcet = \"b\"\n",
    +        )
    +        .unwrap_err();
    +        assert!(e.to_string().contains("bukcet"), "{e}");
    +    }
    +
    +    #[test]
    +    fn the_store_section_is_the_crates_shape() {
    +        let c: BackupConfig =
    +            toml::from_str("[store]\nkind = \"s3\"\n[store.s3]\nbucket = \"b\"\nprefix = \"p/\"\n")
    +                .unwrap();
    +        let store = c.store.unwrap();
    +        assert_eq!(store.kind, BackupStoreKind::S3);
    +        assert_eq!(store.s3.unwrap().bucket, "b");
    +        let c: BackupConfig = toml::from_str("[store]\nkind = \"disabled\"\n").unwrap();
    +        assert_eq!(c.store.unwrap().kind, BackupStoreKind::Disabled);
         }
     
         #[test]
    @@ -313,5 +713,267 @@ mod tests {
                     .unwrap_err()
                     .contains("reconcile_interval_secs")
             );
    +        let empty_spool = BackupConfig {
    +            spool_path: Some("  ".to_owned()),
    +            ..BackupConfig::default()
    +        };
    +        assert!(empty_spool.validate().unwrap_err().contains("spool_path"));
    +    }
    +
    +    fn s3_store() -> BackupStoreConfig {
    +        toml::from_str("kind = \"s3\"\n[s3]\nbucket = \"b\"\n").unwrap()
    +    }
    +
    +    fn fs_store(path: &str) -> BackupStoreConfig {
    +        toml::from_str(&format!(
    +            "kind = \"filesystem\"\n[filesystem]\npath = {path:?}\n"
    +        ))
    +        .unwrap()
    +    }
    +
    +    /// An absent section on a file-backed database is the filesystem store
    +    /// beside the file, marked defaulted; on a server or in-memory database
    +    /// it is unconfigured; `disabled` is unconfigured on both.
    +    #[test]
    +    fn an_absent_section_defaults_per_backend() {
    +        let db = Path::new("/var/lib/extenddb/data.sqlite");
    +        let resolved = BackupConfig::default().resolve_store(Some(db)).unwrap();
    +        assert_eq!(
    +            resolved.store,
    +            EffectiveStore::Configured {
    +                config: fs_store("/var/lib/extenddb/backups"),
    +                defaulted: true,
    +            }
    +        );
    +        assert_eq!(
    +            resolved.spool_path, None,
    +            "a filesystem store needs no spool"
    +        );
    +
    +        let resolved = BackupConfig::default().resolve_store(None).unwrap();
    +        assert_eq!(
    +            resolved.store,
    +            EffectiveStore::Unconfigured(Unconfigured::Absent)
    +        );
    +        assert_eq!(resolved.spool_path, None);
    +
    +        let disabled = BackupConfig {
    +            store: Some(toml::from_str("kind = \"disabled\"").unwrap()),
    +            ..BackupConfig::default()
    +        };
    +        for database in [Some(db), None] {
    +            assert_eq!(
    +                disabled.resolve_store(database).unwrap().store,
    +                EffectiveStore::Unconfigured(Unconfigured::Disabled)
    +            );
    +        }
    +
    +        let explicit = BackupConfig {
    +            store: Some(fs_store("/srv/backups")),
    +            ..BackupConfig::default()
    +        };
    +        assert_eq!(
    +            explicit.resolve_store(Some(db)).unwrap().store,
    +            EffectiveStore::Configured {
    +                config: fs_store("/srv/backups"),
    +                defaulted: false,
    +            }
    +        );
    +    }
    +
    +    /// An S3 store's spool: explicit on any backend, defaulted beside a
    +    /// database file, required otherwise with the key named.
    +    #[test]
    +    fn the_spool_is_required_with_s3_except_beside_a_database_file() {
    +        let db = Path::new("/var/lib/extenddb/data.sqlite");
    +        let s3 = BackupConfig {
    +            store: Some(s3_store()),
    +            ..BackupConfig::default()
    +        };
    +        assert_eq!(
    +            s3.resolve_store(Some(db)).unwrap().spool_path,
    +            Some(PathBuf::from("/var/lib/extenddb/backup-spool"))
    +        );
    +        let e = s3.resolve_store(None).unwrap_err();
    +        assert!(e.contains("backup.spool_path is required"), "{e}");
    +
    +        let with_spool = BackupConfig {
    +            spool_path: Some("/srv/spool".to_owned()),
    +            ..s3.clone()
    +        };
    +        for database in [Some(db), None] {
    +            assert_eq!(
    +                with_spool.resolve_store(database).unwrap().spool_path,
    +                Some(PathBuf::from("/srv/spool"))
    +            );
    +        }
    +    }
    +
    +    /// Every cell of the spool placement rule: inside the root, containing
    +    /// the root, equal to the root; the database directory itself, an
    +    /// ancestor of it; and the accepted shapes (a sibling of the root, a
    +    /// directory under the database directory, a path elsewhere), each
    +    /// against a filesystem store and against an S3 store.
    +    #[test]
    +    fn a_spool_inside_or_containing_the_root_or_the_database_directory_is_refused() {
    +        let db = Path::new("/var/lib/extenddb/data.sqlite");
    +        let resolve = |store: BackupStoreConfig, spool: &str| {
    +            BackupConfig {
    +                store: Some(store),
    +                spool_path: Some(spool.to_owned()),
    +                ..BackupConfig::default()
    +            }
    +            .resolve_store(Some(db))
    +        };
    +        for spool in [
    +            "/srv/backups/spool",
    +            "/srv",
    +            "/srv/backups",
    +            "/srv/backups/../backups",
    +        ] {
    +            let e = resolve(fs_store("/srv/backups"), spool).unwrap_err();
    +            assert!(e.contains("filesystem backup store root"), "{spool}: {e}");
    +        }
    +        for spool in ["/var/lib/extenddb", "/var/lib", "/var/lib/extenddb/./"] {
    +            let e = resolve(fs_store("/srv/backups"), spool).unwrap_err();
    +            assert!(e.contains("database directory"), "{spool}: {e}");
    +            let e = resolve(s3_store(), spool).unwrap_err();
    +            assert!(e.contains("database directory"), "{spool}: {e}");
    +        }
    +        for spool in [
    +            "/srv/backup-spool",
    +            "/var/lib/extenddb/backup-spool",
    +            "/tmp/extenddb-spool",
    +        ] {
    +            resolve(fs_store("/srv/backups"), spool).unwrap_or_else(|e| panic!("{spool}: {e}"));
    +            let resolved = resolve(s3_store(), spool).unwrap_or_else(|e| panic!("{spool}: {e}"));
    +            assert_eq!(resolved.spool_path, Some(PathBuf::from(spool)));
    +        }
    +        // The defaulted root beside the database file is a root too.
    +        let e = BackupConfig {
    +            spool_path: Some("/var/lib/extenddb/backups/spool".to_owned()),
    +            ..BackupConfig::default()
    +        }
    +        .resolve_store(Some(db))
    +        .unwrap_err();
    +        assert!(e.contains("filesystem backup store root"), "{e}");
    +    }
    +
    +    #[test]
    +    fn path_overlap_follows_an_existing_symlink_and_normalizes_dots() {
    +        let dir = tempfile::tempdir().unwrap();
    +        let real = dir.path().join("real");
    +        std::fs::create_dir(&real).unwrap();
    +        let link = dir.path().join("link");
    +        std::os::unix::fs::symlink(&real, &link).unwrap();
    +        assert!(paths_overlap(&link.join("spool"), &real));
    +        assert!(paths_overlap(&real, &link));
    +        assert!(paths_overlap(
    +            &dir.path().join("real/x/../y"),
    +            &dir.path().join("real/y/z")
    +        ));
    +        assert!(!paths_overlap(
    +            &dir.path().join("real-2"),
    +            &dir.path().join("real")
    +        ));
    +    }
    +
    +    /// A filesystem root that is the database directory (`"."` with the
    +    /// database file in the working directory, or the directory spelled
    +    /// out) or an ancestor of it is refused naming the key and both
    +    /// directories; a child of the directory (the default root among them),
    +    /// a sibling and a root elsewhere are accepted, and the rule does not
    +    /// apply to an S3 store or to a server database (no directory).
    +    #[test]
    +    fn a_root_that_is_or_contains_the_database_directory_is_refused() {
    +        let db = Path::new("/var/lib/extenddb/data.sqlite");
    +        let resolve = |path: &str| {
    +            BackupConfig {
    +                store: Some(fs_store(path)),
    +                ..BackupConfig::default()
    +            }
    +            .resolve_store(Some(db))
    +        };
    +        for root in ["/var/lib/extenddb", "/var/lib/extenddb/./", "/var/lib", "/"] {
    +            let e = resolve(root).unwrap_err();
    +            assert!(e.contains("backup.store.filesystem.path"), "{root}: {e}");
    +            assert!(
    +                e.contains("database directory /var/lib/extenddb"),
    +                "{root}: {e}"
    +            );
    +        }
    +        for root in [
    +            "/var/lib/extenddb/backups",
    +            "/var/lib/extenddb-backups",
    +            "/srv/backups",
    +        ] {
    +            resolve(root).unwrap_or_else(|e| panic!("{root}: {e}"));
    +        }
    +        BackupConfig::default()
    +            .resolve_store(Some(db))
    +            .expect("the default root is a child of the directory");
    +        BackupConfig {
    +            store: Some(fs_store("/var/lib/extenddb")),
    +            ..BackupConfig::default()
    +        }
    +        .resolve_store(None)
    +        .expect("a server database has no directory to protect");
    +    }
    +
    +    /// An empty root is compared with nothing: a server started from the
    +    /// database's directory (`cd /var/lib/extenddb && extenddb serve`, the
    +    /// layout `init` leaves) with `path = ""` is not refused as a root that
    +    /// is the database directory, a path the operator never wrote; the
    +    /// configuration resolves with the empty path, which `open` refuses by
    +    /// name. The spool rule likewise compares nothing with it.
    +    #[test]
    +    fn an_empty_root_is_left_for_open_to_refuse() {
    +        let cwd = std::env::current_dir().unwrap();
    +        let db = cwd.join("data.sqlite");
    +        let resolved = BackupConfig {
    +            store: Some(fs_store("")),
    +            ..BackupConfig::default()
    +        }
    +        .resolve_store(Some(&db))
    +        .expect("an empty root is not the working directory");
    +        assert!(matches!(
    +            resolved.store,
    +            EffectiveStore::Configured { config, defaulted: false }
    +                if config.filesystem.as_ref().unwrap().path.as_os_str().is_empty()
    +        ));
    +        BackupConfig {
    +            spool_path: Some("/srv/spool".to_owned()),
    +            store: Some(fs_store("")),
    +            ..BackupConfig::default()
    +        }
    +        .resolve_store(Some(&db))
    +        .expect("a spool is not inside an empty root");
    +    }
    +
    +    /// A refusal spells both paths as they were compared, so a spool given
    +    /// through a symlink and a database directory given by its real path do
    +    /// not read as two different places.
    +    #[test]
    +    fn a_placement_refusal_spells_both_paths_the_same_way() {
    +        let dir = tempfile::tempdir().unwrap();
    +        let real = dir.path().join("real");
    +        std::fs::create_dir(&real).unwrap();
    +        let link = dir.path().join("link");
    +        std::os::unix::fs::symlink(&real, &link).unwrap();
    +        let e = BackupConfig {
    +            spool_path: Some(link.display().to_string()),
    +            store: Some(toml::from_str("kind = \"disabled\"").unwrap()),
    +            ..BackupConfig::default()
    +        }
    +        .resolve_store(Some(&real.join("data.sqlite")))
    +        .unwrap_err();
    +        let real = real.canonicalize().unwrap().display().to_string();
    +        assert!(
    +            e.contains(&format!(
    +                "backup.spool_path {real} is or contains the database directory {real}"
    +            )),
    +            "{e}"
    +        );
    +        assert!(!e.contains("link"), "{e}");
         }
     }
    diff --git a/crates/config/src/lib.rs b/crates/config/src/lib.rs
    index 5b1cb603a..ccdb5833d 100644
    --- a/crates/config/src/lib.rs
    +++ b/crates/config/src/lib.rs
    @@ -179,6 +179,13 @@ impl StorageConfig {
             self.config.startup_warnings()
         }
     
    +    /// The local file the database lives in, when the backend's database is
    +    /// one file on this host ([`extenddb_storage::config::StorageConfig::database_file`]).
    +    #[must_use]
    +    pub fn database_file(&self) -> Option {
    +        self.config.database_file()
    +    }
    +
         /// Get a reference to the underlying trait object for factory calls.
         pub fn as_trait(&self) -> &(dyn extenddb_storage::config::StorageConfig + 'static) {
             &*self.config
    @@ -423,9 +430,13 @@ pub fn load(config_path: &str) -> anyhow::Result {
         validated(config.try_deserialize()?)
     }
     
    -/// The cross-field checks a deserialized config still has to pass.
    -fn validated(config: AppConfig) -> anyhow::Result {
    +/// The cross-field checks a deserialized config still has to pass, and the
    +/// paths made absolute while the working directory is still the shell's.
    +fn validated(mut config: AppConfig) -> anyhow::Result {
         config.backup.validate().map_err(|e| anyhow::anyhow!(e))?;
    +    if let Ok(cwd) = std::env::current_dir() {
    +        config.backup.make_paths_absolute(&cwd);
    +    }
         Ok(config)
     }
     
    diff --git a/crates/engine/Cargo.toml b/crates/engine/Cargo.toml
    index 9f58bea6e..d8e20bc15 100755
    --- a/crates/engine/Cargo.toml
    +++ b/crates/engine/Cargo.toml
    @@ -11,6 +11,7 @@ license.workspace = true
     extenddb-auth = { workspace = true }
     extenddb-core = { workspace = true }
     extenddb-storage = { workspace = true }
    +extenddb-backup-store = { workspace = true }
     serde = { workspace = true }
     serde_json = { workspace = true }
     tracing = { workspace = true }
    @@ -22,3 +23,7 @@ sha2 = "0.10"
     hex = "0.4"
     tokio = { workspace = true }
     futures = { workspace = true }
    +
    +[dev-dependencies]
    +# `start_paused` for the tests of timers that must not depend on the wall clock.
    +tokio = { workspace = true, features = ["test-util"] }
    diff --git a/crates/engine/src/backup.rs b/crates/engine/src/backup.rs
    index 8b7301ad7..432fcb8b6 100755
    --- a/crates/engine/src/backup.rs
    +++ b/crates/engine/src/backup.rs
    @@ -35,7 +35,12 @@ use extenddb_storage::{
     use futures::TryStreamExt;
     use serde_json::{Value, json};
     
    -pub use runtime::BackupRuntime;
    +pub use maintenance::HeldClaim;
    +pub use runtime::{
    +    BackupRuntime, ConfiguredStore, LOCAL_CHECK_TIMEOUT, S3_CHECK_TIMEOUT, Spool, StoreHealth,
    +    StoreHealthReport, check_configured_store, read_identity_bounded,
    +};
    +pub use store_identity::{STORE_IDENTITY_KEY, StoreIdentityFile, StoreProblem};
     
     use crate::{OperationContext, serialize_output};
     
    @@ -420,8 +425,9 @@ pub(crate) async fn handle_create_backup(
     /// `DescribeTable` of the table, which the handler ran.
     ///
     /// This is the in-catalog create, the path while no `[backup.store]` is
    -/// configured; each call logs that the format is deprecated.
    -async fn create_backup(
    +/// configured; each call logs that the format is deprecated. Last before the
    +/// permit, the maintenance claim ([`refuse_while_maintenance_held`]).
    +async fn create_backup(
         storage: &S,
         runtime: &BackupRuntime,
         account_id: &str,
    @@ -442,18 +448,78 @@ async fn create_backup(
         };
         check_backup_arn_fits(region, account_id, table_name)?;
         refuse_backup_of_vector_table(&desc)?;
    +    // A create that writes to the configured store needs it healthy, and is
    +    // refused before a slot is taken when it is not, as every operation that
    +    // touches the store is; the in-catalog create touches no store, so the
    +    // store's health does not gate it (a backend bound to the in-catalog
    +    // create, or a server with an unhealthy store, creates as a server
    +    // without a store does).
    +    let store_backed = runtime.configured_store().is_some()
    +        && !runtime.legacy_create_only()
    +        && STORE_BACKED_CREATE;
    +    if store_backed {
    +        runtime.require_healthy_store()?;
    +    }
    +    refuse_while_maintenance_held(storage, runtime).await?;
         let permit = runtime.take_permit()?;
    -    tracing::warn!(
    -        "CreateBackup of {table_name} (account {account_id}) is writing the in-catalog backup \
    -         format, which a later release stops creating; configure [backup.store] to write \
    -         backups to a backup store"
    -    );
    +    match runtime.configured_store() {
    +        None => tracing::warn!(
    +            "CreateBackup of {table_name} (account {account_id}) is writing the in-catalog \
    +             backup format, which a later release stops creating; configure [backup.store] to \
    +             write backups to a backup store"
    +        ),
    +        Some(store) => tracing::info!(
    +            "CreateBackup of {table_name} (account {account_id}) is writing the in-catalog \
    +             backup format; the configured {} backup store serves restores and deletes of the \
    +             backups it holds, and a later release's CreateBackup writes to it",
    +            store.kind
    +        ),
    +    }
         storage
             .create(account_id, table_name, backup_name, permit)
             .await
             .map_err(|e| create_backup_storage_err_to_dynamo(e, table_name))
     }
     
    +/// The jobs' side of the catalog-level maintenance claim: a backup or
    +/// restore job is refused with `LimitExceededException` while a maintenance
    +/// command (`extenddb backup purge`) holds the claim, read with
    +/// `stale_job_after_secs` so a dead command's claim expires. The check runs
    +/// after every other refusal and before the permit, which on this head is
    +/// before the job's row exists: the in-catalog create and restore commit
    +/// their row inside the backend and detach, so the engine has no point
    +/// between the row and the copy to check at. A claim taken between this
    +/// check and the row's insert is therefore not seen by the job; the command
    +/// create that slips through writes no object of the store the command
    +/// deletes from, and a restore that slips through is a RUNNING record the
    +/// command's wait lists, so the window costs nothing in this release. The
    +/// store-backed job of a later release owns its row from the
    +/// engine and checks after the row exists, as the design's 5.1 step 2 says.
    +async fn refuse_while_maintenance_held(
    +    storage: &S,
    +    runtime: &BackupRuntime,
    +) -> Result<(), DynamoDbError> {
    +    let held = storage
    +        .check_maintenance_claim(runtime.stale_job_after())
    +        .await
    +        .map_err(storage_err_to_dynamo)?;
    +    if held {
    +        return Err(DynamoDbError::LimitExceededException(
    +            "A maintenance command (extenddb backup purge) holds this catalog's maintenance \
    +             claim; backup and restore jobs are refused until it finishes"
    +                .to_owned(),
    +        ));
    +    }
    +    Ok(())
    +}
    +
    +/// Whether CreateBackup writes to the configured store. No store-backed
    +/// create exists on this head: every create writes the in-catalog format,
    +/// whatever the store configuration, and the release that adds the
    +/// store-backed job turns this on for a server with a store configured and
    +/// not bound to the in-catalog create.
    +const STORE_BACKED_CREATE: bool = false;
    +
     /// The backup ARN rules a table name has to fit before a backup of it can be
     /// created, each refusal naming its cause: first the server's own region
     /// (`server.region` outside `[a-z0-9-]` forms no backup ARN whatever the
    @@ -1086,16 +1152,21 @@ async fn delete_backup(
     /// `BackupNotFoundException`; a CREATING row is `BackupInUseException`; a
     /// backup a CREATING restore target still reads is `BackupInUseException`
     /// naming that table ([`BackupCatalog::backup_in_use`]); a row of a store
    -/// other than the configured one is a `ValidationException` naming no root;
    +/// other than the configured one is a `ValidationException` naming no root,
    +/// and a row of the configured store while that store is unhealthy or its
    +/// first check pending is a `ValidationException` naming the store's kind;
     /// then the in-use check runs again with the AVAILABLE-to-DELETING write in
     /// one transaction under the row's lock
     /// ([`BackupCatalog::start_backup_delete`]), so a restore's registration and
     /// this order wholly one before the other: a registration that won leaves
     /// this `BackupInUseException` naming its target. A DELETING row (a retried
    -/// DeleteBackup) resumes the deletion. The request returns the description
    -/// with `BackupStatus` DELETED; the store's objects are deleted by the
    -/// duty a configured store brings, and the row stays DELETING until a
    -/// listing of its prefix returns nothing.
    +/// DeleteBackup) resumes the deletion. Then the deletion itself
    +/// ([`finish_backup_delete`]): the manifest first, so the prefix is no longer
    +/// a complete backup to any reader, then the rest of the prefix, then the row
    +/// to DELETED once a listing of the prefix returns nothing. If any step fails
    +/// the row stays DELETING, the request is an `InternalServerError`, and the
    +/// reconciler finishes the deletion later. The request returns the
    +/// description with `BackupStatus` DELETED.
     ///
     /// The description is built before the DELETING write: a row whose
     /// `source_table_details_json` this build cannot read is answered as an
    @@ -1124,6 +1195,7 @@ async fn delete_catalog_backup(
         if !runtime.is_configured_store(&row.store) {
             return Err(backup_in_another_store(&arn_text));
         }
    +    let store = runtime.require_healthy_store()?;
         let mut desc = row.description().map_err(storage_err_to_dynamo)?;
         match storage
             .start_backup_delete(&key)
    @@ -1148,10 +1220,68 @@ async fn delete_catalog_backup(
                 ))));
             }
         }
    +    finish_backup_delete(storage, &*store.store, arn)
    +        .await
    +        .map_err(|e| {
    +            tracing::error!(
    +                "DeleteBackup {arn_text}: the row is DELETING but its objects could not be \
    +                 deleted; the reconciler retries: {e}"
    +            );
    +            DynamoDbError::InternalServerError("Internal server error".to_owned())
    +        })?;
         desc.backup_details.backup_status = BackupStatus::Deleted.as_str().to_owned();
         Ok(desc)
     }
     
    +/// The deletion of a DELETING row's objects and the row's move to DELETED:
    +/// `extenddb-manifest.json` first (a `delete_prefix` on its exact key, so the
    +/// prefix is no longer a complete backup to any reader), then the rest of the
    +/// prefix, then, once a listing of the prefix returns nothing,
    +/// `set_backup_deleted`. Shared by DeleteBackup and the reconciler's duty for
    +/// a DELETING row; either may run it on a row the other started, and a run
    +/// that finds the row no longer DELETING (`false` from the catalog) changes
    +/// nothing more. Returns whether this run moved the row.
    +///
    +/// # Errors
    +///
    +/// A store failure, a listing that still shows objects (a put that
    +/// overlapped the deletion, which the next run removes), or a catalog write
    +/// failure; the row stays DELETING for the next run.
    +pub(crate) async fn finish_backup_delete(
    +    storage: &S,
    +    store: &dyn extenddb_backup_store::BackupStore,
    +    arn: &BackupArn,
    +) -> Result {
    +    let prefix = arn.store_prefix();
    +    let manifest_key = format!(
    +        "{prefix}{}",
    +        extenddb_core::backup_format::EXTENDDB_MANIFEST_FILE
    +    );
    +    store
    +        .delete_prefix(&manifest_key)
    +        .await
    +        .map_err(|e| format!("deleting the manifest: {e}"))?;
    +    store
    +        .delete_prefix(&prefix)
    +        .await
    +        .map_err(|e| format!("deleting the prefix: {e}"))?;
    +    let remaining = store
    +        .list(&prefix)
    +        .try_next()
    +        .await
    +        .map_err(|e| format!("listing the prefix after the deletion: {e}"))?;
    +    if let Some(object) = remaining {
    +        return Err(format!(
    +            "the prefix still lists {} after the deletion",
    +            object.key
    +        ));
    +    }
    +    storage
    +        .set_backup_deleted(&BackupKey::from_arn(arn))
    +        .await
    +        .map_err(|e| format!("marking the row DELETED: {e}"))
    +}
    +
     /// Overrides refused outright. `VectorIndexOverride` is not in the botocore
     /// 1.33.13 model; newer models and SDKs send it. The index overrides and
     /// `OnDemandThroughputOverride` are applied as the service applies them; see
    @@ -1330,14 +1460,17 @@ where
     /// first ([`check_catalog_row_for_restore`]: a CREATING row or one in use is
     /// `BackupInUseException`, a DELETING, DELETED or FAILED row
     /// `BackupNotFoundException`, a row of another store a
    -/// `ValidationException`), else the legacy reader's restore source;
    -/// (5) refuse what no backend restores faithfully (multi-part key, vector
    +/// `ValidationException`, a row of the configured store while it is
    +/// unhealthy one naming the store's kind), else the legacy reader's restore
    +/// source; (5) refuse what no backend restores faithfully (multi-part key, vector
     /// indexes); (6) check the throughput override against the backup's billing
     /// mode, build the target's `CreateTableInput`, and apply the index and
     /// on-demand throughput overrides to it, refusing what the service refuses
    -/// ([`overrides`]); (7) validate that input as CreateTable does; (8) take the
    -/// job permit, last among the checks (`LimitExceededException` when none is
    -/// free), and ask the backend to create and register the target; (9) map
    +/// ([`overrides`]); (7) validate that input as CreateTable does; (8) refuse
    +/// while a maintenance command holds the catalog's claim
    +/// ([`refuse_while_maintenance_held`]), then take the job permit, last among
    +/// the checks (`LimitExceededException` for either), and ask the backend to
    +/// create and register the target; (9) map
     /// its errors, a taken name being `TableAlreadyExistsException`, a name
     /// being restored into `TableInUseException` (the backend checks the name
     /// again, fenced, under the registration), a backup being restored from
    @@ -1395,14 +1528,16 @@ async fn restore_table_from_backup(
         storage: &S,
         runtime: &BackupRuntime,
    @@ -1446,6 +1584,7 @@ async fn check_catalog_row_for_restore(
         if !runtime.is_configured_store(&row.store) {
             return Err(backup_in_another_store(&arn_text));
         }
    +    runtime.require_healthy_store()?;
         Err(DynamoDbError::ValidationException(format!(
             "Backup {arn_text} is in a backup store; this release restores in-catalog backups only"
         )))
    @@ -1678,8 +1817,10 @@ fn storage_err_to_dynamo(e: StorageError) -> DynamoDbError {
         }
     }
     
    +pub mod maintenance;
     mod overrides;
     pub mod reconciler;
     mod runtime;
    +pub mod store_identity;
     #[cfg(test)]
     mod tests;
    diff --git a/crates/engine/src/backup/maintenance.rs b/crates/engine/src/backup/maintenance.rs
    new file mode 100644
    index 000000000..541982a7e
    --- /dev/null
    +++ b/crates/engine/src/backup/maintenance.rs
    @@ -0,0 +1,537 @@
    +// Copyright 2026 ExtendDB contributors
    +// SPDX-License-Identifier: Apache-2.0
    +
    +//! The holder's side of the catalog-level maintenance claim, and the steps a
    +//! maintenance command (`extenddb backup purge`) runs under it.
    +//!
    +//! [`HeldClaim`] is the handle the command keeps from the take to the
    +//! release, with the heartbeat running on a timer for the handle's whole
    +//! life, so every reader of the claim (`check_maintenance_claim` on every
    +//! backend measures the heartbeat's age, PostgreSQL the lock first) sees it
    +//! held through a wait and a deletion phase of any length. A heartbeat that
    +//! finds the claim lost (another holder took it over after this one looked
    +//! dead), or fails, is recorded for the command to stop at before its next
    +//! destructive step. Dropping the handle without `release` stops the timer
    +//! (the claim then expires by the stale rule, as a dead holder's does);
    +//! `release` stops it, waits for a heartbeat in flight, then releases.
    +//!
    +//! [`wait_for_quiet`] is the wait for the jobs the claim excludes from
    +//! starting to finish, and [`purge_row`] the deletion of one row's objects
    +//! and the row, each written against the catalog's trait so the engine's
    +//! tests run them against a double; the command prints what they return. An
    +//! [`Interrupt`] carries a signal the command received to both, so an
    +//! operator's Ctrl-C ends the wait at once and the deletion loop before its
    +//! next row, never inside one (the row's own steps are the idempotent
    +//! sequence a second deleter finishes).
    +
    +use std::sync::{Arc, Mutex, PoisonError};
    +use std::time::Duration;
    +
    +use extenddb_backup_store::BackupStore;
    +use extenddb_core::backup_format::{BackupArn, EXTENDDB_MANIFEST_FILE};
    +use extenddb_storage::backup_catalog::{
    +    BackupRow, BackupStatus, DeleteStart, Heartbeat, JobStatus, StoreIdentity, is_backup_not_found,
    +};
    +use extenddb_storage::{BackupCatalog, MaintenanceClaim};
    +use futures::TryStreamExt;
    +use tokio::sync::watch;
    +use tokio::time::Instant;
    +
    +/// A maintenance claim with its heartbeat on a timer.
    +pub struct HeldClaim {
    +    claim: Arc,
    +    lost: Arc>>,
    +    timer: tokio::task::JoinHandle<()>,
    +}
    +
    +impl HeldClaim {
    +    /// Hold `claim`, heartbeating it every `interval` from now until
    +    /// [`HeldClaim::release`] or the drop (the caller passes the jobs'
    +    /// heartbeat period, `extenddb_storage::backup_catalog::JOB_HEARTBEAT_INTERVAL`,
    +    /// which every stale window the config accepts is at least three of).
    +    #[must_use]
    +    pub fn start(claim: Box, interval: Duration) -> Self {
    +        let claim: Arc = Arc::from(claim);
    +        let lost = Arc::new(Mutex::new(None));
    +        let timer = tokio::spawn({
    +            let claim = Arc::clone(&claim);
    +            let lost = Arc::clone(&lost);
    +            async move {
    +                let mut ticks = tokio::time::interval(interval);
    +                ticks.tick().await;
    +                loop {
    +                    ticks.tick().await;
    +                    let reason = match claim.heartbeat().await {
    +                        Ok(Heartbeat::Confirmed) => continue,
    +                        Ok(Heartbeat::Lost) => {
    +                            "the maintenance claim was lost (another holder took it over)"
    +                                .to_owned()
    +                        }
    +                        Err(e) => format!("the maintenance claim's heartbeat failed: {e}"),
    +                    };
    +                    *lost.lock().unwrap_or_else(PoisonError::into_inner) = Some(reason);
    +                    return;
    +                }
    +            }
    +        });
    +        Self { claim, lost, timer }
    +    }
    +
    +    /// The claim's fencing token.
    +    #[must_use]
    +    pub fn generation(&self) -> u64 {
    +        self.claim.generation()
    +    }
    +
    +    /// Why the claim is no longer held, once a heartbeat said so; `None`
    +    /// while every heartbeat has confirmed it.
    +    #[must_use]
    +    pub fn lost(&self) -> Option {
    +        self.lost
    +            .lock()
    +            .unwrap_or_else(PoisonError::into_inner)
    +            .clone()
    +    }
    +
    +    /// Stop the heartbeat, wait for one in flight to end, and release the
    +    /// claim.
    +    ///
    +    /// # Errors
    +    ///
    +    /// The backend's, when the release could not be written; the claim then
    +    /// expires by the stale rule.
    +    pub async fn release(mut self) -> Result<(), extenddb_storage::error::StorageError> {
    +        self.timer.abort();
    +        // The abort lands at the task's next await; a heartbeat already sent
    +        // finishes first, so the release never overlaps it on the claim's
    +        // connection. A cancelled task is the expected outcome.
    +        let _ = (&mut self.timer).await;
    +        self.claim.release().await
    +    }
    +}
    +
    +impl Drop for HeldClaim {
    +    fn drop(&mut self) {
    +        self.timer.abort();
    +    }
    +}
    +
    +/// A signal the command received (`SIGINT` or `SIGTERM`), for the wait and
    +/// the deletion loop to stop at: the wait at once, the loop before its next
    +/// row. The sender side is the command's signal listener; a test's is the
    +/// test.
    +#[derive(Clone)]
    +pub struct Interrupt {
    +    rx: watch::Receiver>,
    +}
    +
    +impl Interrupt {
    +    /// A pair: the sender a signal listener sets with the signal's name, and
    +    /// the handle the wait and the loop read.
    +    #[must_use]
    +    pub fn new() -> (watch::Sender>, Self) {
    +        let (tx, rx) = watch::channel(None);
    +        (tx, Self { rx })
    +    }
    +
    +    /// The signal's name once one was received.
    +    #[must_use]
    +    pub fn signalled(&self) -> Option<&'static str> {
    +        *self.rx.borrow()
    +    }
    +
    +    /// Resolves with the signal's name once one is received; pending until
    +    /// then (forever, when the sender is gone and none was).
    +    pub async fn wait(&mut self) -> &'static str {
    +        loop {
    +            if let Some(name) = *self.rx.borrow_and_update() {
    +                return name;
    +            }
    +            if self.rx.changed().await.is_err() {
    +                return std::future::pending().await;
    +            }
    +        }
    +    }
    +}
    +
    +/// How [`wait_for_quiet`] ended.
    +#[derive(Debug, Clone, PartialEq, Eq)]
    +pub enum WaitOutcome {
    +    /// No job the claim excludes is running: the deletion may start.
    +    Quiet,
    +    /// The limit passed with these jobs still running (the operator's text,
    +    /// one per job).
    +    Busy(Vec),
    +    /// A heartbeat found the claim lost or failed; the reason.
    +    Lost(String),
    +    /// The rows could not be read; the backend's error.
    +    CatalogError(String),
    +    /// The command received this signal.
    +    Interrupted(&'static str),
    +}
    +
    +/// The durations of a [`wait_for_quiet`].
    +#[derive(Debug, Clone, Copy)]
    +pub struct Wait {
    +    /// A heartbeat within this of now (this host's clock) is a live owner;
    +    /// zero where every owner is known dead (SQLite under the serve lock).
    +    pub owner_window: Duration,
    +    /// How long to wait before giving up with the jobs still running.
    +    pub limit: Duration,
    +    /// How often the rows are re-read.
    +    pub poll: Duration,
    +}
    +
    +/// Wait until no CREATING or FAILED backup row of `identity` has a live
    +/// owner and no restore job is RUNNING, re-reading every `wait.poll`, for
    +/// at most `wait.limit`; the claim lost, a catalog read failing, or a
    +/// signal ends it early. The jobs found running are reported through
    +/// `announce` the first time they are seen.
    +pub async fn wait_for_quiet(
    +    storage: &S,
    +    identity: &StoreIdentity,
    +    claim: &HeldClaim,
    +    wait: Wait,
    +    interrupt: &mut Interrupt,
    +    announce: impl Fn(&[String]),
    +) -> WaitOutcome {
    +    let started = Instant::now();
    +    let mut announced = false;
    +    loop {
    +        if let Some(reason) = claim.lost() {
    +            return WaitOutcome::Lost(reason);
    +        }
    +        if let Some(signal) = interrupt.signalled() {
    +            return WaitOutcome::Interrupted(signal);
    +        }
    +        let remaining = match busy_jobs(storage, identity, wait.owner_window).await {
    +            Ok(remaining) => remaining,
    +            Err(e) => return WaitOutcome::CatalogError(e),
    +        };
    +        if remaining.is_empty() {
    +            return WaitOutcome::Quiet;
    +        }
    +        if started.elapsed() >= wait.limit {
    +            return WaitOutcome::Busy(remaining);
    +        }
    +        if !announced {
    +            announce(&remaining);
    +            announced = true;
    +        }
    +        tokio::select! {
    +            () = tokio::time::sleep(wait.poll) => {}
    +            signal = interrupt.wait() => return WaitOutcome::Interrupted(signal),
    +        }
    +    }
    +}
    +
    +/// The jobs purge waits on, as text for the operator.
    +async fn busy_jobs(
    +    storage: &S,
    +    identity: &StoreIdentity,
    +    stale_after: Duration,
    +) -> Result, String> {
    +    let mut busy = Vec::new();
    +    let now = std::time::SystemTime::now()
    +        .duration_since(std::time::UNIX_EPOCH)
    +        .map_or(0.0, |d| d.as_secs_f64());
    +    // A live owner heartbeats on the catalog's clock; this host's clock
    +    // stands in for it here, so a skew of up to the stale window makes the
    +    // wait longer, never shorter than the owner's death. A FAILED row whose
    +    // heartbeat is young may still be written to by a job that has not seen
    +    // its claim lost; its prefix waits for the window as the reconciler's
    +    // deletion of it does.
    +    for status in [BackupStatus::Creating, BackupStatus::Failed] {
    +        let rows: Vec = storage
    +            .list_backup_rows(Some(status), identity)
    +            .try_collect()
    +            .await
    +            .map_err(|e| format!("cannot list the catalog's backup rows: {e}"))?;
    +        for row in rows {
    +            if row.heartbeat_at >= now - stale_after.as_secs_f64() {
    +                busy.push(format!(
    +                    "{} {} has a live owner (heartbeat within stale_job_after_secs)",
    +                    if status == BackupStatus::Creating {
    +                        "CreateBackup"
    +                    } else {
    +                        "the FAILED backup"
    +                    },
    +                    row.backup_arn
    +                ));
    +            }
    +        }
    +    }
    +    // Every RUNNING restore job: the dead-job listing with a zero window
    +    // lists every RUNNING record whose heartbeat is in the past, which is
    +    // all of them. It also lists the FAILED records whose target is still
    +    // CREATING (the FAILED-target duty's), which are the reconciler's to
    +    // finish, not jobs: nothing runs under one, and purge deletes no table,
    +    // so they are not waited on.
    +    let listed = storage
    +        .list_dead_restore_jobs(Duration::ZERO)
    +        .await
    +        .map_err(|e| format!("cannot list restore jobs: {e}"))?;
    +    for job in listed {
    +        if job.status != JobStatus::Running {
    +            continue;
    +        }
    +        busy.push(format!(
    +            "a restore into {} (account {}) is RUNNING",
    +            job.target.table_name, job.target.account_id
    +        ));
    +    }
    +    Ok(busy)
    +}
    +
    +/// Delete one row's objects and the row: the manifest first, then the
    +/// prefix, then the row through the status machine to a tombstone and out.
    +/// The row is `row` as the command read it; the status another deleter may
    +/// have moved it to since is read from `start_backup_delete`'s answer, and
    +/// each state the row can be in when the command reaches it ends with no
    +/// row: AVAILABLE is moved to DELETING here; DELETING (another deleter is
    +/// mid-way) and DELETED (another finished) are finished here, the prefix
    +/// deletion being the same idempotent sequence on both sides; a row already
    +/// gone counts as deleted. A CREATING row's owner is dead (the wait saw no
    +/// live one on this host's clock): the row goes first, through the
    +/// dead-owner removal on the catalog's clock, and the prefix only once the
    +/// row is gone, so a clock skew that makes the catalog still see a young
    +/// heartbeat leaves the row and its objects together rather than a row over
    +/// a deleted prefix. A prefix left by a failure between the two steps holds
    +/// no manifest (a CREATING backup has none yet), so nothing adopts it.
    +///
    +/// # Errors
    +///
    +/// The reason the row was left, for the operator: a FAILED or CREATING
    +/// status another writer moved an AVAILABLE row to (neither follows
    +/// AVAILABLE; the row is left for a look), a restore reading the backup, a
    +/// CREATING row whose owner heartbeated after the wait, a store or catalog
    +/// call failing, or a prefix that still lists an object.
    +pub async fn purge_row(
    +    storage: &S,
    +    store: &dyn BackupStore,
    +    row: &BackupRow,
    +    stale_after: Duration,
    +) -> Result<(), String> {
    +    let arn = BackupArn::parse(&row.backup_arn)
    +        .map_err(|e| format!("the row's ARN does not parse: {e}"))?;
    +    let key = row.key();
    +    if row.status == BackupStatus::Creating
    +        && !storage
    +            .remove_dead_creating_backup_row(&key, stale_after)
    +            .await
    +            .map_err(|e| format!("cannot remove the CREATING row: {e}"))?
    +    {
    +        return Err(
    +            "the CREATING row could not be removed (its owner heartbeated after the wait, or its \
    +             status changed); its objects are left with it"
    +                .to_owned(),
    +        );
    +    }
    +    // The status to finish from: the row's as read, or as another deleter
    +    // left it.
    +    let mut status = row.status;
    +    if status == BackupStatus::Available {
    +        match storage.start_backup_delete(&key).await {
    +            Ok(DeleteStart::Deleting | DeleteStart::Status(BackupStatus::Deleting)) => {
    +                status = BackupStatus::Deleting;
    +            }
    +            Ok(DeleteStart::Status(BackupStatus::Deleted)) => status = BackupStatus::Deleted,
    +            Ok(DeleteStart::InUse(table)) => {
    +                return Err(format!("a restore into {table} is reading it"));
    +            }
    +            Ok(DeleteStart::Status(other)) => {
    +                return Err(format!("its status changed to {other}"));
    +            }
    +            Err(e) if is_backup_not_found(&e) => return Ok(()),
    +            Err(e) => return Err(format!("cannot mark the row DELETING: {e}")),
    +        }
    +    }
    +    let prefix = arn.store_prefix();
    +    store
    +        .delete_prefix(&format!("{prefix}{EXTENDDB_MANIFEST_FILE}"))
    +        .await
    +        .map_err(|e| format!("cannot delete the manifest: {e}"))?;
    +    store
    +        .delete_prefix(&prefix)
    +        .await
    +        .map_err(|e| format!("cannot delete the prefix: {e}"))?;
    +    if let Some(left) = store
    +        .list(&prefix)
    +        .try_next()
    +        .await
    +        .map_err(|e| format!("cannot list the prefix after the deletion: {e}"))?
    +    {
    +        return Err(format!(
    +            "the prefix still lists {} after the deletion",
    +            left.key
    +        ));
    +    }
    +    if status == BackupStatus::Creating {
    +        return Ok(());
    +    }
    +    if status == BackupStatus::Deleting {
    +        storage
    +            .set_backup_deleted(&key)
    +            .await
    +            .map_err(|e| format!("cannot mark the row DELETED: {e}"))?;
    +    }
    +    if storage
    +        .remove_backup_row(&key)
    +        .await
    +        .map_err(|e| format!("cannot remove the row: {e}"))?
    +    {
    +        return Ok(());
    +    }
    +    // Not a tombstone: another deleter removed it, or a writer moved it.
    +    match storage
    +        .read_backup_row(&arn)
    +        .await
    +        .map_err(|e| format!("cannot re-read the row: {e}"))?
    +    {
    +        None => Ok(()),
    +        Some(current) => Err(format!(
    +            "the row could not be removed (it is {} now, not a tombstone)",
    +            current.status
    +        )),
    +    }
    +}
    +
    +#[cfg(test)]
    +mod tests {
    +    use extenddb_storage::error::StorageError;
    +    use futures::future::BoxFuture;
    +    use tokio::time::Instant;
    +
    +    use super::*;
    +
    +    /// A claim whose heartbeats are timestamps on tokio's clock, with the age
    +    /// rule every backend's `check_maintenance_claim` applies.
    +    #[derive(Default)]
    +    struct FakeClaim {
    +        beats: Mutex>,
    +        lost: Mutex,
    +        released: Mutex,
    +    }
    +
    +    impl FakeClaim {
    +        /// `check_maintenance_claim(stale)`: held while the last heartbeat is
    +        /// within `stale`.
    +        fn held(&self, stale: Duration) -> bool {
    +            self.beats
    +                .lock()
    +                .unwrap()
    +                .last()
    +                .is_some_and(|at| at.elapsed() < stale)
    +        }
    +        fn beats(&self) -> usize {
    +            self.beats.lock().unwrap().len()
    +        }
    +    }
    +
    +    impl MaintenanceClaim for FakeClaim {
    +        fn generation(&self) -> u64 {
    +            7
    +        }
    +        fn heartbeat(&self) -> BoxFuture<'_, Result> {
    +            let lost = *self.lost.lock().unwrap();
    +            if !lost {
    +                self.beats.lock().unwrap().push(Instant::now());
    +            }
    +            Box::pin(async move {
    +                Ok(if lost {
    +                    Heartbeat::Lost
    +                } else {
    +                    Heartbeat::Confirmed
    +                })
    +            })
    +        }
    +        fn release(&self) -> BoxFuture<'_, Result<(), StorageError>> {
    +            *self.released.lock().unwrap() = true;
    +            Box::pin(async { Ok(()) })
    +        }
    +    }
    +
    +    /// The claim object handed to `HeldClaim`, forwarding to the test's own.
    +    struct ForwardingClaim(Arc);
    +
    +    impl MaintenanceClaim for ForwardingClaim {
    +        fn generation(&self) -> u64 {
    +            self.0.generation()
    +        }
    +        fn heartbeat(&self) -> BoxFuture<'_, Result> {
    +            self.0.heartbeat()
    +        }
    +        fn release(&self) -> BoxFuture<'_, Result<(), StorageError>> {
    +            self.0.release()
    +        }
    +    }
    +
    +    fn held_over(fake: &Arc, interval: Duration) -> HeldClaim {
    +        fake.beats.lock().unwrap().push(Instant::now());
    +        HeldClaim::start(Box::new(ForwardingClaim(Arc::clone(fake))), interval)
    +    }
    +
    +    /// On a paused clock (the timer and the samples share it, so a stall of
    +    /// the test process cannot read as a missed heartbeat): a held claim
    +    /// stays "held" to the jobs' check through a run of thirty stale
    +    /// windows with no heartbeat from the holder's own loop, a lost
    +    /// heartbeat is reported, and the release stops the timer.
    +    #[tokio::test(start_paused = true)]
    +    async fn the_heartbeat_timer_keeps_the_claim_held_through_a_long_run() {
    +        let fake = Arc::new(FakeClaim::default());
    +        let stale = Duration::from_secs(3);
    +        let held = held_over(&fake, Duration::from_secs(1));
    +        assert_eq!(held.generation(), 7);
    +        let started = Instant::now();
    +        let mut samples = 0;
    +        while started.elapsed() < Duration::from_secs(90) {
    +            assert!(
    +                fake.held(stale),
    +                "the claim read as free {:?} after the take",
    +                started.elapsed()
    +            );
    +            assert_eq!(held.lost(), None);
    +            samples += 1;
    +            tokio::time::sleep(Duration::from_millis(500)).await;
    +        }
    +        assert!(samples >= 150, "{samples}");
    +        assert!(
    +            fake.beats() >= 85,
    +            "the timer heartbeated: {}",
    +            fake.beats()
    +        );
    +        // Taken over: the next heartbeat finds the claim lost.
    +        *fake.lost.lock().unwrap() = true;
    +        tokio::time::sleep(Duration::from_millis(2_500)).await;
    +        assert!(
    +            held.lost().is_some_and(|reason| reason.contains("lost")),
    +            "{:?}",
    +            held.lost()
    +        );
    +        let beats = fake.beats();
    +        held.release().await.unwrap();
    +        assert!(*fake.released.lock().unwrap());
    +        tokio::time::sleep(Duration::from_secs(5)).await;
    +        assert_eq!(fake.beats(), beats, "no heartbeat after the release");
    +    }
    +
    +    /// A handle dropped without `release` stops its timer: the claim gets no
    +    /// further heartbeat and expires by the stale rule, as a dead holder's
    +    /// does, rather than being kept alive by a detached task.
    +    #[tokio::test(start_paused = true)]
    +    async fn a_dropped_handle_stops_the_heartbeat() {
    +        let fake = Arc::new(FakeClaim::default());
    +        let held = held_over(&fake, Duration::from_secs(1));
    +        tokio::time::sleep(Duration::from_millis(3_500)).await;
    +        let before = fake.beats();
    +        assert!(before >= 3, "{before}");
    +        drop(held);
    +        tokio::time::sleep(Duration::from_secs(10)).await;
    +        assert_eq!(fake.beats(), before, "a dropped handle heartbeats no more");
    +        assert!(!*fake.released.lock().unwrap(), "a drop is not a release");
    +        assert!(
    +            !fake.held(Duration::from_secs(3)),
    +            "the claim expires by the stale rule"
    +        );
    +    }
    +}
    diff --git a/crates/engine/src/backup/reconciler.rs b/crates/engine/src/backup/reconciler.rs
    index a9c649714..14d48c3bc 100644
    --- a/crates/engine/src/backup/reconciler.rs
    +++ b/crates/engine/src/backup/reconciler.rs
    @@ -2,14 +2,16 @@
     // SPDX-License-Identifier: Apache-2.0
     
     //! The reconciler: the periodic worker that resolves catalog rows whose
    -//! owner is dead.
    +//! owner is dead, and keeps the configured store's health current.
     //!
     //! One cycle runs every `interval`. It first checks the catalog-level
     //! maintenance claim; while a maintenance command holds it the reconciler
    -//! runs only the duties that resolve the states the command waits on, which
    -//! today are every duty it has, so the check is what decides when the other
    -//! duties are added. The duties, over the records `list_dead_restore_jobs`
    -//! returns:
    +//! runs only the duties below, which resolve the states the command waits
    +//! on or finishes itself (the dead-owner duties, the store's check and the
    +//! DELETING rows); the duties a later release adds that delete or move a
    +//! row the command does not wait on (a FAILED row's prefix, tombstone
    +//! retention) go in the branch the flag skips. The duties, over the records
    +//! `list_dead_restore_jobs` returns:
     //!
     //! - A RUNNING restore job record whose `heartbeat_at` is older than
     //!   `stale_job_after` (on the catalog's clock) has a dead owner. One of kind
    @@ -27,6 +29,28 @@
     //! - An in-catalog CREATING backup row whose create is dead (reported by the
     //!   backend's `abandoned_creating_backups`) is removed through the reader's
     //!   `delete`, which takes its items with it.
    +//! - With a store configured: the store's check runs again
    +//!   ([`super::check_configured_store`]: `validate` while it has not passed,
    +//!   then the identity check and manifest probe) and sets the store's
    +//!   health, so a store whose checks run after the listener is up, or that
    +//!   failed at startup or later, is healthy once a check passes, and one
    +//!   that fails is refused from the next cycle on. The check runs after the
    +//!   duties above and under the store's bound
    +//!   ([`super::ConfiguredStore::check_timeout`]: 5 s for a filesystem root,
    +//!   30 s for a bucket), so a store that has stopped answering costs each
    +//!   cycle that bound and nothing more: the dead-owner duties run every
    +//!   `interval` plus the bound, the check is recorded as a transport
    +//!   failure, and a filesystem check leaves one blocking thread parked on
    +//!   the mount until it answers, after which no check starts until that
    +//!   call returns (the runtime module's in-flight rule), so a mount that
    +//!   stays stalled parks one thread and not one per cycle. The check is
    +//!   also selected against the shutdown token, so a cycle that is checking
    +//!   when the server stops returns at once and the drain finishes.
    +//!   While the store is healthy, each DELETING row of the configured store
    +//!   (a DeleteBackup whose deletion did not finish) is finished
    +//!   (`finish_backup_delete` in the engine's backup module): the manifest, then the prefix, then
    +//!   the row to DELETED. While it is unhealthy the reconciler acts on no row
    +//!   of the store.
     //!
     //! A cycle that fails logs and waits for the next; each cycle records the
     //! worker metrics the other workers record (`WorkerLastSuccess` and
    @@ -37,10 +61,16 @@ use std::collections::HashSet;
     use std::sync::Arc;
     use std::time::Duration;
     
    +use extenddb_core::backup_format::BackupArn;
     use extenddb_core::metrics::{MetricsCollector, QuerySource};
    -use extenddb_storage::backup_catalog::{JobStatus, RestoreJobRow, RestoreKind, RestoreTargetKey};
    +use extenddb_storage::backup_catalog::{
    +    BackupStatus, JobStatus, RestoreJobRow, RestoreKind, RestoreTargetKey,
    +};
     use extenddb_storage::error::StorageError;
     use extenddb_storage::{BackupCatalog, CancellationToken, LegacyBackupReader, sleep_or_shutdown};
    +use futures::TryStreamExt;
    +
    +use super::BackupRuntime;
     
     /// The two durations a reconciler reads from `[backup]`.
     #[derive(Debug, Clone, Copy, PartialEq, Eq)]
    @@ -75,31 +105,62 @@ pub struct CycleReport {
         pub manifest_restores_logged: Vec,
         /// In-catalog CREATING backup rows removed, by ARN.
         pub legacy_backups_removed: Vec,
    +    /// The configured store's health after this cycle's check; `None` with
    +    /// no store configured.
    +    pub store_healthy: Option,
    +    /// DELETING rows of the configured store moved to DELETED, by ARN.
    +    pub deletions_finished: Vec,
     }
     
     /// What a reconciler carries from one cycle to the next: the manifest
     /// records it has already logged, so a record it cannot act on is reported
    -/// once rather than every cycle.
    +/// once rather than every cycle, and the worker's shutdown token, which the
    +/// store's check is selected against.
     #[derive(Debug, Default)]
     pub struct ReconcilerState {
         logged_manifest_restores: HashSet,
    +    shutdown: Option,
    +}
    +
    +impl ReconcilerState {
    +    /// A state whose cycles stop waiting on the store's check when `shutdown`
    +    /// fires (the health is then left as the last check left it). A default
    +    /// state, which commands and tests use, waits the check's bound.
    +    #[must_use]
    +    pub fn with_shutdown(shutdown: CancellationToken) -> Self {
    +        Self {
    +            logged_manifest_restores: HashSet::new(),
    +            shutdown: Some(shutdown),
    +        }
    +    }
     }
     
     /// Run cycles every `settings.interval` until `shutdown` fires. One cycle
     /// runs at once at startup, so a restart resolves what the stopped process
    -/// left without waiting an interval.
    +/// left without waiting an interval. `backend` names the storage backend in
    +/// an identity file this worker has to rewrite.
     pub async fn run(
         storage: Arc,
    +    runtime: Arc,
         settings: ReconcilerSettings,
    +    backend: String,
         metrics: Arc,
         shutdown: CancellationToken,
     ) where
         S: BackupCatalog + LegacyBackupReader + ?Sized,
     {
    -    let mut state = ReconcilerState::default();
    +    let mut state = ReconcilerState::with_shutdown(shutdown.clone());
         loop {
             let started = std::time::Instant::now();
    -        match cycle(&*storage, settings.stale_job_after, &mut state).await {
    +        match cycle(
    +            &*storage,
    +            &runtime,
    +            settings.stale_job_after,
    +            &backend,
    +            &mut state,
    +        )
    +        .await
    +        {
                 Ok(report) => {
                     #[allow(clippy::cast_precision_loss)]
                     let cycle_us = started.elapsed().as_micros() as f64;
    @@ -107,12 +168,14 @@ pub async fn run(
                     if !report.legacy_restores_removed.is_empty()
                         || !report.failed_targets_removed.is_empty()
                         || !report.legacy_backups_removed.is_empty()
    +                    || !report.deletions_finished.is_empty()
                     {
                         tracing::info!(
                             "backup reconciler: removed {} abandoned restore target(s) and {} \
    -                         abandoned in-catalog backup(s)",
    +                         abandoned in-catalog backup(s), finished {} deletion(s)",
                             report.legacy_restores_removed.len() + report.failed_targets_removed.len(),
    -                        report.legacy_backups_removed.len()
    +                        report.legacy_backups_removed.len(),
    +                        report.deletions_finished.len()
                         );
                     }
                 }
    @@ -135,20 +198,22 @@ pub async fn run(
     /// logged and the cycle goes on to the next row.
     pub async fn cycle(
         storage: &S,
    +    runtime: &BackupRuntime,
         stale_job_after: Duration,
    +    backend: &str,
         state: &mut ReconcilerState,
     ) -> Result
     where
         S: BackupCatalog + LegacyBackupReader + ?Sized,
     {
         let mut report = CycleReport {
    -        maintenance_held: storage.check_maintenance_claim().await?,
    +        maintenance_held: storage.check_maintenance_claim(stale_job_after).await?,
             ..CycleReport::default()
         };
         if report.maintenance_held {
             tracing::info!(
                 "backup reconciler: a maintenance claim is held; running only the duties that \
    -             resolve rows whose owner is dead"
    +             resolve rows whose owner is dead, the store's check and the DELETING rows"
             );
         }
     
    @@ -220,9 +285,92 @@ where
                 ),
             }
         }
    +
    +    if let Some(store) = runtime.configured_store() {
    +        let check = super::check_configured_store(store, storage, backend);
    +        let outcome = match &state.shutdown {
    +            Some(shutdown) => tokio::select! {
    +                outcome = check => Some(outcome),
    +                () = shutdown.cancelled() => None,
    +            },
    +            None => Some(check.await),
    +        };
    +        match outcome {
    +            Some(outcome) => {
    +                store.record_check(outcome);
    +            }
    +            None => {
    +                tracing::info!(
    +                    "backup store ({}) check was stopped by shutdown; its health is left as the \
    +                     last check left it",
    +                    store.kind
    +                );
    +                report.store_healthy = Some(store.is_healthy());
    +                return Ok(report);
    +            }
    +        }
    +        report.store_healthy = Some(store.is_healthy());
    +        if let Some(identity) = store.identity().filter(|_| store.is_healthy()) {
    +            report.deletions_finished = finish_deleting_rows(storage, store, &identity).await?;
    +        }
    +    }
    +
    +    if !report.maintenance_held {
    +        // The duties a maintenance command does not wait on and must not
    +        // race: a later release's deletion of a FAILED row's prefix and its
    +        // removal of tombstones past their retention run here.
    +    }
         Ok(report)
     }
     
    +/// The duty for DELETING rows of the configured store: finish each deletion.
    +/// Returns the ARNs moved to DELETED; a row whose deletion fails is logged
    +/// and left for the next cycle.
    +async fn finish_deleting_rows(
    +    storage: &S,
    +    store: &super::ConfiguredStore,
    +    identity: &extenddb_storage::backup_catalog::StoreIdentity,
    +) -> Result, StorageError> {
    +    let rows: Vec<_> = storage
    +        .list_backup_rows(Some(BackupStatus::Deleting), identity)
    +        .try_collect()
    +        .await?;
    +    let mut finished = Vec::new();
    +    for row in rows {
    +        let arn = match BackupArn::parse(&row.backup_arn) {
    +            Ok(arn) => arn,
    +            Err(e) => {
    +                tracing::error!(
    +                    "DELETING row {} carries an ARN this build cannot parse; left as it is: {e}",
    +                    row.backup_arn
    +                );
    +                continue;
    +            }
    +        };
    +        match super::finish_backup_delete(storage, &*store.store, &arn).await {
    +            Ok(true) => {
    +                tracing::info!(
    +                    "finished the deletion of backup {} (account {})",
    +                    row.backup_arn,
    +                    row.account_id
    +                );
    +                finished.push(row.backup_arn);
    +            }
    +            Ok(false) => tracing::info!(
    +                "backup {} was no longer DELETING when its objects were gone; left to its owner",
    +                row.backup_arn
    +            ),
    +            Err(e) => tracing::error!(
    +                "could not finish the deletion of backup {} (account {}); the next cycle \
    +                 retries: {e}",
    +                row.backup_arn,
    +                row.account_id
    +            ),
    +        }
    +    }
    +    Ok(finished)
    +}
    +
     /// Mark a dead legacy restore FAILED and remove its target. `None` when the
     /// FAILED write did not match (the record is no longer RUNNING and belongs to
     /// whoever moved it) or errored; otherwise the removal's outcome.
    @@ -295,8 +443,8 @@ async fn remove_restore_target(
                     "the abandoned restore into {} (account {}) {how} but its target ({}) was no \
                      longer a CREATING restore target; left to its current owner",
                     target.table_name,
    -                target.account_id,
    -                target.table_id
    +                target.table_id,
    +                target.account_id
                 );
                 Removal::LeftToOwner
             }
    diff --git a/crates/engine/src/backup/runtime.rs b/crates/engine/src/backup/runtime.rs
    index c4f816bbb..0301caa07 100644
    --- a/crates/engine/src/backup/runtime.rs
    +++ b/crates/engine/src/backup/runtime.rs
    @@ -3,29 +3,605 @@
     
     //! The per-process state of the backup operations: the job slots every
     //! backup and restore job holds one of, and the store this instance is
    -//! configured to write.
    +//! configured to write, with its health.
    +//!
    +//! The store's health is a state the checks set ([`super::store_identity`]
    +//! runs them; [`check_configured_store`] is the one the reconciler calls each
    +//! cycle): `Checking` until the first check of a store whose checks run after
    +//! the listener is up (an S3 store; the server binds and serves first, so an
    +//! endpoint that stalls never holds the data plane), `Healthy`, or
    +//! `Unhealthy` with the problem. A transition between the three is logged
    +//! once; while checking or unhealthy, the operations that touch the store
    +//! refuse with a `ValidationException` that names the store's kind and not
    +//! its root ([`BackupRuntime::require_healthy_store`]), `/health` reports the
    +//! `backup_store` component `checking` or `unhealthy`
    +//! ([`BackupRuntime::store_health`]), and the reconciler acts on no row of
    +//! the store. A check that cannot read the catalog changes nothing: the
    +//! condition is the catalog's, not the store's, and the next cycle reads
    +//! again.
    +//!
    +//! A check that does not finish within its bound is abandoned. An S3 store's
    +//! stalled request ends with the dropped future, before the SDK retries it,
    +//! and the next cycle sends a fresh one. A filesystem store's stalled call is
    +//! a blocking thread parked in a filesystem call until the mount answers,
    +//! which dropping the future cannot end, so a filesystem root's calls run as
    +//! tasks of their own that the abandoned check does not take with it, and
    +//! the store remembers that a call is in flight
    +//! ([`ConfiguredStore::check_call_in_flight`]): the next
    +//! [`check_configured_store`] starts no check while one is, recording the
    +//! same `unreachable` the abandoned one did, so a mount that stays stalled
    +//! parks one thread, not one per cycle. The thread costs nothing at exit:
    +//! the binary ends its runtime with a bounded shutdown that detaches it.
     
    -use std::sync::Arc;
    +use std::sync::atomic::{AtomicBool, Ordering};
    +use std::sync::{Arc, RwLock};
     
    +use extenddb_backup_store::{BackupStore, ByteStream, ObjectMeta, StoreError};
     use extenddb_core::error::DynamoDbError;
    -use extenddb_storage::JobPermit;
    -use extenddb_storage::backup_catalog::{JobBound, StoreIdentity, limit_exceeded_message};
    +use extenddb_storage::backup_catalog::{
    +    JobBound, StoreIdentity, StoreKind, limit_exceeded_message,
    +};
    +use extenddb_storage::{BackupCatalog, JobPermit};
    +use futures::StreamExt;
    +use futures::future::BoxFuture;
    +use futures::stream::BoxStream;
     use tokio::sync::Semaphore;
     
    +use super::store_identity::{self, StoreProblem};
    +
     /// The configuration key of the job bound the runtime enforces, as the
     /// refusal names it.
     const MAX_CONCURRENT_JOBS_KEY: &str = "[backup] max_concurrent_jobs";
     
    +/// How long a filesystem store's check may run, at startup and in a
    +/// reconciler cycle, before the store is treated as not answering: a local
    +/// disk answers in milliseconds, and a network mount that takes longer is
    +/// one that has stopped answering.
    +pub const LOCAL_CHECK_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(5);
    +
    +/// How long an S3 store's check may run in a reconciler cycle. The check
    +/// sends a few small requests (`validate`'s listing, the identity file's
    +/// `GetObject`, the manifest's `HeadObject`), each answered in well under a
    +/// second by a store that answers at all; the bound sits under the SDK's
    +/// per-attempt timeout ([`extenddb_backup_store::S3_ATTEMPT_TIMEOUT`], 60 s)
    +/// so a timed-out check is dropped before the SDK starts a retry of its
    +/// stalled request, and no retry outlives the check. A stalled endpoint thus
    +/// costs a cycle this bound, not the SDK's whole retry budget.
    +pub const S3_CHECK_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(30);
    +const _: () = assert!(
    +    S3_CHECK_TIMEOUT.as_secs() < extenddb_backup_store::S3_ATTEMPT_TIMEOUT.as_secs(),
    +    "the S3 check bound must fire before the SDK's attempt timeout"
    +);
    +
     /// One per server process, shared by every request through the operation
     /// context.
    -#[derive(Debug)]
     pub struct BackupRuntime {
         slots: Arc,
         max_concurrent_jobs: usize,
         /// The store this instance writes new backups to and deletes from;
         /// `None` while no store is configured, when every catalog row is a row
         /// of another store.
    -    configured_store: Option,
    +    configured_store: Option,
    +    /// Whether CreateBackup takes the in-catalog path whatever the store
    +    /// configuration (a backend that cannot pin a table snapshot); such a
    +    /// create is not refused by an unhealthy store.
    +    legacy_create_only: bool,
    +    /// `[backup] stale_job_after_secs`: the window a job's claim check reads
    +    /// the maintenance claim with.
    +    stale_job_after: std::time::Duration,
    +}
    +
    +impl std::fmt::Debug for BackupRuntime {
    +    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
    +        f.debug_struct("BackupRuntime")
    +            .field("max_concurrent_jobs", &self.max_concurrent_jobs)
    +            .field(
    +                "configured_store",
    +                &self
    +                    .configured_store
    +                    .as_ref()
    +                    .map(ConfiguredStore::identity),
    +            )
    +            .field("legacy_create_only", &self.legacy_create_only)
    +            .finish_non_exhaustive()
    +    }
    +}
    +
    +/// The store this instance is configured with: its kind and root (the
    +/// configuration's), its id once the identity check has read it, the handle,
    +/// the spool for restores from it, and its health.
    +///
    +/// The id is `None` until a check passes: a store whose startup check failed
    +/// (a transport failure, say) has an unknown id, and a row of its kind and
    +/// root is still a row of this store for the refusals, so those rows answer
    +/// "the store is unhealthy" rather than "another store".
    +pub struct ConfiguredStore {
    +    /// `filesystem` or `s3`.
    +    pub kind: StoreKind,
    +    /// The root label rows record (`store_root`).
    +    pub root: String,
    +    /// The opened store.
    +    pub store: Arc,
    +    /// The restore spool for an S3 store: the directory and the
    +    /// `:` label a restore job record names.
    +    pub spool: Option,
    +    store_id: RwLock>,
    +    health: RwLock,
    +    /// Whether the store's `validate` has passed; the checks run it first
    +    /// until it has.
    +    validated: AtomicBool,
    +    /// Whether a call of a check on a filesystem root is still running
    +    /// (module doc): set when the call starts, cleared by the call's own task
    +    /// when the mount answers, so a check abandoned at its bound leaves it
    +    /// set until then. Never set on an S3 store.
    +    check_call_in_flight: Arc,
    +}
    +
    +/// The restore spool of an S3 store.
    +#[derive(Debug, Clone, PartialEq, Eq)]
    +pub struct Spool {
    +    pub path: std::path::PathBuf,
    +    /// `:`, the host whose spool holds a restore's
    +    /// downloaded files.
    +    pub spool_host: String,
    +}
    +
    +/// The store's health as the last check left it.
    +#[derive(Debug, Clone, PartialEq, Eq)]
    +pub enum StoreHealth {
    +    /// No check has finished yet: the store's checks run after the listener
    +    /// is up and the first has not returned.
    +    Checking,
    +    Healthy,
    +    Unhealthy(StoreProblem),
    +}
    +
    +impl StoreHealth {
    +    /// The one or two words the API refusal and `/health` name: the
    +    /// problem's category ([`StoreProblem::category`]), or `check pending`.
    +    #[must_use]
    +    pub fn category(&self) -> Option<&'static str> {
    +        match self {
    +            Self::Healthy => None,
    +            Self::Checking => Some("check pending"),
    +            Self::Unhealthy(problem) => Some(problem.category()),
    +        }
    +    }
    +}
    +
    +/// What `/health` reports for the `backup_store` component.
    +#[derive(Debug, Clone, PartialEq, Eq)]
    +pub struct StoreHealthReport {
    +    /// `filesystem` or `s3`.
    +    pub kind: &'static str,
    +    /// `healthy`, `unhealthy` or `checking`.
    +    pub status: &'static str,
    +    /// The problem's category and nothing else: `/health` is unauthenticated,
    +    /// and the ids, ARNs and catalog id the operator's log line carries stay
    +    /// out of it. `None` when healthy or checking.
    +    pub reason: Option<&'static str>,
    +}
    +
    +impl ConfiguredStore {
    +    /// A store of `kind` at `root` whose startup check (`validate` passed,
    +    /// then `establish`) ended in `outcome`: the identity it settled
    +    /// (healthy), or the problem that makes the store unhealthy from the
    +    /// start (a transport failure, or a failed identity check or probe:
    +    /// section 7's rule that a store problem never stops the data plane). A
    +    /// problem leaves the id unknown until a later check passes.
    +    #[must_use]
    +    pub fn new(
    +        kind: StoreKind,
    +        root: String,
    +        store: Arc,
    +        spool: Option,
    +        outcome: Result,
    +    ) -> Self {
    +        let (store_id, health) = match outcome {
    +            Ok(identity) => (Some(identity.store_id), StoreHealth::Healthy),
    +            Err(problem) => (None, StoreHealth::Unhealthy(problem)),
    +        };
    +        Self {
    +            kind,
    +            root,
    +            store,
    +            spool,
    +            store_id: RwLock::new(store_id),
    +            health: RwLock::new(health),
    +            validated: AtomicBool::new(true),
    +            check_call_in_flight: Arc::new(AtomicBool::new(false)),
    +        }
    +    }
    +
    +    /// A store of `kind` at `root` that `open` accepted and whose checks
    +    /// (`validate`, then the identity check) have not run: the first
    +    /// [`check_configured_store`] runs both and records the result. Until
    +    /// then the store is [`StoreHealth::Checking`].
    +    #[must_use]
    +    pub fn checking(
    +        kind: StoreKind,
    +        root: String,
    +        store: Arc,
    +        spool: Option,
    +    ) -> Self {
    +        Self {
    +            kind,
    +            root,
    +            store,
    +            spool,
    +            store_id: RwLock::new(None),
    +            health: RwLock::new(StoreHealth::Checking),
    +            validated: AtomicBool::new(false),
    +            check_call_in_flight: Arc::new(AtomicBool::new(false)),
    +        }
    +    }
    +
    +    /// The store's identity, once a check has read its id.
    +    #[must_use]
    +    pub fn identity(&self) -> Option {
    +        let id = self
    +            .store_id
    +            .read()
    +            .unwrap_or_else(std::sync::PoisonError::into_inner)
    +            .clone()?;
    +        Some(StoreIdentity {
    +            store_kind: self.kind,
    +            store_root: self.root.clone(),
    +            store_id: id,
    +        })
    +    }
    +
    +    /// Whether a row's store is this one: the kind and the root, and the id
    +    /// when this store's is known.
    +    #[must_use]
    +    pub fn is_store_of(&self, row_store: &StoreIdentity) -> bool {
    +        if row_store.store_kind != self.kind || row_store.store_root != self.root {
    +            return false;
    +        }
    +        match self
    +            .store_id
    +            .read()
    +            .unwrap_or_else(std::sync::PoisonError::into_inner)
    +            .as_deref()
    +        {
    +            Some(id) => id == row_store.store_id,
    +            None => true,
    +        }
    +    }
    +
    +    /// The health as the last check left it.
    +    #[must_use]
    +    pub fn health(&self) -> StoreHealth {
    +        self.health
    +            .read()
    +            .unwrap_or_else(std::sync::PoisonError::into_inner)
    +            .clone()
    +    }
    +
    +    /// Whether the last check passed.
    +    #[must_use]
    +    pub fn is_healthy(&self) -> bool {
    +        matches!(self.health(), StoreHealth::Healthy)
    +    }
    +
    +    /// Record the outcome of a check: the identity it settled (the id is
    +    /// kept), or the problem. A transition is logged once: to unhealthy at
    +    /// `error` with the problem's detail, to healthy at `info`; a check
    +    /// whose outcome is in the last one's category logs nothing, so a store
    +    /// that stays down costs one line however its error text varies between
    +    /// cycles. A check that could not read the catalog
    +    /// ([`StoreProblem::Catalog`]) leaves the health as it was and logs the
    +    /// read failure: the condition is not the store's. Returns whether this
    +    /// check changed the health's category (and so logged a transition).
    +    pub fn record_check(&self, outcome: Result) -> bool {
    +        let next = match outcome {
    +            Ok(identity) => {
    +                *self
    +                    .store_id
    +                    .write()
    +                    .unwrap_or_else(std::sync::PoisonError::into_inner) = Some(identity.store_id);
    +                StoreHealth::Healthy
    +            }
    +            Err(StoreProblem::Catalog(e)) => {
    +                tracing::warn!(
    +                    "backup store ({}) check could not read the catalog ({e}); the store's \
    +                     health is left as it was and the next cycle reads again",
    +                    self.kind
    +                );
    +                return false;
    +            }
    +            Err(problem) => StoreHealth::Unhealthy(problem),
    +        };
    +        let mut health = self
    +            .health
    +            .write()
    +            .unwrap_or_else(std::sync::PoisonError::into_inner);
    +        if health.category() == next.category() {
    +            *health = next;
    +            return false;
    +        }
    +        match (&*health, &next) {
    +            (StoreHealth::Checking, StoreHealth::Healthy) => tracing::info!(
    +                "backup store ({}) {} passed its check (store id {}); it serves restores and \
    +                 deletes of the backups it holds",
    +                self.kind,
    +                self.root,
    +                self.identity().map(|i| i.store_id).unwrap_or_default()
    +            ),
    +            (_, StoreHealth::Healthy) => {
    +                tracing::info!("backup store ({}) is healthy again", self.kind);
    +            }
    +            (_, StoreHealth::Unhealthy(problem)) => tracing::error!(
    +                "backup store ({}) {} is unhealthy: {problem}; restore and delete of a backup \
    +                 in the store are refused until the check passes; the check runs again each \
    +                 reconciler cycle",
    +                self.kind,
    +                self.root
    +            ),
    +            (_, StoreHealth::Checking) => {}
    +        }
    +        *health = next;
    +        true
    +    }
    +
    +    /// Whether the store's `validate` has passed.
    +    #[must_use]
    +    pub fn is_validated(&self) -> bool {
    +        self.validated.load(Ordering::Acquire)
    +    }
    +
    +    /// How long one check of this store may run before it is abandoned
    +    /// ([`LOCAL_CHECK_TIMEOUT`] for a filesystem root, [`S3_CHECK_TIMEOUT`]
    +    /// for a bucket).
    +    #[must_use]
    +    pub fn check_timeout(&self) -> std::time::Duration {
    +        match self.kind {
    +            StoreKind::Filesystem => LOCAL_CHECK_TIMEOUT,
    +            StoreKind::S3 => S3_CHECK_TIMEOUT,
    +        }
    +    }
    +
    +    /// Whether a call of an earlier check has not returned (module doc).
    +    #[must_use]
    +    pub fn check_call_in_flight(&self) -> bool {
    +        self.check_call_in_flight.load(Ordering::Acquire)
    +    }
    +}
    +
    +/// The problem a check, or a command's identity read, that did not finish
    +/// within `bound` is recorded as: a transport failure naming the bound.
    +fn did_not_finish(what: &str, bound: std::time::Duration) -> StoreProblem {
    +    StoreProblem::Transport(format!(
    +        "the {what} did not finish within {} s",
    +        bound.as_secs()
    +    ))
    +}
    +
    +/// The check of the configured store a reconciler cycle, and a command that
    +/// needs the store's identity, run and record through
    +/// [`ConfiguredStore::record_check`]: `validate` first while it has not
    +/// passed (its failure is the problem [`StoreProblem::from`] makes of the
    +/// store's error, a transport failure [`StoreProblem::Transport`] with the
    +/// cause chain, a denial [`StoreProblem::Denied`], anything else
    +/// [`StoreProblem::Validate`]; each is retried next cycle), then the
    +/// identity check, which is [`store_identity::recheck`] for a store whose
    +/// id is known and [`store_identity::establish`] for one whose checks have
    +/// not passed yet (it settles the id when it passes). `backend` names the
    +/// storage backend in an identity file the check has to write.
    +///
    +/// The whole check runs under [`ConfiguredStore::check_timeout`]: a store
    +/// that does not answer within it (a stalled endpoint, a mount that has
    +/// stopped answering) is [`StoreProblem::Transport`] naming the bound, and
    +/// the caller's cycle goes on. An S3 store's abandoned request is dropped
    +/// with the future. A filesystem root's calls run as tasks of their own
    +/// ([`CheckCalls`]), which the abandoned check does not take with it: the
    +/// call's synchronous work runs on the blocking pool, where the thread stays
    +/// parked until the mount answers, and the task is what clears the store's
    +/// in-flight mark when it does. While that mark is set no check starts: the
    +/// store is recorded `unreachable` again, with the reason, and nothing new
    +/// is parked.
    +///
    +/// # Errors
    +///
    +/// The problem that makes the store unhealthy
    +/// ([`store_identity`]'s module doc).
    +pub async fn check_configured_store(
    +    store: &ConfiguredStore,
    +    catalog: &C,
    +    backend: &str,
    +) -> Result {
    +    if store.check_call_in_flight() {
    +        tracing::info!(
    +            "backup store ({}) check was not started: a call of the check abandoned at an \
    +             earlier cycle has not returned (the store is still not answering); the next cycle \
    +             checks again",
    +            store.kind
    +        );
    +        return Err(StoreProblem::Transport(
    +            "a call of an earlier check has not returned".to_owned(),
    +        ));
    +    }
    +    let bound = store.check_timeout();
    +    match tokio::time::timeout(bound, check_unbounded(store, catalog, backend)).await {
    +        Ok(outcome) => outcome,
    +        Err(_) => {
    +            tracing::info!(
    +                "backup store ({}) check did not finish within {} s and was abandoned; the \
    +                 cycle's other duties ran, and the next cycle checks again",
    +                store.kind,
    +                bound.as_secs()
    +            );
    +            Err(did_not_finish("check", bound))
    +        }
    +    }
    +}
    +
    +/// [`check_configured_store`] without its bound.
    +async fn check_unbounded(
    +    store: &ConfiguredStore,
    +    catalog: &C,
    +    backend: &str,
    +) -> Result {
    +    // A filesystem root's calls run as tasks of their own (`CheckCalls`): a
    +    // stalled call is a parked thread, which dropping the future cannot
    +    // end. An S3 store's calls are the SDK's requests, which dropping the
    +    // future does end, before the SDK retries them (the bound sits under
    +    // its attempt timeout for that), so they run in the check's own future
    +    // and the next cycle sends a fresh request.
    +    let calls;
    +    let via: &dyn BackupStore = match store.kind {
    +        StoreKind::Filesystem => {
    +            calls = CheckCalls {
    +                store: &store.store,
    +                in_flight: &store.check_call_in_flight,
    +            };
    +            &calls
    +        }
    +        StoreKind::S3 => &*store.store,
    +    };
    +    if !store.is_validated() {
    +        match via.validate().await {
    +            Ok(()) => store.validated.store(true, Ordering::Release),
    +            Err(e) => return Err(StoreProblem::from(e)),
    +        }
    +    }
    +    match store.identity() {
    +        Some(identity) => store_identity::recheck(via, &identity, catalog, backend)
    +            .await
    +            .map(|()| identity),
    +        None => store_identity::establish(via, store.kind, &store.root, catalog, backend).await,
    +    }
    +}
    +
    +/// A command's read of the store's identity ([`store_identity::read_identity`],
    +/// which writes nothing), under the store's [`ConfiguredStore::check_timeout`]
    +/// as the server's check is: a read that does not finish within the bound is
    +/// [`StoreProblem::Transport`] naming it, where the store's own retries (the
    +/// SDK's three attempts of 60 s on an endpoint that accepts and never
    +/// answers) or a stalled mount would hold the command for minutes or for
    +/// good.
    +///
    +/// # Errors
    +///
    +/// The problem that would make the store unhealthy
    +/// ([`store_identity`]'s module doc), or the bound.
    +pub async fn read_identity_bounded(
    +    store: &ConfiguredStore,
    +    catalog: &C,
    +) -> Result, StoreProblem> {
    +    let bound = store.check_timeout();
    +    tokio::time::timeout(
    +        bound,
    +        store_identity::read_identity(&*store.store, store.kind, &store.root, catalog),
    +    )
    +    .await
    +    .unwrap_or_else(|_| Err(did_not_finish("identity read", bound)))
    +}
    +
    +/// A filesystem store as a check calls it (module doc): each call runs as a
    +/// task of its own, owning the store handle, so a check abandoned at its
    +/// bound leaves the call running until the mount answers, and the task
    +/// clears the store's in-flight mark when the call returns. A call that finds the mark
    +/// set (a call of an earlier check has not returned) is refused at once as
    +/// a transport failure, so no second call is parked beside the first. The
    +/// check calls `validate`, `get`, `head` and `put_if_absent`; `put`, `list`
    +/// and `delete_prefix` are forwarded for the trait's sake and marked the
    +/// same way.
    +struct CheckCalls<'a> {
    +    store: &'a Arc,
    +    in_flight: &'a Arc,
    +}
    +
    +/// Clears the in-flight mark when the call's task ends, by any exit.
    +struct ClearOnDrop(Arc);
    +
    +impl Drop for ClearOnDrop {
    +    fn drop(&mut self) {
    +        self.0.store(false, Ordering::Release);
    +    }
    +}
    +
    +impl CheckCalls<'_> {
    +    /// Run `call` on the store as a task of its own, marked in flight until
    +    /// it returns.
    +    fn run(&self, call: F) -> BoxFuture<'static, Result>
    +    where
    +        T: Send + 'static,
    +        F: FnOnce(Arc) -> BoxFuture<'static, Result>
    +            + Send
    +            + 'static,
    +    {
    +        if self.in_flight.swap(true, Ordering::AcqRel) {
    +            return Box::pin(std::future::ready(Err(StoreError::Transport {
    +                source: "a call of an earlier check has not returned".into(),
    +            })));
    +        }
    +        let clear = ClearOnDrop(Arc::clone(self.in_flight));
    +        let store = Arc::clone(self.store);
    +        let task = tokio::spawn(async move {
    +            let _clear = clear;
    +            call(store).await
    +        });
    +        Box::pin(async move {
    +            task.await
    +                .map_err(|e| StoreError::Other(format!("check call task failed: {e}")))?
    +        })
    +    }
    +}
    +
    +impl BackupStore for CheckCalls<'_> {
    +    fn put<'a>(
    +        &'a self,
    +        key: &'a str,
    +        body: ByteStream,
    +    ) -> BoxFuture<'a, Result> {
    +        let key = key.to_owned();
    +        self.run(move |store| Box::pin(async move { store.put(&key, body).await }))
    +    }
    +
    +    fn put_if_absent<'a>(
    +        &'a self,
    +        key: &'a str,
    +        body: ByteStream,
    +    ) -> BoxFuture<'a, Result, StoreError>> {
    +        let key = key.to_owned();
    +        self.run(move |store| Box::pin(async move { store.put_if_absent(&key, body).await }))
    +    }
    +
    +    fn get<'a>(&'a self, key: &'a str) -> BoxFuture<'a, Result> {
    +        let key = key.to_owned();
    +        self.run(move |store| Box::pin(async move { store.get(&key).await }))
    +    }
    +
    +    fn head<'a>(&'a self, key: &'a str) -> BoxFuture<'a, Result, StoreError>> {
    +        let key = key.to_owned();
    +        self.run(move |store| Box::pin(async move { store.head(&key).await }))
    +    }
    +
    +    fn list<'a>(&'a self, prefix: &'a str) -> BoxStream<'a, Result> {
    +        let prefix = prefix.to_owned();
    +        let listed = self.run(move |store| {
    +            Box::pin(async move {
    +                use futures::TryStreamExt as _;
    +                store.list(&prefix).try_collect::>().await
    +            })
    +        });
    +        Box::pin(
    +            futures::stream::once(listed).flat_map(|listed| match listed {
    +                Ok(metas) => futures::stream::iter(metas.into_iter().map(Ok)).boxed(),
    +                Err(e) => futures::stream::once(async move { Err(e) }).boxed(),
    +            }),
    +        )
    +    }
    +
    +    fn delete_prefix<'a>(&'a self, prefix: &'a str) -> BoxFuture<'a, Result> {
    +        let prefix = prefix.to_owned();
    +        self.run(move |store| Box::pin(async move { store.delete_prefix(&prefix).await }))
    +    }
    +
    +    fn validate(&self) -> BoxFuture<'_, Result<(), StoreError>> {
    +        self.run(|store| Box::pin(async move { store.validate().await }))
    +    }
     }
     
     impl BackupRuntime {
    @@ -36,16 +612,41 @@ impl BackupRuntime {
                 slots: Arc::new(Semaphore::new(max_concurrent_jobs)),
                 max_concurrent_jobs,
                 configured_store: None,
    +            legacy_create_only: false,
    +            stale_job_after: std::time::Duration::from_secs(300),
             }
         }
     
    +    /// The same runtime with `[backup] stale_job_after_secs` set, the window
    +    /// the jobs' maintenance-claim check reads with (the default is the
    +    /// configuration's default, 300 s).
    +    #[must_use]
    +    pub fn with_stale_job_after(mut self, stale_job_after: std::time::Duration) -> Self {
    +        self.stale_job_after = stale_job_after;
    +        self
    +    }
    +
    +    /// `[backup] stale_job_after_secs`.
    +    #[must_use]
    +    pub fn stale_job_after(&self) -> std::time::Duration {
    +        self.stale_job_after
    +    }
    +
         /// The same runtime with `store` configured.
         #[must_use]
    -    pub fn with_store(mut self, store: StoreIdentity) -> Self {
    +    pub fn with_store(mut self, store: ConfiguredStore) -> Self {
             self.configured_store = Some(store);
             self
         }
     
    +    /// The same runtime with CreateBackup bound to the in-catalog path
    +    /// whatever the store configuration.
    +    #[must_use]
    +    pub fn with_legacy_create_only(mut self, legacy_create_only: bool) -> Self {
    +        self.legacy_create_only = legacy_create_only;
    +        self
    +    }
    +
         /// The slots jobs take permits from.
         #[must_use]
         pub fn slots(&self) -> &Arc {
    @@ -58,6 +659,51 @@ impl BackupRuntime {
             self.slots.available_permits()
         }
     
    +    /// The configured store, if any.
    +    #[must_use]
    +    pub fn configured_store(&self) -> Option<&ConfiguredStore> {
    +        self.configured_store.as_ref()
    +    }
    +
    +    /// The configured store, taken out of a runtime a command built only to
    +    /// reach it.
    +    #[must_use]
    +    pub fn into_configured_store(self) -> Option {
    +        self.configured_store
    +    }
    +
    +    /// Whether CreateBackup takes the in-catalog path whatever the store
    +    /// configuration.
    +    #[must_use]
    +    pub fn legacy_create_only(&self) -> bool {
    +        self.legacy_create_only
    +    }
    +
    +    /// What `/health` reports for the `backup_store` component; `None` when
    +    /// no store is configured.
    +    #[must_use]
    +    pub fn store_health(&self) -> Option {
    +        let store = self.configured_store.as_ref()?;
    +        let kind = store.kind.as_str();
    +        Some(match store.health() {
    +            StoreHealth::Healthy => StoreHealthReport {
    +                kind,
    +                status: "healthy",
    +                reason: None,
    +            },
    +            StoreHealth::Checking => StoreHealthReport {
    +                kind,
    +                status: "checking",
    +                reason: None,
    +            },
    +            StoreHealth::Unhealthy(problem) => StoreHealthReport {
    +                kind,
    +                status: "unhealthy",
    +                reason: Some(problem.category()),
    +            },
    +        })
    +    }
    +
         /// Take a job slot without waiting, or `LimitExceededException` when every
         /// slot is held: a queued job would hold a snapshot or a transaction
         /// while it waited. The text is the one every refusal over a job bound
    @@ -74,6 +720,35 @@ impl BackupRuntime {
         /// Whether `store` is the one this instance is configured with. With no
         /// store configured, no row's store is.
         pub(crate) fn is_configured_store(&self, store: &StoreIdentity) -> bool {
    -        self.configured_store.as_ref() == Some(store)
    +        self.configured_store
    +            .as_ref()
    +            .is_some_and(|configured| configured.is_store_of(store))
    +    }
    +
    +    /// The configured store, when its last check passed; otherwise (no
    +    /// check has finished, or the last one failed) the `ValidationException`
    +    /// the operations that touch the store answer, naming the store's kind
    +    /// and the problem's category and not its root (no API error names a
    +    /// root).
    +    pub(crate) fn require_healthy_store(&self) -> Result<&ConfiguredStore, DynamoDbError> {
    +        let store = self.configured_store.as_ref().ok_or_else(|| {
    +            DynamoDbError::ValidationException(
    +                "No backup store is configured on this server ([backup.store])".to_owned(),
    +            )
    +        })?;
    +        match store.health() {
    +            StoreHealth::Healthy => Ok(store),
    +            StoreHealth::Checking => Err(DynamoDbError::ValidationException(format!(
    +                "The {} backup store configured on this server has not passed a check yet (its \
    +                 first check is pending); the operation is refused until it does",
    +                store.kind
    +            ))),
    +            StoreHealth::Unhealthy(problem) => Err(DynamoDbError::ValidationException(format!(
    +                "The {} backup store configured on this server is unhealthy ({}); the operation \
    +                 is refused until the store's check passes",
    +                store.kind,
    +                problem.category()
    +            ))),
    +        }
         }
     }
    diff --git a/crates/engine/src/backup/store_identity.rs b/crates/engine/src/backup/store_identity.rs
    new file mode 100644
    index 000000000..a74f03c1c
    --- /dev/null
    +++ b/crates/engine/src/backup/store_identity.rs
    @@ -0,0 +1,665 @@
    +// Copyright 2026 ExtendDB contributors
    +// SPDX-License-Identifier: Apache-2.0
    +
    +//! The store identity file and the checks that decide a configured store's
    +//! health.
    +//!
    +//! A store's root carries `extenddb-store.json` ([`STORE_IDENTITY_KEY`]),
    +//! holding a random store id and the `catalog_id` of the catalog that owns
    +//! the root; every backup row records the store id in `store_id`. Server
    +//! startup ([`establish`]) and every reconciler cycle ([`recheck`]) read the
    +//! file and check it against the catalog. The store is unhealthy when:
    +//!
    +//! - the file names another catalog (a root another deployment owns, or a
    +//!   copy of one; an operator adopts it on purpose, never a server by
    +//!   accident);
    +//! - its store id differs from the `store_id` of the catalog's rows for the
    +//!   configured kind and root (the root was replaced under the rows);
    +//! - the catalog has rows for the root and the root has no file (the root
    +//!   was emptied or re-pointed);
    +//! - those rows carry more than one `store_id`;
    +//! - the manifest probe fails: the newest AVAILABLE row's
    +//!   `extenddb-manifest.json` is not under the root. Instances that share a
    +//!   catalog must share one store root; host-local roots at one path give
    +//!   each instance different bytes under one `store_root`, which the ids
    +//!   alone cannot see, so the probe reads the root independently of the
    +//!   host;
    +//! - the store cannot be reached or read, or the file cannot be parsed.
    +//!
    +//! On a root with no file and a catalog with no rows for it, the server
    +//! creates the file with `put_if_absent`, then re-reads it and uses the id
    +//! it holds. Of two instances of one catalog starting against one empty root,
    +//! one creates the file and the other's create finds it present, and both
    +//! use the winner's id. An S3-compatible server that ignores `If-None-Match`
    +//! makes the create a plain put, last writer wins: the re-read catches a race
    +//! lost before it (the file holds another id than this instance wrote) and
    +//! marks the store unhealthy naming both ids; a race lost after it is caught
    +//! by the next check, which finds rows carrying an id the file does not.
    +//!
    +//! An unhealthy store is not a startup refusal: the caller logs the problem,
    +//! reports it on `/health`, refuses the operations that touch the store, and
    +//! runs the check again each cycle until it passes.
    +
    +use extenddb_backup_store::{BackupStore, StoreError, byte_stream_from};
    +use extenddb_core::backup_format::{BackupArn, EXTENDDB_MANIFEST_FILE};
    +use extenddb_storage::BackupCatalog;
    +use extenddb_storage::backup_catalog::{BackupStatus, StoreIdentity, StoreKind};
    +use futures::TryStreamExt;
    +use serde::{Deserialize, Serialize};
    +
    +/// The key of the identity file, at the root, under no `/` prefix.
    +pub const STORE_IDENTITY_KEY: &str = "extenddb-store.json";
    +
    +/// The one version of the file's shape this build writes and reads.
    +pub const STORE_IDENTITY_FORMAT_VERSION: u32 = 1;
    +
    +/// The most bytes a check reads of `extenddb-store.json`. The file this
    +/// build writes is under 300 bytes; an object larger than this under the
    +/// key is not an identity file, and is reported as unreadable without being
    +/// read into memory on every cycle.
    +pub const STORE_IDENTITY_MAX_BYTES: usize = 64 * 1024;
    +
    +/// The contents of `extenddb-store.json`.
    +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
    +pub struct StoreIdentityFile {
    +    pub format_version: u32,
    +    /// The random id every row of this store carries.
    +    pub store_id: String,
    +    /// The catalog that owns the root.
    +    pub catalog_id: String,
    +    /// The storage backend of the owning catalog, for the operator reading
    +    /// the file; nothing is checked against it.
    +    pub backend: String,
    +    /// When the file was created, epoch milliseconds on the creating host.
    +    pub created_at_ms: u64,
    +}
    +
    +impl StoreIdentityFile {
    +    /// A new identity for `catalog_id`.
    +    fn new(catalog_id: &str, backend: &str) -> Self {
    +        Self {
    +            format_version: STORE_IDENTITY_FORMAT_VERSION,
    +            store_id: uuid::Uuid::new_v4().to_string(),
    +            catalog_id: catalog_id.to_owned(),
    +            backend: backend.to_owned(),
    +            created_at_ms: std::time::SystemTime::now()
    +                .duration_since(std::time::UNIX_EPOCH)
    +                .map_or(0, |d| u64::try_from(d.as_millis()).unwrap_or(u64::MAX)),
    +        }
    +    }
    +
    +    /// Parse the file's bytes.
    +    ///
    +    /// # Errors
    +    ///
    +    /// [`StoreProblem::FileUnreadable`] for bytes that are not the documented
    +    /// shape, or carry a version this build does not read.
    +    pub fn parse(bytes: &[u8]) -> Result {
    +        let file: Self = serde_json::from_slice(bytes)
    +            .map_err(|e| StoreProblem::FileUnreadable(format!("not a store identity file: {e}")))?;
    +        if file.format_version != STORE_IDENTITY_FORMAT_VERSION {
    +            return Err(StoreProblem::FileUnreadable(format!(
    +                "format_version {} is not {STORE_IDENTITY_FORMAT_VERSION}",
    +                file.format_version
    +            )));
    +        }
    +        if file.store_id.is_empty() || file.catalog_id.is_empty() {
    +            return Err(StoreProblem::FileUnreadable(
    +                "store_id and catalog_id must be non-empty".to_owned(),
    +            ));
    +        }
    +        Ok(file)
    +    }
    +
    +    /// The file's bytes.
    +    #[must_use]
    +    pub fn to_bytes(&self) -> Vec {
    +        let mut bytes = serde_json::to_vec_pretty(self).expect("a plain struct serializes");
    +        bytes.push(b'\n');
    +        bytes
    +    }
    +}
    +
    +/// Why a store is unhealthy. `Display` is the operator's line (ids, the
    +/// catalog id, the store's own error text, never a credential);
    +/// [`StoreProblem::public_reason`] is the part of it that names no root, and
    +/// [`StoreProblem::category`] alone is what `/health` reports.
    +#[derive(Debug, Clone, PartialEq, Eq)]
    +pub enum StoreProblem {
    +    /// The store could not be reached, read or written; the store's error.
    +    Transport(String),
    +    /// The store refused an operation: a root or a directory under it that
    +    /// another user can reach, a denied grant; the store's error.
    +    Denied(String),
    +    /// The store's `validate`, or a request of the identity check, failed for
    +    /// a reason that is neither a transport failure nor a denial (a bucket
    +    /// that does not exist, an I/O error on a filesystem root); the store's
    +    /// error. The configuration errors `validate` reports before the
    +    /// listener stop startup instead. One category whether `validate` or a
    +    /// later check's request found the condition, so a bucket deleted under
    +    /// a running server reads as it does at startup.
    +    Validate(String),
    +    /// `extenddb-store.json` is there but is not a readable identity file.
    +    FileUnreadable(String),
    +    /// The file names a catalog other than this one.
    +    ForeignCatalog {
    +        file_catalog_id: String,
    +        catalog_id: String,
    +    },
    +    /// The file's id is not the id the catalog's rows for the root carry.
    +    IdMismatch {
    +        file_id: String,
    +        row_ids: Vec,
    +    },
    +    /// The catalog's rows for the root carry more than one id.
    +    MultipleRowIds(Vec),
    +    /// The catalog has rows for the root and the root has no file.
    +    FileMissingWithRows(Vec),
    +    /// This instance's create reported success and the re-read holds another
    +    /// id: the store did not honour the conditional put.
    +    CreateRaceLost { ours: String, file: String },
    +    /// The newest AVAILABLE row's manifest is not under the root.
    +    ManifestMissing { backup_arn: String },
    +    /// The catalog could not be read.
    +    Catalog(String),
    +}
    +
    +impl StoreProblem {
    +    /// The problem's category, one or two words, for an API refusal.
    +    #[must_use]
    +    pub fn category(&self) -> &'static str {
    +        match self {
    +            Self::Transport(_) => "unreachable",
    +            Self::Denied(_) => "permission denied",
    +            Self::Validate(_) => "failed validation",
    +            Self::FileUnreadable(_) => "identity file unreadable",
    +            Self::ForeignCatalog { .. } => "owned by another catalog",
    +            Self::IdMismatch { .. } | Self::MultipleRowIds(_) | Self::CreateRaceLost { .. } => {
    +                "identity mismatch"
    +            }
    +            Self::FileMissingWithRows(_) => "identity file missing",
    +            Self::ManifestMissing { .. } => "manifest probe failed",
    +            Self::Catalog(_) => "catalog unreadable",
    +        }
    +    }
    +
    +    /// The problem with the ids involved and never a root (a filesystem path
    +    /// or a bucket) or the store's own error text, which may carry one: the
    +    /// operator's line for the identity problems. Not the `/health` text,
    +    /// which is [`StoreProblem::category`].
    +    #[must_use]
    +    pub fn public_reason(&self) -> String {
    +        match self {
    +            Self::Transport(_) => "the store could not be reached".to_owned(),
    +            Self::Denied(_) => "the store refused an operation (permission denied)".to_owned(),
    +            Self::Validate(_) => "the store failed its validation".to_owned(),
    +            Self::FileUnreadable(_) => format!("{STORE_IDENTITY_KEY} could not be read"),
    +            Self::ForeignCatalog {
    +                file_catalog_id,
    +                catalog_id,
    +            } => format!(
    +                "{STORE_IDENTITY_KEY} names catalog {file_catalog_id}; this catalog is \
    +                 {catalog_id}"
    +            ),
    +            Self::IdMismatch { file_id, row_ids } => format!(
    +                "{STORE_IDENTITY_KEY} holds store id {file_id}; the catalog's rows for the root \
    +                 carry {}",
    +                row_ids.join(", ")
    +            ),
    +            Self::MultipleRowIds(ids) => format!(
    +                "the catalog's rows for the root carry more than one store id: {}",
    +                ids.join(", ")
    +            ),
    +            Self::FileMissingWithRows(ids) => format!(
    +                "the root has no {STORE_IDENTITY_KEY} but the catalog has rows for it (store id \
    +                 {})",
    +                ids.join(", ")
    +            ),
    +            Self::CreateRaceLost { ours, file } => format!(
    +                "this server created {STORE_IDENTITY_KEY} with store id {ours} but the file \
    +                 holds {file}; the store did not honour the conditional put"
    +            ),
    +            Self::ManifestMissing { backup_arn } => {
    +                format!("the manifest of {backup_arn} is not under the root")
    +            }
    +            Self::Catalog(_) => "the catalog could not be read".to_owned(),
    +        }
    +    }
    +}
    +
    +impl std::fmt::Display for StoreProblem {
    +    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
    +        match self {
    +            Self::Transport(e) => write!(f, "the store could not be reached: {e}"),
    +            Self::Denied(e) => write!(f, "the store refused an operation: {e}"),
    +            Self::Validate(e) => write!(f, "the store failed its validation: {e}"),
    +            Self::FileUnreadable(e) => write!(f, "{STORE_IDENTITY_KEY} could not be read: {e}"),
    +            Self::Catalog(e) => write!(f, "the catalog could not be read: {e}"),
    +            other => f.write_str(&other.public_reason()),
    +        }
    +    }
    +}
    +
    +/// The problem a store's error is: a transport failure keeps its class with
    +/// the store error's whole cause chain (the SDK's top link is a bare
    +/// category such as "dispatch failure"; the cause is a link below it), a
    +/// denial is [`StoreProblem::Denied`], and every other refusal
    +/// ([`StoreProblem::Validate`]'s doc) is the same category `validate` gives
    +/// it.
    +impl From for StoreProblem {
    +    fn from(e: StoreError) -> Self {
    +        match e {
    +            StoreError::PermissionDenied(m) => Self::Denied(m),
    +            transport @ StoreError::Transport { .. } => Self::Transport(transport.to_string()),
    +            other => Self::Validate(other.to_string()),
    +        }
    +    }
    +}
    +
    +/// Read the identity file, or `None` when the root has none. At most
    +/// [`STORE_IDENTITY_MAX_BYTES`] are read; a larger object is
    +/// [`StoreProblem::FileUnreadable`].
    +async fn read_file(store: &dyn BackupStore) -> Result, StoreProblem> {
    +    let mut body = match store.get(STORE_IDENTITY_KEY).await {
    +        Ok(body) => body,
    +        Err(StoreError::NotFound) => return Ok(None),
    +        Err(e) => return Err(e.into()),
    +    };
    +    let mut bytes = Vec::new();
    +    while let Some(chunk) = body.try_next().await? {
    +        if bytes.len() + chunk.len() > STORE_IDENTITY_MAX_BYTES {
    +            return Err(StoreProblem::FileUnreadable(format!(
    +                "larger than {STORE_IDENTITY_MAX_BYTES} bytes"
    +            )));
    +        }
    +        bytes.extend_from_slice(&chunk);
    +    }
    +    StoreIdentityFile::parse(&bytes).map(Some)
    +}
    +
    +/// The distinct ids the catalog's rows for the root carry, at most one.
    +async fn row_ids(
    +    catalog: &C,
    +    kind: StoreKind,
    +    root: &str,
    +) -> Result, StoreProblem> {
    +    let ids = catalog
    +        .read_store_ids(kind, root)
    +        .await
    +        .map_err(|e| StoreProblem::Catalog(e.to_string()))?;
    +    if ids.len() > 1 {
    +        return Err(StoreProblem::MultipleRowIds(ids));
    +    }
    +    Ok(ids)
    +}
    +
    +/// The startup check: `validate` has passed; read the file, create it when
    +/// the root is empty of both file and rows, compare it with the rows and
    +/// with `catalog_id`, then probe the newest manifest. Returns the store's
    +/// identity, the rows' and the file's.
    +///
    +/// # Errors
    +///
    +/// The problem that makes the store unhealthy (module doc).
    +pub async fn establish(
    +    store: &dyn BackupStore,
    +    kind: StoreKind,
    +    root: &str,
    +    catalog: &C,
    +    backend: &str,
    +) -> Result {
    +    let catalog_id = catalog
    +        .catalog_id()
    +        .await
    +        .map_err(|e| StoreProblem::Catalog(e.to_string()))?;
    +    let row_ids = row_ids(catalog, kind, root).await?;
    +    let file = match read_file(store).await? {
    +        Some(file) => file,
    +        None if !row_ids.is_empty() => {
    +            return Err(StoreProblem::FileMissingWithRows(row_ids));
    +        }
    +        None => {
    +            let ours = StoreIdentityFile::new(&catalog_id, backend);
    +            let created = store
    +                .put_if_absent(STORE_IDENTITY_KEY, byte_stream_from(ours.to_bytes()))
    +                .await?
    +                .is_some();
    +            // The key holds nothing the store reads as an object and refuses
    +            // the create (a FIFO or a socket at the path, say): nothing was
    +            // created, and nothing can be read.
    +            let file = read_file(store).await?.ok_or_else(|| {
    +                StoreProblem::Transport(format!(
    +                    "{STORE_IDENTITY_KEY} is neither readable nor creatable"
    +                ))
    +            })?;
    +            if created && file.store_id != ours.store_id {
    +                return Err(StoreProblem::CreateRaceLost {
    +                    ours: ours.store_id,
    +                    file: file.store_id,
    +                });
    +            }
    +            if created {
    +                tracing::info!(
    +                    "backup store ({kind}) root was empty; created {STORE_IDENTITY_KEY} with \
    +                     store id {}",
    +                    file.store_id
    +                );
    +            } else {
    +                tracing::info!(
    +                    "backup store ({kind}) root already carried {STORE_IDENTITY_KEY} (store id \
    +                     {}), created by another instance of this catalog",
    +                    file.store_id
    +                );
    +            }
    +            file
    +        }
    +    };
    +    let identity = compare(&file, &catalog_id, row_ids, kind, root)?;
    +    manifest_probe(store, &identity, catalog).await?;
    +    Ok(identity)
    +}
    +
    +/// A read of the store's identity that writes nothing: the file, compared
    +/// with the catalog as [`establish`] compares it, without the create of a
    +/// missing file and without the manifest probe. `Ok(None)` for a root with
    +/// no file and no rows (a fresh root). For a command that reads beside a
    +/// server (`backup list`), which must leave the root as it found it.
    +///
    +/// # Errors
    +///
    +/// The problem that would make the store unhealthy (module doc).
    +pub async fn read_identity(
    +    store: &dyn BackupStore,
    +    kind: StoreKind,
    +    root: &str,
    +    catalog: &C,
    +) -> Result, StoreProblem> {
    +    let catalog_id = catalog
    +        .catalog_id()
    +        .await
    +        .map_err(|e| StoreProblem::Catalog(e.to_string()))?;
    +    let row_ids = row_ids(catalog, kind, root).await?;
    +    match read_file(store).await? {
    +        Some(file) => compare(&file, &catalog_id, row_ids, kind, root).map(Some),
    +        None if !row_ids.is_empty() => Err(StoreProblem::FileMissingWithRows(row_ids)),
    +        None => Ok(None),
    +    }
    +}
    +
    +/// The file against the catalog: it must name this catalog, and the rows
    +/// for the root, when there are any, must carry its id.
    +fn compare(
    +    file: &StoreIdentityFile,
    +    catalog_id: &str,
    +    row_ids: Vec,
    +    kind: StoreKind,
    +    root: &str,
    +) -> Result {
    +    if file.catalog_id != catalog_id {
    +        return Err(StoreProblem::ForeignCatalog {
    +            file_catalog_id: file.catalog_id.clone(),
    +            catalog_id: catalog_id.to_owned(),
    +        });
    +    }
    +    if !row_ids.is_empty() && row_ids[0] != file.store_id {
    +        return Err(StoreProblem::IdMismatch {
    +            file_id: file.store_id.clone(),
    +            row_ids,
    +        });
    +    }
    +    Ok(StoreIdentity {
    +        store_kind: kind,
    +        store_root: root.to_owned(),
    +        store_id: file.store_id.clone(),
    +    })
    +}
    +
    +/// The check every reconciler cycle runs on a store whose identity startup
    +/// settled: the file is still there, names this catalog and holds the
    +/// identity's id; the rows for the root carry no other id; the newest
    +/// manifest is under the root. A root that lost its file while the catalog
    +/// has no rows for it gets the file back, written with the identity's id
    +/// (`put_if_absent`, so a file another instance wrote first wins and is then
    +/// compared like any other).
    +///
    +/// # Errors
    +///
    +/// The problem that makes the store unhealthy (module doc).
    +pub async fn recheck(
    +    store: &dyn BackupStore,
    +    identity: &StoreIdentity,
    +    catalog: &C,
    +    backend: &str,
    +) -> Result<(), StoreProblem> {
    +    let catalog_id = catalog
    +        .catalog_id()
    +        .await
    +        .map_err(|e| StoreProblem::Catalog(e.to_string()))?;
    +    let row_ids = row_ids(catalog, identity.store_kind, &identity.store_root).await?;
    +    let file = match read_file(store).await? {
    +        Some(file) => file,
    +        None if !row_ids.is_empty() => {
    +            return Err(StoreProblem::FileMissingWithRows(row_ids));
    +        }
    +        None => {
    +            let ours = StoreIdentityFile {
    +                store_id: identity.store_id.clone(),
    +                ..StoreIdentityFile::new(&catalog_id, backend)
    +            };
    +            store
    +                .put_if_absent(STORE_IDENTITY_KEY, byte_stream_from(ours.to_bytes()))
    +                .await?;
    +            tracing::warn!(
    +                "backup store ({}) root had lost {STORE_IDENTITY_KEY}; rewrote it with store id \
    +                 {}",
    +                identity.store_kind,
    +                identity.store_id
    +            );
    +            read_file(store).await?.ok_or_else(|| {
    +                StoreProblem::Transport(format!(
    +                    "{STORE_IDENTITY_KEY} is neither readable nor creatable"
    +                ))
    +            })?
    +        }
    +    };
    +    if file.catalog_id != catalog_id {
    +        return Err(StoreProblem::ForeignCatalog {
    +            file_catalog_id: file.catalog_id,
    +            catalog_id,
    +        });
    +    }
    +    if file.store_id != identity.store_id {
    +        // The file changed under a running server: the root was replaced.
    +        return Err(StoreProblem::IdMismatch {
    +            file_id: file.store_id,
    +            row_ids: vec![identity.store_id.clone()],
    +        });
    +    }
    +    if let Some(other) = row_ids.iter().find(|id| **id != identity.store_id) {
    +        return Err(StoreProblem::IdMismatch {
    +            file_id: file.store_id,
    +            row_ids: vec![other.clone()],
    +        });
    +    }
    +    manifest_probe(store, identity, catalog).await
    +}
    +
    +/// `head` the manifest of the newest AVAILABLE row of the store; a root with
    +/// no AVAILABLE row passes (there is nothing to probe).
    +///
    +/// # Errors
    +///
    +/// [`StoreProblem::ManifestMissing`] when the manifest is absent,
    +/// [`StoreProblem::Transport`] when the store cannot be read,
    +/// [`StoreProblem::Catalog`] when the rows cannot be listed.
    +pub async fn manifest_probe(
    +    store: &dyn BackupStore,
    +    identity: &StoreIdentity,
    +    catalog: &C,
    +) -> Result<(), StoreProblem> {
    +    let rows: Vec<_> = catalog
    +        .list_backup_rows(Some(BackupStatus::Available), identity)
    +        .try_collect()
    +        .await
    +        .map_err(|e| StoreProblem::Catalog(e.to_string()))?;
    +    // Listings are ordered (`created_at`, `backup_arn`) ascending; the
    +    // newest row is the last.
    +    let Some(newest) = rows.last() else {
    +        return Ok(());
    +    };
    +    let arn = BackupArn::parse(&newest.backup_arn).map_err(|e| {
    +        StoreProblem::Catalog(format!("row {} carries a bad ARN: {e}", newest.backup_arn))
    +    })?;
    +    let key = format!("{}{EXTENDDB_MANIFEST_FILE}", arn.store_prefix());
    +    match store.head(&key).await? {
    +        Some(_) => Ok(()),
    +        None => Err(StoreProblem::ManifestMissing {
    +            backup_arn: newest.backup_arn.clone(),
    +        }),
    +    }
    +}
    +
    +#[cfg(test)]
    +mod tests {
    +    use super::*;
    +
    +    #[test]
    +    fn the_file_round_trips_and_refuses_other_shapes() {
    +        let file = StoreIdentityFile::new("cat-1", "postgres");
    +        assert_eq!(file.format_version, 1);
    +        assert_eq!(file.catalog_id, "cat-1");
    +        assert_eq!(file.backend, "postgres");
    +        assert!(!file.store_id.is_empty());
    +        let parsed = StoreIdentityFile::parse(&file.to_bytes()).unwrap();
    +        assert_eq!(parsed, file);
    +
    +        let v2 = serde_json::json!({
    +            "format_version": 2, "store_id": "s", "catalog_id": "c", "backend": "b",
    +            "created_at_ms": 0
    +        });
    +        let err = StoreIdentityFile::parse(v2.to_string().as_bytes()).unwrap_err();
    +        assert!(matches!(&err, StoreProblem::FileUnreadable(m) if m.contains("format_version 2")));
    +        let empty = serde_json::json!({
    +            "format_version": 1, "store_id": "", "catalog_id": "c", "backend": "b",
    +            "created_at_ms": 0
    +        });
    +        assert!(matches!(
    +            StoreIdentityFile::parse(empty.to_string().as_bytes()),
    +            Err(StoreProblem::FileUnreadable(_))
    +        ));
    +        assert!(matches!(
    +            StoreIdentityFile::parse(b"not json"),
    +            Err(StoreProblem::FileUnreadable(_))
    +        ));
    +    }
    +
    +    /// `public_reason` names ids and the catalog, never a root or the store's
    +    /// own error text (which may carry a path or a bucket); `Display` carries
    +    /// that text, and `/health` carries the category alone.
    +    #[test]
    +    fn the_public_reason_names_no_root() {
    +        let transport = StoreProblem::Transport("connect /srv/backups: refused".to_owned());
    +        assert!(!transport.public_reason().contains("/srv"));
    +        assert!(transport.to_string().contains("/srv/backups"));
    +        assert_eq!(transport.category(), "unreachable");
    +        let foreign = StoreProblem::ForeignCatalog {
    +            file_catalog_id: "other".to_owned(),
    +            catalog_id: "mine".to_owned(),
    +        };
    +        assert!(foreign.public_reason().contains("other"));
    +        assert!(foreign.public_reason().contains("mine"));
    +        assert_eq!(foreign.category(), "owned by another catalog");
    +        let probe = StoreProblem::ManifestMissing {
    +            backup_arn: "arn:x".to_owned(),
    +        };
    +        assert!(probe.public_reason().contains("arn:x"));
    +        assert_eq!(probe.category(), "manifest probe failed");
    +        let unreadable = StoreProblem::FileUnreadable("at /srv/x".to_owned());
    +        assert!(!unreadable.public_reason().contains("/srv"));
    +        let denied: StoreProblem =
    +            StoreError::PermissionDenied("/srv/backups/a has mode 0755".to_owned()).into();
    +        assert_eq!(denied.category(), "permission denied");
    +        assert!(!denied.public_reason().contains("/srv"));
    +        assert!(denied.to_string().contains("/srv/backups/a"));
    +        // A transport failure keeps its whole cause chain: the SDK's top
    +        // link names only the class, and the cause is the link below it.
    +        let chain: StoreProblem = StoreError::Transport {
    +            source: Box::new(TwoLinks),
    +        }
    +        .into();
    +        assert_eq!(chain.category(), "unreachable");
    +        assert!(chain.to_string().contains("dispatch failure"), "{chain}");
    +        assert!(chain.to_string().contains("connection refused"), "{chain}");
    +        // Any other store refusal is the category `validate` gives it, so a
    +        // bucket that disappears under a running server reads as a bucket
    +        // that was missing at startup.
    +        let missing: StoreProblem = StoreError::Other("NoSuchBucket".to_owned()).into();
    +        assert_eq!(missing.category(), "failed validation");
    +        assert!(missing.to_string().contains("NoSuchBucket"));
    +    }
    +
    +    /// A two-link error chain: "dispatch failure" over "connection refused".
    +    #[derive(Debug)]
    +    struct TwoLinks;
    +
    +    impl std::fmt::Display for TwoLinks {
    +        fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
    +            f.write_str("dispatch failure")
    +        }
    +    }
    +
    +    impl std::error::Error for TwoLinks {
    +        fn source(&self) -> Option<&(dyn std::error::Error + 'static)> {
    +            Some(&CONNECTION_REFUSED)
    +        }
    +    }
    +
    +    static CONNECTION_REFUSED: ConnectionRefused = ConnectionRefused;
    +
    +    #[derive(Debug)]
    +    struct ConnectionRefused;
    +
    +    impl std::fmt::Display for ConnectionRefused {
    +        fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
    +            f.write_str("connection refused")
    +        }
    +    }
    +
    +    impl std::error::Error for ConnectionRefused {}
    +
    +    #[test]
    +    fn compare_applies_the_three_rules() {
    +        let file = StoreIdentityFile {
    +            format_version: 1,
    +            store_id: "s-1".to_owned(),
    +            catalog_id: "c-1".to_owned(),
    +            backend: "sqlite".to_owned(),
    +            created_at_ms: 0,
    +        };
    +        let ok = compare(&file, "c-1", vec![], StoreKind::Filesystem, "/r").unwrap();
    +        assert_eq!(ok.store_id, "s-1");
    +        assert_eq!(ok.store_root, "/r");
    +        compare(
    +            &file,
    +            "c-1",
    +            vec!["s-1".to_owned()],
    +            StoreKind::Filesystem,
    +            "/r",
    +        )
    +        .unwrap();
    +        assert!(matches!(
    +            compare(&file, "c-2", vec![], StoreKind::Filesystem, "/r"),
    +            Err(StoreProblem::ForeignCatalog { .. })
    +        ));
    +        assert!(matches!(
    +            compare(
    +                &file,
    +                "c-1",
    +                vec!["s-9".to_owned()],
    +                StoreKind::S3,
    +                "s3://b/"
    +            ),
    +            Err(StoreProblem::IdMismatch { .. })
    +        ));
    +    }
    +}
    diff --git a/crates/engine/src/backup/tests.rs b/crates/engine/src/backup/tests.rs
    index adb0fc616..e77f2f80e 100644
    --- a/crates/engine/src/backup/tests.rs
    +++ b/crates/engine/src/backup/tests.rs
    @@ -5,7 +5,7 @@
     //! reader that records what the engine asked of it.
     
     use std::collections::HashMap;
    -use std::sync::Mutex;
    +use std::sync::{Arc, Mutex};
     use std::time::Duration;
     
     use extenddb_core::backup_format::BackupArn;
    @@ -30,6 +30,7 @@ use extenddb_storage::{
         BackupCatalog, BackupRestoreSource, BoxedFuture, JobPermit, LegacyBackupReader, RestoreRequest,
         TableEngine,
     };
    +use futures::TryStreamExt;
     use futures::stream::BoxStream;
     use serde_json::{Value, json};
     
    @@ -87,9 +88,9 @@ struct Fake {
         abandoned_backups: Mutex>,
         deleted: Mutex>,
         maintenance_held: Mutex,
    -    /// How many times `check_maintenance_claim` ran: one per reconciler
    -    /// cycle, so a test can see that a cycle ran.
    -    maintenance_checks: Mutex,
    +    /// The windows `check_maintenance_claim` was called with: one entry per
    +    /// reconciler cycle, so a test can see that a cycle ran.
    +    maintenance_checks: Mutex>,
         /// When set, `check_maintenance_claim` fails with it (every call).
         maintenance_error: Mutex>,
         /// The `TimeBounds` `list_account_backup_rows` received.
    @@ -99,6 +100,16 @@ struct Fake {
         tables: Mutex>,
         /// The names `describe_table` was asked for.
         table_reads: Mutex>,
    +    /// What `catalog_id` answers; `catalog-1` unless a test changes it.
    +    catalog_id: Mutex>,
    +    /// When set, `catalog_id` fails as a catalog that cannot be read.
    +    catalog_id_error: Mutex,
    +    /// Rows `set_backup_deleted` moved to DELETED, by ARN.
    +    set_deleted: Mutex>,
    +    /// Rows `remove_backup_row` removed, by ARN.
    +    removed_rows: Mutex>,
    +    /// When set, `list_backup_rows` fails with it.
    +    list_error: Mutex>,
     }
     
     impl Fake {
    @@ -135,6 +146,299 @@ fn store() -> StoreIdentity {
         }
     }
     
    +/// An in-memory `BackupStore` for the engine's tests: objects by key, a
    +/// switch that makes every call fail as a transport error, a switch that
    +/// makes every call wait forever (a store that has stopped answering), and
    +/// the keys each operation was called with.
    +#[derive(Default)]
    +struct MemStore {
    +    objects: Mutex>>,
    +    /// When set, every operation fails with this text as the cause under a
    +    /// "dispatch failure" link, the shape the SDK's transport errors have.
    +    down: Mutex>,
    +    /// When set, every operation's future never completes; `stalls` counts
    +    /// the calls that entered the stall. A stalled call returns once the
    +    /// store is un-stalled, as a parked filesystem call returns when its
    +    /// mount answers.
    +    stalled: std::sync::atomic::AtomicBool,
    +    stalls: std::sync::atomic::AtomicUsize,
    +    deletes: Mutex>,
    +    puts: Mutex>,
    +    /// When set, the bytes `put_if_absent` leaves under the key right after
    +    /// it stored its own: a server that ignores `If-None-Match` and a second
    +    /// writer that won the plain-put race.
    +    overwrite_after_conditional_put: Mutex>>,
    +    /// When set, `put_if_absent` finds the key taken (`None`) while `get`
    +    /// and `head` find no object: something that is not an object (a FIFO,
    +    /// a socket) sits at the path.
    +    key_taken_by_a_non_object: std::sync::atomic::AtomicBool,
    +}
    +
    +/// A transport error with the SDK's shape: a top link naming only the class
    +/// and the cause one link below it.
    +#[derive(Debug)]
    +struct DispatchFailure(ConnectError);
    +
    +#[derive(Debug)]
    +struct ConnectError(String);
    +
    +impl std::fmt::Display for DispatchFailure {
    +    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
    +        f.write_str("dispatch failure")
    +    }
    +}
    +
    +impl std::error::Error for DispatchFailure {
    +    fn source(&self) -> Option<&(dyn std::error::Error + 'static)> {
    +        Some(&self.0)
    +    }
    +}
    +
    +impl std::fmt::Display for ConnectError {
    +    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
    +        f.write_str(&self.0)
    +    }
    +}
    +
    +impl std::error::Error for ConnectError {}
    +
    +impl MemStore {
    +    /// A store that is down fails every call as the S3 store fails one it
    +    /// cannot send: a transport error whose cause is a link below the top.
    +    fn check(&self) -> Result<(), extenddb_backup_store::StoreError> {
    +        match self.down.lock().unwrap().as_ref() {
    +            Some(text) => Err(extenddb_backup_store::StoreError::Transport {
    +                source: Box::new(DispatchFailure(ConnectError(text.clone()))),
    +            }),
    +            None => Ok(()),
    +        }
    +    }
    +    /// Wait while the store is stalled; otherwise return at once.
    +    async fn stall(&self) {
    +        if self.stalled.load(std::sync::atomic::Ordering::Acquire) {
    +            self.stalls
    +                .fetch_add(1, std::sync::atomic::Ordering::AcqRel);
    +            while self.stalled.load(std::sync::atomic::Ordering::Acquire) {
    +                tokio::time::sleep(Duration::from_millis(10)).await;
    +            }
    +        }
    +    }
    +    fn set_down(&self, down: bool) {
    +        *self.down.lock().unwrap() = down.then(|| "connection refused".to_owned());
    +    }
    +    fn set_stalled(&self, stalled: bool) {
    +        self.stalled
    +            .store(stalled, std::sync::atomic::Ordering::Release);
    +    }
    +    fn insert(&self, key: &str, body: &[u8]) {
    +        self.objects
    +            .lock()
    +            .unwrap()
    +            .insert(key.to_owned(), body.to_vec());
    +    }
    +    fn keys(&self) -> Vec {
    +        self.objects.lock().unwrap().keys().cloned().collect()
    +    }
    +    fn matches(key: &str, prefix: &str) -> bool {
    +        let prefix = prefix.trim_end_matches('/');
    +        prefix.is_empty() || key == prefix || key.starts_with(&format!("{prefix}/"))
    +    }
    +}
    +
    +async fn collect_body(
    +    body: extenddb_backup_store::ByteStream,
    +) -> Result, extenddb_backup_store::StoreError> {
    +    body.try_fold(Vec::new(), |mut acc, chunk| async move {
    +        acc.extend_from_slice(&chunk);
    +        Ok(acc)
    +    })
    +    .await
    +}
    +
    +impl extenddb_backup_store::BackupStore for MemStore {
    +    fn put<'a>(
    +        &'a self,
    +        key: &'a str,
    +        body: extenddb_backup_store::ByteStream,
    +    ) -> BoxedFuture<'a, Result>
    +    {
    +        Box::pin(async move {
    +            self.stall().await;
    +            self.check()?;
    +            let bytes = collect_body(body).await?;
    +            self.puts.lock().unwrap().push(key.to_owned());
    +            let size = bytes.len() as u64;
    +            self.insert(key, &bytes);
    +            Ok(extenddb_backup_store::ObjectMeta {
    +                key: key.to_owned(),
    +                size,
    +                last_modified: None,
    +                etag: None,
    +            })
    +        })
    +    }
    +
    +    fn put_if_absent<'a>(
    +        &'a self,
    +        key: &'a str,
    +        body: extenddb_backup_store::ByteStream,
    +    ) -> BoxedFuture<
    +        'a,
    +        Result, extenddb_backup_store::StoreError>,
    +    > {
    +        Box::pin(async move {
    +            self.stall().await;
    +            self.check()?;
    +            let bytes = collect_body(body).await?;
    +            if self
    +                .key_taken_by_a_non_object
    +                .load(std::sync::atomic::Ordering::Acquire)
    +            {
    +                return Ok(None);
    +            }
    +            let mut objects = self.objects.lock().unwrap();
    +            if objects.contains_key(key) {
    +                return Ok(None);
    +            }
    +            self.puts.lock().unwrap().push(key.to_owned());
    +            let size = bytes.len() as u64;
    +            objects.insert(key.to_owned(), bytes);
    +            if let Some(other) = self.overwrite_after_conditional_put.lock().unwrap().clone() {
    +                objects.insert(key.to_owned(), other);
    +            }
    +            Ok(Some(extenddb_backup_store::ObjectMeta {
    +                key: key.to_owned(),
    +                size,
    +                last_modified: None,
    +                etag: None,
    +            }))
    +        })
    +    }
    +
    +    fn get<'a>(
    +        &'a self,
    +        key: &'a str,
    +    ) -> BoxedFuture<'a, Result>
    +    {
    +        Box::pin(async move {
    +            self.stall().await;
    +            self.check()?;
    +            let bytes = self
    +                .objects
    +                .lock()
    +                .unwrap()
    +                .get(key)
    +                .cloned()
    +                .ok_or(extenddb_backup_store::StoreError::NotFound)?;
    +            Ok(extenddb_backup_store::byte_stream_from(bytes))
    +        })
    +    }
    +
    +    fn head<'a>(
    +        &'a self,
    +        key: &'a str,
    +    ) -> BoxedFuture<
    +        'a,
    +        Result, extenddb_backup_store::StoreError>,
    +    > {
    +        Box::pin(async move {
    +            self.stall().await;
    +            self.check()?;
    +            Ok(self.objects.lock().unwrap().get(key).map(|bytes| {
    +                extenddb_backup_store::ObjectMeta {
    +                    key: key.to_owned(),
    +                    size: bytes.len() as u64,
    +                    last_modified: None,
    +                    etag: None,
    +                }
    +            }))
    +        })
    +    }
    +
    +    fn list<'a>(
    +        &'a self,
    +        prefix: &'a str,
    +    ) -> BoxStream<'a, Result>
    +    {
    +        if let Err(e) = self.check() {
    +            return Box::pin(futures::stream::once(async move { Err(e) }));
    +        }
    +        let metas: Vec<_> = self
    +            .objects
    +            .lock()
    +            .unwrap()
    +            .iter()
    +            .filter(|(k, _)| Self::matches(k, prefix))
    +            .map(|(k, v)| {
    +                Ok(extenddb_backup_store::ObjectMeta {
    +                    key: k.clone(),
    +                    size: v.len() as u64,
    +                    last_modified: None,
    +                    etag: None,
    +                })
    +            })
    +            .collect();
    +        Box::pin(futures::stream::iter(metas))
    +    }
    +
    +    fn delete_prefix<'a>(
    +        &'a self,
    +        prefix: &'a str,
    +    ) -> BoxedFuture<'a, Result> {
    +        Box::pin(async move {
    +            self.stall().await;
    +            self.check()?;
    +            self.deletes.lock().unwrap().push(prefix.to_owned());
    +            let mut objects = self.objects.lock().unwrap();
    +            let before = objects.len();
    +            objects.retain(|k, _| !Self::matches(k, prefix));
    +            Ok((before - objects.len()) as u64)
    +        })
    +    }
    +
    +    fn validate(&self) -> BoxedFuture<'_, Result<(), extenddb_backup_store::StoreError>> {
    +        Box::pin(async move {
    +            self.stall().await;
    +            self.check()
    +        })
    +    }
    +}
    +
    +/// A runtime with `store` configured as `identity`, healthy.
    +fn runtime_on(mem: &Arc, identity: StoreIdentity) -> BackupRuntime {
    +    runtime().with_store(ConfiguredStore::new(
    +        identity.store_kind,
    +        identity.store_root.clone(),
    +        Arc::clone(mem) as Arc,
    +        None,
    +        Ok(identity),
    +    ))
    +}
    +
    +/// A runtime whose configured store (`store()`'s kind and root) failed its
    +/// startup check with `problem`: the id is unknown.
    +fn runtime_unhealthy(mem: &Arc, problem: StoreProblem) -> BackupRuntime {
    +    runtime().with_store(ConfiguredStore::new(
    +        store().store_kind,
    +        store().store_root,
    +        Arc::clone(mem) as Arc,
    +        None,
    +        Err(problem),
    +    ))
    +}
    +
    +/// The identity file of `store()` for catalog `catalog_id`, as bytes.
    +fn identity_file(store_id: &str, catalog_id: &str) -> Vec {
    +    StoreIdentityFile {
    +        format_version: 1,
    +        store_id: store_id.to_owned(),
    +        catalog_id: catalog_id.to_owned(),
    +        backend: "test".to_owned(),
    +        created_at_ms: 0,
    +    }
    +    .to_bytes()
    +}
    +
     /// A catalog row for `arn`, with the manifest fixture's source table (the
     /// `Music` table of account 123456789012).
     fn row(arn: &str, status: BackupStatus) -> BackupRow {
    @@ -164,7 +468,16 @@ fn row(arn: &str, status: BackupStatus) -> BackupRow {
     
     impl BackupCatalog for Fake {
         fn catalog_id(&self) -> BoxedFuture<'_, Result> {
    -        Box::pin(async { Ok("catalog-1".to_owned()) })
    +        if *self.catalog_id_error.lock().unwrap() {
    +            return Box::pin(async { Err(StorageError::Internal("pool exhausted".to_owned())) });
    +        }
    +        let id = self
    +            .catalog_id
    +            .lock()
    +            .unwrap()
    +            .clone()
    +            .unwrap_or_else(|| "catalog-1".to_owned());
    +        Box::pin(async move { Ok(id) })
         }
     
         fn claim_backup<'a>(
    @@ -174,12 +487,33 @@ impl BackupCatalog for Fake {
             unused()
         }
     
    +    /// The backend's listing: rows of the store (kind, root and id), of the
    +    /// status when given, in (`created_at`, `backup_arn`) order.
         fn list_backup_rows<'a>(
             &'a self,
    -        _status: Option,
    -        _store: &'a StoreIdentity,
    +        status: Option,
    +        store: &'a StoreIdentity,
         ) -> BoxStream<'a, Result> {
    -        Box::pin(futures::stream::empty())
    +        if let Some(e) = self.list_error.lock().unwrap().clone() {
    +            return Box::pin(futures::stream::once(async move {
    +                Err(StorageError::Internal(e))
    +            }));
    +        }
    +        let mut rows: Vec = self
    +            .rows
    +            .lock()
    +            .unwrap()
    +            .values()
    +            .filter(|r| r.store == *store)
    +            .filter(|r| status.is_none_or(|s| r.status == s))
    +            .cloned()
    +            .collect();
    +        rows.sort_by(|a, b| {
    +            a.created_at
    +                .total_cmp(&b.created_at)
    +                .then_with(|| a.backup_arn.cmp(&b.backup_arn))
    +        });
    +        Box::pin(futures::stream::iter(rows.into_iter().map(Ok)))
         }
     
         fn list_account_backup_rows<'a>(
    @@ -231,12 +565,17 @@ impl BackupCatalog for Fake {
                 .unwrap()
                 .get(&key.backup_arn)
                 .cloned();
    -        let status = self
    -            .rows
    -            .lock()
    -            .unwrap()
    -            .get(&key.backup_arn)
    -            .map(|r| r.status);
    +        let mut rows = self.rows.lock().unwrap();
    +        let status = rows.get(&key.backup_arn).map(|r| r.status);
    +        // The natural answer moves an AVAILABLE row to DELETING, as the
    +        // backends do; a preset answer leaves the row as the test set it.
    +        if answer.is_none()
    +            && status == Some(BackupStatus::Available)
    +            && let Some(row) = rows.get_mut(&key.backup_arn)
    +        {
    +            row.status = BackupStatus::Deleting;
    +        }
    +        drop(rows);
             Box::pin(async move {
                 match (answer, status) {
                     (Some(answer), _) => Ok(answer),
    @@ -247,18 +586,72 @@ impl BackupCatalog for Fake {
             })
         }
     
    +    /// DELETING to DELETED, conditional, as the backends do.
         fn set_backup_deleted<'a>(
             &'a self,
    -        _key: &'a BackupKey,
    +        key: &'a BackupKey,
         ) -> BoxedFuture<'a, Result> {
    -        unused()
    +        let moved = match self.rows.lock().unwrap().get_mut(&key.backup_arn) {
    +            Some(row) if row.status == BackupStatus::Deleting => {
    +                row.status = BackupStatus::Deleted;
    +                true
    +            }
    +            _ => false,
    +        };
    +        if moved {
    +            self.set_deleted
    +                .lock()
    +                .unwrap()
    +                .push(key.backup_arn.clone());
    +        }
    +        Box::pin(async move { Ok(moved) })
         }
     
    +    /// Remove a DELETED or FAILED tombstone, as the backends do.
         fn remove_backup_row<'a>(
             &'a self,
    -        _key: &'a BackupKey,
    +        key: &'a BackupKey,
         ) -> BoxedFuture<'a, Result> {
    -        unused()
    +        let mut rows = self.rows.lock().unwrap();
    +        let removed = match rows.get(&key.backup_arn) {
    +            Some(row) if matches!(row.status, BackupStatus::Deleted | BackupStatus::Failed) => {
    +                rows.remove(&key.backup_arn);
    +                true
    +            }
    +            _ => false,
    +        };
    +        if removed {
    +            self.removed_rows
    +                .lock()
    +                .unwrap()
    +                .push(key.backup_arn.clone());
    +        }
    +        Box::pin(async move { Ok(removed) })
    +    }
    +
    +    /// Remove a CREATING row whose heartbeat is older than `stale_after`
    +    /// (on this process's clock, which stands for the catalog's here).
    +    fn remove_dead_creating_backup_row<'a>(
    +        &'a self,
    +        key: &'a BackupKey,
    +        stale_after: Duration,
    +    ) -> BoxedFuture<'a, Result> {
    +        let now = std::time::SystemTime::now()
    +            .duration_since(std::time::UNIX_EPOCH)
    +            .unwrap()
    +            .as_secs_f64();
    +        let mut rows = self.rows.lock().unwrap();
    +        let removed = match rows.get(&key.backup_arn) {
    +            Some(row)
    +                if row.status == BackupStatus::Creating
    +                    && row.heartbeat_at < now - stale_after.as_secs_f64() =>
    +            {
    +                rows.remove(&key.backup_arn);
    +                true
    +            }
    +            _ => false,
    +        };
    +        Box::pin(async move { Ok(removed) })
         }
     
         fn backup_in_use<'a>(
    @@ -299,8 +692,37 @@ impl BackupCatalog for Fake {
             Box::pin(async move { Ok(answer) })
         }
     
    -    fn check_maintenance_claim(&self) -> BoxedFuture<'_, Result> {
    -        *self.maintenance_checks.lock().unwrap() += 1;
    +    fn read_store_ids<'a>(
    +        &'a self,
    +        store_kind: StoreKind,
    +        store_root: &'a str,
    +    ) -> BoxedFuture<'a, Result, StorageError>> {
    +        let mut ids: Vec = self
    +            .rows
    +            .lock()
    +            .unwrap()
    +            .values()
    +            .filter(|row| row.store.store_kind == store_kind && row.store.store_root == store_root)
    +            .map(|row| row.store.store_id.clone())
    +            .collect();
    +        ids.sort();
    +        ids.dedup();
    +        Box::pin(async move { Ok(ids) })
    +    }
    +
    +    fn maintenance_claim<'a>(
    +        &'a self,
    +        _holder: &'a str,
    +        _stale_after: Duration,
    +    ) -> BoxedFuture<'a, Result, StorageError>> {
    +        unused()
    +    }
    +
    +    fn check_maintenance_claim(
    +        &self,
    +        stale_after: Duration,
    +    ) -> BoxedFuture<'_, Result> {
    +        self.maintenance_checks.lock().unwrap().push(stale_after);
             let held = *self.maintenance_held.lock().unwrap();
             let error = self.maintenance_error.lock().unwrap().clone();
             Box::pin(async move {
    @@ -666,13 +1088,22 @@ fn restore_body(extra: Value) -> Value {
     }
     
     async fn restore(fake: &Fake, body: Value) -> Result {
    +    restore_on(fake, &runtime(), &body).await
    +}
    +
    +/// `restore` on `runtime` rather than the tests' default one.
    +async fn restore_on(
    +    fake: &Fake,
    +    runtime: &BackupRuntime,
    +    body: &Value,
    +) -> Result {
         restore_table_from_backup(
             fake,
    -        &runtime(),
    +        runtime,
             ACCOUNT,
             REGION,
             &LimitsConfig::default(),
    -        &body,
    +        body,
         )
         .await
         .map(|(_, v)| v)
    @@ -3670,7 +4101,7 @@ fn fake_with_row(status: BackupStatus) -> Fake {
     }
     
     fn runtime_with_store() -> BackupRuntime {
    -    runtime().with_store(store())
    +    runtime_on(&Arc::new(MemStore::default()), store())
     }
     
     /// DescribeBackup answers from the catalog row when one exists, derived from
    @@ -4255,7 +4686,13 @@ async fn restore_over_a_row_refuses_before_the_permit() {
             assert!(fake.restores().is_empty());
             err
         }
    -    let configured = BackupRuntime::new(1).with_store(store());
    +    let configured = BackupRuntime::new(1).with_store(ConfiguredStore::new(
    +        store().store_kind,
    +        store().store_root,
    +        Arc::new(MemStore::default()) as Arc,
    +        None,
    +        Ok(store()),
    +    ));
         let _held_configured = JobPermit::try_acquire(configured.slots()).unwrap();
         let none = BackupRuntime::new(1);
         let _held_none = JobPermit::try_acquire(none.slots()).unwrap();
    @@ -4304,10 +4741,16 @@ async fn restore_over_a_row_refuses_before_the_permit() {
         assert!(msg.contains("store other than"), "{msg}");
         // A configured store that is not the row's: the row is of another store,
         // not of "the" store because one is configured.
    -    let other = BackupRuntime::new(1).with_store(StoreIdentity {
    -        store_id: "store-2".to_owned(),
    -        ..store()
    -    });
    +    let other = BackupRuntime::new(1).with_store(ConfiguredStore::new(
    +        store().store_kind,
    +        store().store_root,
    +        Arc::new(MemStore::default()) as Arc,
    +        None,
    +        Ok(StoreIdentity {
    +            store_id: "store-2".to_owned(),
    +            ..store()
    +        }),
    +    ));
         let _held_other = JobPermit::try_acquire(other.slots()).unwrap();
         let msg = validation_message(check(fake_with_row(BackupStatus::Available), &other).await);
         assert!(
    @@ -4735,93 +5178,556 @@ async fn create_backup_under_a_region_outside_the_arn_alphabet_names_the_setting
         assert_eq!(fake.row_reads.lock().unwrap().len(), 1, "no further read");
     }
     
    -// --- the reconciler -------------------------------------------------------------
    -
    -fn dead_job(kind: extenddb_storage::backup_catalog::RestoreKind, name: &str) -> RestoreJobRow {
    -    RestoreJobRow {
    -        target: RestoreTargetKey {
    -            account_id: ACCOUNT.to_owned(),
    -            table_name: name.to_owned(),
    -            table_id: format!("tid-{name}"),
    -        },
    -        source_backup_arn: arn(ACCOUNT),
    -        kind,
    -        status: extenddb_storage::backup_catalog::JobStatus::Running,
    -        owner_generation: 0,
    -        heartbeat_at: 0.0,
    -    }
    -}
    -
    -/// A legacy restore whose owner is dead is marked FAILED and its target
    -/// removed; a manifest restore is left for its resumer; an abandoned
    -/// in-catalog CREATING backup is removed through the reader's delete.
    +/// While a maintenance command holds the catalog's claim, CreateBackup and
    +/// RestoreTableFromBackup are refused with LimitExceeded naming the command,
    +/// after every other check (a missing table is still TableNotFound) and
    +/// before the permit (no slot is taken, and the backend is not asked); once
    +/// the claim is gone both run.
     #[tokio::test]
    -async fn the_reconciler_fails_dead_legacy_restores_and_removes_abandoned_backups() {
    -    use extenddb_storage::backup_catalog::RestoreKind;
    -    let fake = Fake::default();
    -    *fake.target_still_creating.lock().unwrap() = true;
    -    *fake.record_still_running.lock().unwrap() = true;
    -    fake.dead_jobs.lock().unwrap().extend([
    -        dead_job(RestoreKind::Legacy, "LegacyTarget"),
    -        dead_job(RestoreKind::Manifest, "ManifestTarget"),
    -    ]);
    -    fake.abandoned_backups
    -        .lock()
    -        .unwrap()
    -        .push(AbandonedLegacyBackup {
    -            account_id: ACCOUNT.to_owned(),
    -            backup_arn: arn(ACCOUNT),
    -        });
    -    fake.descriptions
    -        .lock()
    -        .unwrap()
    -        .insert(arn(ACCOUNT), backup_description("CREATING", "PROVISIONED"));
    -
    -    let report = reconciler::cycle(
    +async fn jobs_are_refused_while_a_maintenance_claim_is_held() {
    +    let fake = Fake::with_source(source(Some(definition("PROVISIONED"))));
    +    *fake.maintenance_held.lock().unwrap() = true;
    +    let runtime = BackupRuntime::new(1).with_stale_job_after(Duration::from_secs(7));
    +    let err = create_backup(
             &fake,
    -        Duration::from_secs(300),
    -        &mut reconciler::ReconcilerState::default(),
    +        &runtime,
    +        ACCOUNT,
    +        REGION,
    +        "Music",
    +        "nightly",
    +        Ok(active_table("Music")),
         )
         .await
    -    .unwrap();
    -    assert!(!report.maintenance_held);
    -    assert_eq!(report.legacy_restores_removed, ["LegacyTarget"]);
    -    assert!(report.failed_targets_removed.is_empty());
    -    assert_eq!(report.manifest_restores_seen, ["ManifestTarget"]);
    -    assert_eq!(report.manifest_restores_logged, ["ManifestTarget"]);
    -    assert_eq!(report.legacy_backups_removed, [arn(ACCOUNT)]);
    +    .unwrap_err();
    +    assert!(
    +        matches!(&err, DynamoDbError::LimitExceededException(m) if m.contains("backup purge")),
    +        "{err:?}"
    +    );
    +    assert!(fake.creates.lock().unwrap().is_empty());
    +    assert_eq!(runtime.free_slots(), 1, "no slot was taken");
    +    let err = create_backup(
    +        &fake,
    +        &runtime,
    +        ACCOUNT,
    +        REGION,
    +        "Music",
    +        "nightly",
    +        Err(StorageError::TableNotFound("Music".to_owned())),
    +    )
    +    .await
    +    .unwrap_err();
    +    assert!(
    +        matches!(err, DynamoDbError::TableNotFoundException(_)),
    +        "the table check comes first: {err:?}"
    +    );
     
    -    let failed = fake.failed_jobs.lock().unwrap().clone();
    -    assert_eq!(failed.len(), 1);
    -    assert_eq!(failed[0].0.table_name, "LegacyTarget");
    -    assert_eq!(failed[0].1, reconciler::OWNER_DEAD);
    +    let err = restore_on(&fake, &runtime, &restore_body(json!({})))
    +        .await
    +        .unwrap_err();
    +    assert!(
    +        matches!(&err, DynamoDbError::LimitExceededException(m) if m.contains("backup purge")),
    +        "{err:?}"
    +    );
    +    assert!(fake.restores().is_empty(), "the backend was not asked");
    +    assert_eq!(runtime.free_slots(), 1);
         assert_eq!(
    -        fake.removed_targets.lock().unwrap().as_slice(),
    -        [dead_job(RestoreKind::Legacy, "LegacyTarget").target]
    +        *fake.maintenance_checks.lock().unwrap(),
    +        vec![Duration::from_secs(7); 2],
    +        "the claim is read with stale_job_after_secs"
         );
    -    assert_eq!(fake.deleted.lock().unwrap().as_slice(), [arn(ACCOUNT)]);
    -}
     
    -/// A record that is no longer RUNNING when the reconciler reaches it belongs
    -/// to whoever moved it: the target is not removed.
    -#[tokio::test]
    -async fn the_reconciler_leaves_a_target_whose_record_moved() {
    -    use extenddb_storage::backup_catalog::RestoreKind;
    -    let fake = Fake::default();
    -    *fake.record_still_running.lock().unwrap() = false;
    -    *fake.target_still_creating.lock().unwrap() = true;
    -    fake.dead_jobs
    -        .lock()
    -        .unwrap()
    -        .push(dead_job(RestoreKind::Legacy, "LegacyTarget"));
    -    let report = reconciler::cycle(
    +    *fake.maintenance_held.lock().unwrap() = false;
    +    create_backup(
             &fake,
    -        Duration::from_secs(300),
    -        &mut reconciler::ReconcilerState::default(),
    +        &runtime,
    +        ACCOUNT,
    +        REGION,
    +        "Music",
    +        "nightly",
    +        Ok(active_table("Music")),
         )
         .await
         .unwrap();
    -    assert!(report.legacy_restores_removed.is_empty());
    +    fake.permits.lock().unwrap().clear();
    +    restore_on(&fake, &runtime, &restore_body(json!({})))
    +        .await
    +        .unwrap();
    +    assert_eq!(fake.creates.lock().unwrap().len(), 1);
    +    assert_eq!(fake.restores().len(), 1);
    +
    +    // Before the permit: with every slot held and the claim held, the
    +    // refusal names the command, not the slot limit, so the check cannot
    +    // have run after `take_permit`.
    +    *fake.maintenance_held.lock().unwrap() = true;
    +    let full = BackupRuntime::new(1).with_stale_job_after(Duration::from_secs(7));
    +    let held_slot = full.take_permit().unwrap();
    +    assert_eq!(full.free_slots(), 0, "every slot is held");
    +    let err = create_backup(
    +        &fake,
    +        &full,
    +        ACCOUNT,
    +        REGION,
    +        "Music",
    +        "nightly",
    +        Ok(active_table("Music")),
    +    )
    +    .await
    +    .unwrap_err();
    +    assert!(
    +        matches!(&err, DynamoDbError::LimitExceededException(m) if m.contains("backup purge")),
    +        "the claim is checked before the permit: {err:?}"
    +    );
    +    let err = restore_on(&fake, &full, &restore_body(json!({})))
    +        .await
    +        .unwrap_err();
    +    assert!(
    +        matches!(&err, DynamoDbError::LimitExceededException(m) if m.contains("backup purge")),
    +        "the claim is checked before the permit: {err:?}"
    +    );
    +    drop(held_slot);
    +}
    +
    +// --- purge: the wait and the row deletion (`maintenance`) ----------------------
    +
    +/// A maintenance claim handle for `HeldClaim`: heartbeats confirm until
    +/// `lost` is set.
    +#[derive(Default)]
    +struct FakeHolder {
    +    lost: Mutex,
    +    released: Mutex,
    +}
    +
    +impl extenddb_storage::MaintenanceClaim for FakeHolder {
    +    fn generation(&self) -> u64 {
    +        11
    +    }
    +    fn heartbeat(
    +        &self,
    +    ) -> BoxedFuture<'_, Result> {
    +        let lost = *self.lost.lock().unwrap();
    +        Box::pin(async move {
    +            Ok(if lost {
    +                extenddb_storage::backup_catalog::Heartbeat::Lost
    +            } else {
    +                extenddb_storage::backup_catalog::Heartbeat::Confirmed
    +            })
    +        })
    +    }
    +    fn release(&self) -> BoxedFuture<'_, Result<(), StorageError>> {
    +        *self.released.lock().unwrap() = true;
    +        Box::pin(async { Ok(()) })
    +    }
    +}
    +
    +/// A forwarding handle, so the test keeps its own reference to the holder.
    +struct Forward(Arc);
    +
    +impl extenddb_storage::MaintenanceClaim for Forward {
    +    fn generation(&self) -> u64 {
    +        self.0.generation()
    +    }
    +    fn heartbeat(
    +        &self,
    +    ) -> BoxedFuture<'_, Result> {
    +        self.0.heartbeat()
    +    }
    +    fn release(&self) -> BoxedFuture<'_, Result<(), StorageError>> {
    +        self.0.release()
    +    }
    +}
    +
    +fn held(holder: &Arc) -> maintenance::HeldClaim {
    +    maintenance::HeldClaim::start(
    +        Box::new(Forward(Arc::clone(holder))),
    +        Duration::from_secs(1),
    +    )
    +}
    +
    +fn seconds_since_epoch() -> f64 {
    +    std::time::SystemTime::now()
    +        .duration_since(std::time::UNIX_EPOCH)
    +        .unwrap()
    +        .as_secs_f64()
    +}
    +
    +/// A row of `store()` under ARN `arn` in `status`, with a manifest and a
    +/// data file under its prefix in `mem`.
    +fn seed(fake: &Fake, mem: &MemStore, arn: &str, status: BackupStatus) {
    +    let prefix = BackupArn::parse(arn).unwrap().store_prefix();
    +    mem.insert(&format!("{prefix}extenddb-manifest.json"), b"{}");
    +    mem.insert(&format!("{prefix}data/00001.json.gz"), b"x");
    +    fake.rows
    +        .lock()
    +        .unwrap()
    +        .insert(arn.to_owned(), row(arn, status));
    +}
    +
    +fn arn_n(n: u32) -> String {
    +    format!("arn:aws:dynamodb:{REGION}:{ACCOUNT}:table/Music/backup/0179197643143{n}-0000000{n}")
    +}
    +
    +/// The states a row can be in when purge reaches it after reading it
    +/// AVAILABLE, a DeleteBackup having run meanwhile: still AVAILABLE; moved to
    +/// DELETING (the other deleter mid-way); moved to DELETED (the other
    +/// finished); gone (finished and its tombstone removed). Each ends with no
    +/// row and no object, counted as deleted; a status that cannot follow
    +/// AVAILABLE (FAILED) is the one case left for a look. A tombstone read
    +/// and removed meanwhile (a second purge) counts as deleted too.
    +#[tokio::test]
    +async fn purge_finishes_a_row_another_deleter_moved_or_removed_meanwhile() {
    +    let fake = Fake::default();
    +    let mem = MemStore::default();
    +    let arns: Vec = (1..=6).map(arn_n).collect();
    +    for arn in &arns[..5] {
    +        seed(&fake, &mem, arn, BackupStatus::Available);
    +    }
    +    // A DELETED tombstone the command read, whose prefix is already empty.
    +    fake.rows
    +        .lock()
    +        .unwrap()
    +        .insert(arns[5].clone(), row(&arns[5], BackupStatus::Deleted));
    +    // The command's read under the claim.
    +    let read: Vec = fake
    +        .list_backup_rows(None, &store())
    +        .try_collect()
    +        .await
    +        .unwrap();
    +    assert_eq!(read.len(), 6);
    +    // Between the read and the rows' turns, DeleteBackup moves rows 2 and 3
    +    // and finishes 4 (its tombstone removed too); a writer moves 5 to
    +    // FAILED, which no DeleteBackup does; the tombstone 6 is removed by a
    +    // second purge.
    +    {
    +        let mut rows = fake.rows.lock().unwrap();
    +        rows.get_mut(&arns[1]).unwrap().status = BackupStatus::Deleting;
    +        rows.get_mut(&arns[2]).unwrap().status = BackupStatus::Deleted;
    +        rows.remove(&arns[3]);
    +        rows.get_mut(&arns[4]).unwrap().status = BackupStatus::Failed;
    +        rows.remove(&arns[5]);
    +    }
    +    // The deleter that finished rows 3 and 4 deleted their objects.
    +    for arn in [&arns[2], &arns[3]] {
    +        let prefix = BackupArn::parse(arn).unwrap().store_prefix();
    +        extenddb_backup_store::BackupStore::delete_prefix(&mem, &prefix)
    +            .await
    +            .unwrap();
    +    }
    +
    +    let mut outcomes = Vec::new();
    +    for row in &read {
    +        outcomes.push(maintenance::purge_row(&fake, &mem, row, Duration::from_secs(300)).await);
    +    }
    +    assert_eq!(
    +        outcomes[0],
    +        Ok(()),
    +        "still AVAILABLE: moved and deleted here"
    +    );
    +    assert_eq!(
    +        outcomes[1],
    +        Ok(()),
    +        "moved to DELETING by another deleter: finished here"
    +    );
    +    assert_eq!(
    +        outcomes[2],
    +        Ok(()),
    +        "moved to DELETED by another deleter: the tombstone removed here, counted as deleted"
    +    );
    +    assert_eq!(outcomes[3], Ok(()), "gone: counted as deleted");
    +    assert_eq!(
    +        outcomes[4],
    +        Err("its status changed to FAILED".to_owned()),
    +        "a status that cannot follow AVAILABLE is left"
    +    );
    +    assert_eq!(
    +        outcomes[5],
    +        Ok(()),
    +        "a tombstone removed meanwhile: counted as deleted"
    +    );
    +    let left: Vec = fake.rows.lock().unwrap().keys().cloned().collect();
    +    assert_eq!(left, vec![arns[4].clone()], "no row of the five remains");
    +    let failed_prefix = BackupArn::parse(&arns[4]).unwrap().store_prefix();
    +    assert_eq!(
    +        mem.keys(),
    +        vec![
    +            format!("{failed_prefix}data/00001.json.gz"),
    +            format!("{failed_prefix}extenddb-manifest.json"),
    +        ],
    +        "only the left row's objects remain"
    +    );
    +    // The row moved to DELETED was never moved by purge (DELETING to
    +    // DELETED is the other deleter's write); the one left DELETING and the
    +    // one still AVAILABLE were finished by purge; the three tombstones were
    +    // removed here.
    +    assert_eq!(
    +        *fake.set_deleted.lock().unwrap(),
    +        vec![arns[0].clone(), arns[1].clone()]
    +    );
    +    assert_eq!(
    +        *fake.removed_rows.lock().unwrap(),
    +        vec![arns[0].clone(), arns[1].clone(), arns[2].clone()]
    +    );
    +}
    +
    +/// A CREATING row with a dead owner goes through the dead-owner removal and
    +/// then its prefix; one whose heartbeat is young is left with its objects
    +/// under the live window, and removed under the zero window SQLite uses.
    +#[tokio::test]
    +async fn purge_removes_a_dead_creating_row_and_leaves_a_live_one() {
    +    let fake = Fake::default();
    +    let mem = MemStore::default();
    +    let dead = arn_n(1);
    +    let live = arn_n(2);
    +    seed(&fake, &mem, &dead, BackupStatus::Creating);
    +    seed(&fake, &mem, &live, BackupStatus::Creating);
    +    fake.rows
    +        .lock()
    +        .unwrap()
    +        .get_mut(&live)
    +        .unwrap()
    +        .heartbeat_at = seconds_since_epoch();
    +    let window = Duration::from_secs(300);
    +    let dead_row = fake.rows.lock().unwrap()[&dead].clone();
    +    let live_row = fake.rows.lock().unwrap()[&live].clone();
    +    assert_eq!(
    +        maintenance::purge_row(&fake, &mem, &dead_row, window).await,
    +        Ok(())
    +    );
    +    let err = maintenance::purge_row(&fake, &mem, &live_row, window)
    +        .await
    +        .unwrap_err();
    +    assert!(
    +        err.contains("the CREATING row could not be removed"),
    +        "{err}"
    +    );
    +    let left: Vec = fake.rows.lock().unwrap().keys().cloned().collect();
    +    assert_eq!(left, vec![live.clone()]);
    +    assert_eq!(mem.keys().len(), 2, "the live row's objects stay");
    +    // A heartbeat written a moment ago is older than a zero window once a
    +    // moment has passed.
    +    tokio::time::sleep(Duration::from_millis(20)).await;
    +    assert_eq!(
    +        maintenance::purge_row(&fake, &mem, &live_row, Duration::ZERO).await,
    +        Ok(())
    +    );
    +    assert!(fake.rows.lock().unwrap().is_empty());
    +    assert!(mem.keys().is_empty());
    +}
    +
    +/// The wait's five outcomes are told apart: quiet, also with a FAILED
    +/// restore record whose target is still CREATING (the reconciler's duty,
    +/// not a running job) and a FAILED backup row whose owner is dead; busy at
    +/// the limit, with the jobs named (a CREATING row and a FAILED row with a
    +/// live heartbeat, a RUNNING restore) and announced once; a signal, during
    +/// a sleep and before the first read; the catalog unreadable; the claim
    +/// lost.
    +#[tokio::test(start_paused = true)]
    +async fn the_wait_reports_each_outcome_as_itself() {
    +    use extenddb_storage::backup_catalog::{JobStatus, RestoreKind};
    +    use maintenance::{Interrupt, Wait, WaitOutcome, wait_for_quiet};
    +    let fake = Fake::default();
    +    let holder = Arc::new(FakeHolder::default());
    +    let claim = held(&holder);
    +    let limit = Duration::from_secs(20);
    +    let wait = Wait {
    +        owner_window: Duration::from_secs(10),
    +        limit,
    +        poll: Duration::from_secs(2),
    +    };
    +    let announced: Mutex>> = Mutex::new(Vec::new());
    +    let announce = |jobs: &[String]| announced.lock().unwrap().push(jobs.to_vec());
    +
    +    let (_tx, mut quiet) = Interrupt::new();
    +    assert_eq!(
    +        wait_for_quiet(&fake, &store(), &claim, wait, &mut quiet, announce).await,
    +        WaitOutcome::Quiet
    +    );
    +    // A FAILED record whose target is still CREATING is listed by the
    +    // zero-window dead-job listing with the RUNNING ones; it is the
    +    // reconciler's to finish and no job runs under it. A FAILED backup row
    +    // whose heartbeat is older than the window has a dead owner.
    +    let mut failed_record = dead_job(RestoreKind::Legacy, "FailedTarget");
    +    failed_record.status = JobStatus::Failed;
    +    fake.dead_jobs.lock().unwrap().push(failed_record);
    +    fake.rows
    +        .lock()
    +        .unwrap()
    +        .insert(arn_n(3), row(&arn_n(3), BackupStatus::Failed));
    +    let started = tokio::time::Instant::now();
    +    assert_eq!(
    +        wait_for_quiet(&fake, &store(), &claim, wait, &mut quiet, announce).await,
    +        WaitOutcome::Quiet,
    +        "a FAILED restore record and a FAILED row with a dead owner are not waited on"
    +    );
    +    assert_eq!(started.elapsed(), Duration::ZERO, "quiet at the first read");
    +    assert!(announced.lock().unwrap().is_empty());
    +
    +    // Busy: a CREATING row and a FAILED row whose heartbeats are young, and
    +    // a RUNNING restore.
    +    for (n, status) in [(1, BackupStatus::Creating), (2, BackupStatus::Failed)] {
    +        fake.rows.lock().unwrap().insert(arn_n(n), {
    +            let mut r = row(&arn_n(n), status);
    +            r.heartbeat_at = seconds_since_epoch();
    +            r
    +        });
    +    }
    +    fake.dead_jobs
    +        .lock()
    +        .unwrap()
    +        .push(dead_job(RestoreKind::Legacy, "Target"));
    +    let started = tokio::time::Instant::now();
    +    let busy = wait_for_quiet(&fake, &store(), &claim, wait, &mut quiet, announce).await;
    +    let expected = vec![
    +        format!(
    +            "CreateBackup {} has a live owner (heartbeat within stale_job_after_secs)",
    +            arn_n(1)
    +        ),
    +        format!(
    +            "the FAILED backup {} has a live owner (heartbeat within stale_job_after_secs)",
    +            arn_n(2)
    +        ),
    +        format!("a restore into Target (account {ACCOUNT}) is RUNNING"),
    +    ];
    +    assert_eq!(busy, WaitOutcome::Busy(expected.clone()));
    +    assert!(started.elapsed() >= limit, "{:?}", started.elapsed());
    +    assert_eq!(*announced.lock().unwrap(), vec![expected]);
    +
    +    // A signal during a sleep ends the wait before the limit.
    +    let (tx, mut interrupted) = Interrupt::new();
    +    tokio::spawn(async move {
    +        tokio::time::sleep(Duration::from_secs(3)).await;
    +        let _ = tx.send(Some("SIGTERM"));
    +    });
    +    let started = tokio::time::Instant::now();
    +    assert_eq!(
    +        wait_for_quiet(&fake, &store(), &claim, wait, &mut interrupted, announce).await,
    +        WaitOutcome::Interrupted("SIGTERM")
    +    );
    +    assert!(
    +        started.elapsed() < Duration::from_millis(3_500),
    +        "the signal ends the sleep, not the next poll: {:?}",
    +        started.elapsed()
    +    );
    +    // One already received ends it before the first read.
    +    assert_eq!(interrupted.signalled(), Some("SIGTERM"));
    +    assert_eq!(
    +        wait_for_quiet(&fake, &store(), &claim, wait, &mut interrupted, announce).await,
    +        WaitOutcome::Interrupted("SIGTERM")
    +    );
    +
    +    // The catalog unreadable.
    +    *fake.list_error.lock().unwrap() = Some("pool exhausted".to_owned());
    +    assert_eq!(
    +        wait_for_quiet(&fake, &store(), &claim, wait, &mut quiet, announce).await,
    +        WaitOutcome::CatalogError(
    +            "cannot list the catalog's backup rows: Internal error: pool exhausted".to_owned()
    +        )
    +    );
    +    *fake.list_error.lock().unwrap() = None;
    +
    +    // The claim lost: the next heartbeat records it, and the wait stops.
    +    *holder.lost.lock().unwrap() = true;
    +    tokio::time::sleep(Duration::from_millis(1_500)).await;
    +    assert_eq!(
    +        wait_for_quiet(&fake, &store(), &claim, wait, &mut quiet, announce).await,
    +        WaitOutcome::Lost(
    +            "the maintenance claim was lost (another holder took it over)".to_owned()
    +        )
    +    );
    +    claim.release().await.unwrap();
    +    assert!(*holder.released.lock().unwrap());
    +}
    +
    +// --- the reconciler -------------------------------------------------------------
    +
    +fn dead_job(kind: extenddb_storage::backup_catalog::RestoreKind, name: &str) -> RestoreJobRow {
    +    RestoreJobRow {
    +        target: RestoreTargetKey {
    +            account_id: ACCOUNT.to_owned(),
    +            table_name: name.to_owned(),
    +            table_id: format!("tid-{name}"),
    +        },
    +        source_backup_arn: arn(ACCOUNT),
    +        kind,
    +        status: extenddb_storage::backup_catalog::JobStatus::Running,
    +        owner_generation: 0,
    +        heartbeat_at: 0.0,
    +    }
    +}
    +
    +/// A legacy restore whose owner is dead is marked FAILED and its target
    +/// removed; a manifest restore is left for its resumer; an abandoned
    +/// in-catalog CREATING backup is removed through the reader's delete.
    +#[tokio::test]
    +async fn the_reconciler_fails_dead_legacy_restores_and_removes_abandoned_backups() {
    +    use extenddb_storage::backup_catalog::RestoreKind;
    +    let fake = Fake::default();
    +    *fake.target_still_creating.lock().unwrap() = true;
    +    *fake.record_still_running.lock().unwrap() = true;
    +    fake.dead_jobs.lock().unwrap().extend([
    +        dead_job(RestoreKind::Legacy, "LegacyTarget"),
    +        dead_job(RestoreKind::Manifest, "ManifestTarget"),
    +    ]);
    +    fake.abandoned_backups
    +        .lock()
    +        .unwrap()
    +        .push(AbandonedLegacyBackup {
    +            account_id: ACCOUNT.to_owned(),
    +            backup_arn: arn(ACCOUNT),
    +        });
    +    fake.descriptions
    +        .lock()
    +        .unwrap()
    +        .insert(arn(ACCOUNT), backup_description("CREATING", "PROVISIONED"));
    +
    +    let report = reconciler::cycle(
    +        &fake,
    +        &runtime(),
    +        Duration::from_secs(300),
    +        "test",
    +        &mut reconciler::ReconcilerState::default(),
    +    )
    +    .await
    +    .unwrap();
    +    assert!(!report.maintenance_held);
    +    assert_eq!(report.legacy_restores_removed, ["LegacyTarget"]);
    +    assert!(report.failed_targets_removed.is_empty());
    +    assert_eq!(report.manifest_restores_seen, ["ManifestTarget"]);
    +    assert_eq!(report.manifest_restores_logged, ["ManifestTarget"]);
    +    assert_eq!(report.legacy_backups_removed, [arn(ACCOUNT)]);
    +
    +    let failed = fake.failed_jobs.lock().unwrap().clone();
    +    assert_eq!(failed.len(), 1);
    +    assert_eq!(failed[0].0.table_name, "LegacyTarget");
    +    assert_eq!(failed[0].1, reconciler::OWNER_DEAD);
    +    assert_eq!(
    +        fake.removed_targets.lock().unwrap().as_slice(),
    +        [dead_job(RestoreKind::Legacy, "LegacyTarget").target]
    +    );
    +    assert_eq!(fake.deleted.lock().unwrap().as_slice(), [arn(ACCOUNT)]);
    +}
    +
    +/// A record that is no longer RUNNING when the reconciler reaches it belongs
    +/// to whoever moved it: the target is not removed.
    +#[tokio::test]
    +async fn the_reconciler_leaves_a_target_whose_record_moved() {
    +    use extenddb_storage::backup_catalog::RestoreKind;
    +    let fake = Fake::default();
    +    *fake.record_still_running.lock().unwrap() = false;
    +    *fake.target_still_creating.lock().unwrap() = true;
    +    fake.dead_jobs
    +        .lock()
    +        .unwrap()
    +        .push(dead_job(RestoreKind::Legacy, "LegacyTarget"));
    +    let report = reconciler::cycle(
    +        &fake,
    +        &runtime(),
    +        Duration::from_secs(300),
    +        "test",
    +        &mut reconciler::ReconcilerState::default(),
    +    )
    +    .await
    +    .unwrap();
    +    assert!(report.legacy_restores_removed.is_empty());
         assert!(report.targets_left_to_owner.is_empty());
         assert_eq!(fake.failed_jobs.lock().unwrap().len(), 1);
         assert!(fake.removed_targets.lock().unwrap().is_empty());
    @@ -4848,7 +5754,9 @@ async fn the_reconciler_leaves_a_target_that_is_no_longer_creating_after_the_fai
             .extend([dead.clone(), failed.clone()]);
         let report = reconciler::cycle(
             &fake,
    +        &runtime(),
             Duration::from_secs(300),
    +        "test",
             &mut reconciler::ReconcilerState::default(),
         )
         .await
    @@ -4873,11 +5781,16 @@ async fn the_reconciler_leaves_a_target_that_is_no_longer_creating_after_the_fai
     }
     
     /// The maintenance claim is checked first and reported; the duties that
    -/// resolve dead owners still run while it is held.
    +/// resolve dead owners and the DELETING rows of the configured store still
    +/// run while it is held (the command waits on the first and finishes the
    +/// second itself, so neither races it).
     #[tokio::test]
     async fn the_reconciler_reports_a_held_maintenance_claim_and_still_resolves_dead_owners() {
         use extenddb_storage::backup_catalog::RestoreKind;
    -    let fake = Fake::default();
    +    let mem = Arc::new(MemStore::default());
    +    mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1"));
    +    mem.insert(&manifest_key(&arn(ACCOUNT)), b"{}");
    +    let fake = fake_with_rows(&[(arn(ACCOUNT), BackupStatus::Deleting)]);
         *fake.maintenance_held.lock().unwrap() = true;
         *fake.target_still_creating.lock().unwrap() = true;
         *fake.record_still_running.lock().unwrap() = true;
    @@ -4887,13 +5800,25 @@ async fn the_reconciler_reports_a_held_maintenance_claim_and_still_resolves_dead
             .push(dead_job(RestoreKind::Legacy, "LegacyTarget"));
         let report = reconciler::cycle(
             &fake,
    +        &runtime_on(&mem, store()),
             Duration::from_secs(300),
    +        "test",
             &mut reconciler::ReconcilerState::default(),
         )
         .await
         .unwrap();
         assert!(report.maintenance_held);
         assert_eq!(report.legacy_restores_removed, ["LegacyTarget"]);
    +    assert_eq!(
    +        report.deletions_finished,
    +        [arn(ACCOUNT)],
    +        "a DELETING row is finished while the claim is held"
    +    );
    +    assert_eq!(
    +        fake.rows.lock().unwrap()[&arn(ACCOUNT)].status,
    +        BackupStatus::Deleted
    +    );
    +    assert_eq!(mem.keys(), [STORE_IDENTITY_KEY]);
     }
     
     /// The worker runs a cycle at once (observed through the maintenance check,
    @@ -4908,15 +5833,17 @@ async fn the_reconciler_worker_runs_at_startup_and_stops_on_shutdown() {
         let shutdown = extenddb_storage::CancellationToken::new();
         let worker = tokio::spawn(reconciler::run(
             Arc::clone(&fake),
    +        Arc::new(runtime()),
             reconciler::ReconcilerSettings {
                 interval: Duration::from_secs(3600),
                 stale_job_after: Duration::from_secs(300),
             },
    +        "test".to_owned(),
             Arc::clone(&metrics),
             shutdown.clone(),
         ));
         let deadline = tokio::time::Instant::now() + Duration::from_secs(5);
    -    while *fake.maintenance_checks.lock().unwrap() == 0 {
    +    while fake.maintenance_checks.lock().unwrap().is_empty() {
             assert!(
                 tokio::time::Instant::now() < deadline,
                 "the first cycle did not run"
    @@ -4924,7 +5851,7 @@ async fn the_reconciler_worker_runs_at_startup_and_stops_on_shutdown() {
             tokio::time::sleep(Duration::from_millis(5)).await;
         }
         assert_eq!(
    -        *fake.maintenance_checks.lock().unwrap(),
    +        fake.maintenance_checks.lock().unwrap().len(),
             1,
             "one cycle before the first interval"
         );
    @@ -4933,7 +5860,7 @@ async fn the_reconciler_worker_runs_at_startup_and_stops_on_shutdown() {
             .await
             .expect("the worker stops at shutdown")
             .unwrap();
    -    assert_eq!(*fake.maintenance_checks.lock().unwrap(), 1);
    +    assert_eq!(fake.maintenance_checks.lock().unwrap().len(), 1);
         let recorded: Vec = metrics
             .drain(Duration::ZERO)
             .into_iter()
    @@ -4959,15 +5886,17 @@ async fn a_failed_reconciler_cycle_records_a_worker_error() {
         let shutdown = extenddb_storage::CancellationToken::new();
         let worker = tokio::spawn(reconciler::run(
             Arc::clone(&fake),
    +        Arc::new(runtime()),
             reconciler::ReconcilerSettings {
                 interval: Duration::from_millis(5),
                 stale_job_after: Duration::from_secs(300),
             },
    +        "test".to_owned(),
             Arc::clone(&metrics),
             shutdown.clone(),
         ));
         let deadline = tokio::time::Instant::now() + Duration::from_secs(5);
    -    while *fake.maintenance_checks.lock().unwrap() < 2 {
    +    while fake.maintenance_checks.lock().unwrap().len() < 2 {
             assert!(
                 tokio::time::Instant::now() < deadline,
                 "the worker did not go on after a failed cycle"
    @@ -5008,7 +5937,9 @@ async fn the_reconciler_removes_the_target_of_a_failed_record_without_writing_fa
             .extend([failed.clone(), failed_manifest.clone()]);
         let report = reconciler::cycle(
             &fake,
    +        &runtime(),
             Duration::from_secs(300),
    +        "test",
             &mut reconciler::ReconcilerState::default(),
         )
         .await
    @@ -5040,14 +5971,26 @@ async fn a_dead_manifest_record_is_logged_once_per_record() {
             .unwrap()
             .push(dead_job(RestoreKind::Manifest, "ManifestTarget"));
         let mut state = reconciler::ReconcilerState::default();
    -    let first = reconciler::cycle(&fake, Duration::from_secs(300), &mut state)
    -        .await
    -        .unwrap();
    +    let first = reconciler::cycle(
    +        &fake,
    +        &runtime(),
    +        Duration::from_secs(300),
    +        "test",
    +        &mut state,
    +    )
    +    .await
    +    .unwrap();
         assert_eq!(first.manifest_restores_seen, ["ManifestTarget"]);
         assert_eq!(first.manifest_restores_logged, ["ManifestTarget"]);
    -    let second = reconciler::cycle(&fake, Duration::from_secs(300), &mut state)
    -        .await
    -        .unwrap();
    +    let second = reconciler::cycle(
    +        &fake,
    +        &runtime(),
    +        Duration::from_secs(300),
    +        "test",
    +        &mut state,
    +    )
    +    .await
    +    .unwrap();
         assert_eq!(second.manifest_restores_seen, ["ManifestTarget"]);
         assert!(
             second.manifest_restores_logged.is_empty(),
    @@ -5057,9 +6000,15 @@ async fn a_dead_manifest_record_is_logged_once_per_record() {
             .lock()
             .unwrap()
             .push(dead_job(RestoreKind::Manifest, "Another"));
    -    let third = reconciler::cycle(&fake, Duration::from_secs(300), &mut state)
    -        .await
    -        .unwrap();
    +    let third = reconciler::cycle(
    +        &fake,
    +        &runtime(),
    +        Duration::from_secs(300),
    +        "test",
    +        &mut state,
    +    )
    +    .await
    +    .unwrap();
         assert_eq!(third.manifest_restores_logged, ["Another"]);
         assert!(
             fake.failed_jobs.lock().unwrap().is_empty()
    @@ -5067,3 +6016,1266 @@ async fn a_dead_manifest_record_is_logged_once_per_record() {
             "a manifest record is never acted on by this release"
         );
     }
    +
    +// --- the store identity file and the store's health ---------------------------
    +
    +const MANIFEST_KEY_SUFFIX: &str = "extenddb-manifest.json";
    +
    +/// The manifest key of `arn`'s prefix.
    +fn manifest_key(arn: &str) -> String {
    +    format!(
    +        "{}{MANIFEST_KEY_SUFFIX}",
    +        BackupArn::parse(arn).unwrap().store_prefix()
    +    )
    +}
    +
    +/// An ACTIVE table description, the probe CreateBackup's handler ran.
    +fn active_table(name: &str) -> TableDescription {
    +    TableDescription {
    +        table_name: name.to_owned(),
    +        table_status: TableStatus::Active,
    +        ..Default::default()
    +    }
    +}
    +
    +/// A second, newer backup of the `Music` table.
    +fn newer_arn(account: &str) -> String {
    +    format!("arn:aws:dynamodb:{REGION}:{account}:table/Music/backup/01791976999999-0badf00d")
    +}
    +
    +fn fake_with_rows(rows: &[(String, BackupStatus)]) -> Fake {
    +    let fake = Fake::default();
    +    for (arn, status) in rows {
    +        fake.rows
    +            .lock()
    +            .unwrap()
    +            .insert(arn.clone(), row(arn, *status));
    +    }
    +    fake
    +}
    +
    +/// An empty root and a catalog with no rows for it: startup creates the file
    +/// with this catalog's id and a fresh store id, re-reads it and uses that id.
    +#[tokio::test]
    +async fn establish_creates_the_identity_file_on_an_empty_root() {
    +    let mem = MemStore::default();
    +    let fake = Fake::default();
    +    let identity = store_identity::establish(&mem, StoreKind::Filesystem, "/r", &fake, "test")
    +        .await
    +        .unwrap();
    +    assert_eq!(identity.store_kind, StoreKind::Filesystem);
    +    assert_eq!(identity.store_root, "/r");
    +    assert_eq!(mem.keys(), [STORE_IDENTITY_KEY]);
    +    let file = StoreIdentityFile::parse(&mem.objects.lock().unwrap()[STORE_IDENTITY_KEY]).unwrap();
    +    assert_eq!(file.store_id, identity.store_id);
    +    assert_eq!(file.catalog_id, "catalog-1");
    +    assert_eq!(file.backend, "test");
    +    assert_eq!(mem.puts.lock().unwrap().as_slice(), [STORE_IDENTITY_KEY]);
    +    // A second startup against the same root reads the file and uses its id.
    +    let again = store_identity::establish(&mem, StoreKind::Filesystem, "/r", &fake, "test")
    +        .await
    +        .unwrap();
    +    assert_eq!(again, identity);
    +    assert_eq!(
    +        mem.puts.lock().unwrap().len(),
    +        1,
    +        "the file is not rewritten"
    +    );
    +}
    +
    +/// Two instances of one catalog starting against one empty root: one creates
    +/// the file, the other finds it present, and both use the winner's id.
    +#[tokio::test]
    +async fn two_instances_on_one_empty_root_end_with_one_id() {
    +    let mem = Arc::new(MemStore::default());
    +    let fakes = [Fake::default(), Fake::default()];
    +    let (a, b) = tokio::join!(
    +        store_identity::establish(&*mem, StoreKind::S3, "s3://b/", &fakes[0], "test"),
    +        store_identity::establish(&*mem, StoreKind::S3, "s3://b/", &fakes[1], "test"),
    +    );
    +    let (a, b) = (a.unwrap(), b.unwrap());
    +    assert_eq!(a, b);
    +    assert_eq!(mem.keys(), [STORE_IDENTITY_KEY]);
    +    assert_eq!(
    +        mem.puts.lock().unwrap().len(),
    +        1,
    +        "one create stored its file"
    +    );
    +}
    +
    +/// A root whose file names another catalog is unhealthy, and stays so until
    +/// the file is restored; the problem names both catalogs and no root.
    +#[tokio::test]
    +async fn a_file_naming_another_catalog_is_unhealthy_until_restored() {
    +    let mem = MemStore::default();
    +    mem.insert(
    +        STORE_IDENTITY_KEY,
    +        &identity_file("store-1", "another-catalog"),
    +    );
    +    let fake = fake_with_row(BackupStatus::Available);
    +    let err = store_identity::establish(&mem, StoreKind::Filesystem, "/r", &fake, "test")
    +        .await
    +        .unwrap_err();
    +    assert_eq!(
    +        err,
    +        StoreProblem::ForeignCatalog {
    +            file_catalog_id: "another-catalog".to_owned(),
    +            catalog_id: "catalog-1".to_owned(),
    +        }
    +    );
    +    assert_eq!(err.category(), "owned by another catalog");
    +    assert!(!err.public_reason().contains("/r"));
    +    // Restored: the check passes.
    +    mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1"));
    +    mem.insert(&manifest_key(&arn(ACCOUNT)), b"{}");
    +    let identity = store_identity::recheck(&mem, &store(), &fake, "test").await;
    +    assert_eq!(identity, Ok(()));
    +}
    +
    +/// Something that is not an object at the identity file's key (a FIFO, a
    +/// socket): the create finds the key taken and the read finds no object, so
    +/// nothing was created and nothing can be read, and the problem says so
    +/// rather than claiming a file was created.
    +#[tokio::test]
    +async fn a_non_object_at_the_identity_key_is_neither_readable_nor_creatable() {
    +    let mem = MemStore::default();
    +    mem.key_taken_by_a_non_object
    +        .store(true, std::sync::atomic::Ordering::Release);
    +    let fake = Fake::default();
    +    let err = store_identity::establish(&mem, StoreKind::Filesystem, "/r", &fake, "test")
    +        .await
    +        .unwrap_err();
    +    assert_eq!(
    +        err,
    +        StoreProblem::Transport("extenddb-store.json is neither readable nor creatable".to_owned())
    +    );
    +    assert!(mem.keys().is_empty(), "nothing was created");
    +}
    +
    +/// The rows for the root carry an id the file does not, or more than one id,
    +/// or the root has no file while rows exist: each is unhealthy.
    +#[tokio::test]
    +async fn ids_that_disagree_or_a_missing_file_with_rows_are_unhealthy() {
    +    // The file holds another id than the rows.
    +    let mem = MemStore::default();
    +    mem.insert(STORE_IDENTITY_KEY, &identity_file("store-2", "catalog-1"));
    +    let fake = fake_with_row(BackupStatus::Available);
    +    let err = store_identity::establish(
    +        &mem,
    +        StoreKind::Filesystem,
    +        store().store_root.as_str(),
    +        &fake,
    +        "test",
    +    )
    +    .await
    +    .unwrap_err();
    +    assert_eq!(
    +        err,
    +        StoreProblem::IdMismatch {
    +            file_id: "store-2".to_owned(),
    +            row_ids: vec!["store-1".to_owned()],
    +        }
    +    );
    +    assert_eq!(err.category(), "identity mismatch");
    +
    +    // No file, rows exist.
    +    let mem = MemStore::default();
    +    let err = store_identity::establish(
    +        &mem,
    +        StoreKind::Filesystem,
    +        store().store_root.as_str(),
    +        &fake,
    +        "test",
    +    )
    +    .await
    +    .unwrap_err();
    +    assert_eq!(
    +        err,
    +        StoreProblem::FileMissingWithRows(vec!["store-1".to_owned()])
    +    );
    +    assert!(mem.keys().is_empty(), "nothing is created over rows");
    +    // The file put back: healthy.
    +    mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1"));
    +    mem.insert(&manifest_key(&arn(ACCOUNT)), b"{}");
    +    store_identity::establish(
    +        &mem,
    +        StoreKind::Filesystem,
    +        store().store_root.as_str(),
    +        &fake,
    +        "test",
    +    )
    +    .await
    +    .unwrap();
    +
    +    // Rows carrying two ids.
    +    let fake = fake_with_row(BackupStatus::Available);
    +    let mut other = row(&newer_arn(ACCOUNT), BackupStatus::Available);
    +    other.store.store_id = "store-9".to_owned();
    +    fake.rows.lock().unwrap().insert(newer_arn(ACCOUNT), other);
    +    let err = store_identity::establish(
    +        &mem,
    +        StoreKind::Filesystem,
    +        store().store_root.as_str(),
    +        &fake,
    +        "test",
    +    )
    +    .await
    +    .unwrap_err();
    +    assert_eq!(
    +        err,
    +        StoreProblem::MultipleRowIds(vec!["store-1".to_owned(), "store-9".to_owned()])
    +    );
    +}
    +
    +/// The manifest probe: the newest AVAILABLE row's manifest must be under the
    +/// root. A second instance with a host-local root at the same path and the
    +/// first's file copied to it has the file and none of the objects, which is
    +/// what the probe catches; a root with no AVAILABLE row is not probed.
    +#[tokio::test]
    +async fn the_manifest_probe_reads_the_newest_available_rows_manifest() {
    +    let mem = MemStore::default();
    +    mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1"));
    +    let fake = fake_with_rows(&[
    +        (arn(ACCOUNT), BackupStatus::Available),
    +        (newer_arn(ACCOUNT), BackupStatus::Available),
    +        (
    +            music_arn(ACCOUNT, "01791976999999-0bad0bad"),
    +            BackupStatus::Creating,
    +        ),
    +    ]);
    +    // Only the older manifest is there: the newest is probed and missing.
    +    mem.insert(&manifest_key(&arn(ACCOUNT)), b"{}");
    +    let err = store_identity::establish(
    +        &mem,
    +        StoreKind::Filesystem,
    +        &store().store_root,
    +        &fake,
    +        "test",
    +    )
    +    .await
    +    .unwrap_err();
    +    assert_eq!(
    +        err,
    +        StoreProblem::ManifestMissing {
    +            backup_arn: newer_arn(ACCOUNT),
    +        }
    +    );
    +    assert_eq!(err.category(), "manifest probe failed");
    +    mem.insert(&manifest_key(&newer_arn(ACCOUNT)), b"{}");
    +    store_identity::establish(
    +        &mem,
    +        StoreKind::Filesystem,
    +        &store().store_root,
    +        &fake,
    +        "test",
    +    )
    +    .await
    +    .unwrap();
    +    // The CREATING row's manifest is not probed: it has none yet.
    +    // A root with rows but none AVAILABLE is not probed either.
    +    let fake = fake_with_rows(&[(arn(ACCOUNT), BackupStatus::Creating)]);
    +    let empty = MemStore::default();
    +    empty.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1"));
    +    store_identity::establish(
    +        &empty,
    +        StoreKind::Filesystem,
    +        &store().store_root,
    +        &fake,
    +        "test",
    +    )
    +    .await
    +    .unwrap();
    +}
    +
    +/// A store that ignores the conditional put makes the create a plain put: a
    +/// race lost before the re-read leaves another id in the file, which the
    +/// re-read catches, naming both ids.
    +#[tokio::test]
    +async fn a_create_race_lost_on_a_store_ignoring_the_condition_is_caught_by_the_re_read() {
    +    let mem = MemStore::default();
    +    *mem.overwrite_after_conditional_put.lock().unwrap() =
    +        Some(identity_file("the-other-instance", "catalog-1"));
    +    let fake = Fake::default();
    +    let err = store_identity::establish(&mem, StoreKind::S3, "s3://b/", &fake, "test")
    +        .await
    +        .unwrap_err();
    +    match err {
    +        StoreProblem::CreateRaceLost { ours, file } => {
    +            assert_eq!(file, "the-other-instance");
    +            assert_ne!(ours, file);
    +            let reason = StoreProblem::CreateRaceLost {
    +                ours: ours.clone(),
    +                file: file.clone(),
    +            }
    +            .public_reason();
    +            assert!(reason.contains(&ours) && reason.contains(&file), "{reason}");
    +        }
    +        other => panic!("expected CreateRaceLost, got {other:?}"),
    +    }
    +}
    +
    +/// An unreachable store is unhealthy with the store's error in the log line
    +/// and not in the `/health` text; the check that runs again (the
    +/// reconciler's) recovers it once the store answers.
    +#[tokio::test]
    +async fn an_unreachable_store_is_unhealthy_and_recovers_on_recheck() {
    +    let mem = Arc::new(MemStore::default());
    +    mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1"));
    +    mem.set_down(true);
    +    let fake = Fake::default();
    +    let err = store_identity::establish(
    +        &*mem,
    +        StoreKind::Filesystem,
    +        &store().store_root,
    +        &fake,
    +        "test",
    +    )
    +    .await
    +    .unwrap_err();
    +    // The operator's line carries the whole cause chain: the top link names
    +    // the class ("dispatch failure"), the cause is the link below it.
    +    assert!(
    +        matches!(&err, StoreProblem::Transport(m)
    +            if m.contains("dispatch failure") && m.contains("connection refused")),
    +        "{err:?}"
    +    );
    +    assert_eq!(err.public_reason(), "the store could not be reached");
    +    assert!(err.to_string().contains("connection refused"));
    +
    +    let runtime = runtime_unhealthy(&mem, err);
    +    assert_eq!(
    +        runtime.configured_store().unwrap().identity(),
    +        None,
    +        "the id is unknown until a check passes"
    +    );
    +    let report = runtime.store_health().unwrap();
    +    assert_eq!(report.kind, "filesystem");
    +    assert_eq!(report.status, "unhealthy");
    +    assert_eq!(report.reason, Some("unreachable"));
    +
    +    // Still down: the cycle leaves it unhealthy.
    +    let report = reconciler::cycle(
    +        &fake,
    +        &runtime,
    +        Duration::from_secs(300),
    +        "test",
    +        &mut reconciler::ReconcilerState::default(),
    +    )
    +    .await
    +    .unwrap();
    +    assert_eq!(report.store_healthy, Some(false));
    +    // Up again: the cycle's recheck passes and the component recovers.
    +    mem.set_down(false);
    +    let report = reconciler::cycle(
    +        &fake,
    +        &runtime,
    +        Duration::from_secs(300),
    +        "test",
    +        &mut reconciler::ReconcilerState::default(),
    +    )
    +    .await
    +    .unwrap();
    +    assert_eq!(report.store_healthy, Some(true));
    +    assert_eq!(runtime.store_health().unwrap().status, "healthy");
    +    assert_eq!(runtime.store_health().unwrap().reason, None);
    +    assert_eq!(
    +        runtime.configured_store().unwrap().identity(),
    +        Some(store()),
    +        "the check that passed settled the id from the file"
    +    );
    +    // And back to unhealthy when it goes away again.
    +    mem.set_down(true);
    +    let report = reconciler::cycle(
    +        &fake,
    +        &runtime,
    +        Duration::from_secs(300),
    +        "test",
    +        &mut reconciler::ReconcilerState::default(),
    +    )
    +    .await
    +    .unwrap();
    +    assert_eq!(report.store_healthy, Some(false));
    +}
    +
    +/// A store that has stopped answering costs a cycle its check bound and
    +/// nothing more: the dead-owner duties ran before the check, the check is
    +/// abandoned at `check_timeout` (5 s for a filesystem root) and recorded as
    +/// a transport failure naming the bound, the cycle returns, and the next
    +/// cycle on a store that answers again recovers it.
    +#[tokio::test(start_paused = true)]
    +async fn a_stalled_store_check_is_abandoned_at_its_bound() {
    +    use extenddb_storage::backup_catalog::RestoreKind;
    +    let mem = Arc::new(MemStore::default());
    +    mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1"));
    +    mem.set_stalled(true);
    +    let runtime = runtime_on(&mem, store());
    +    let configured = runtime.configured_store().unwrap();
    +    assert_eq!(configured.check_timeout(), LOCAL_CHECK_TIMEOUT);
    +    let fake = Fake::default();
    +    *fake.record_still_running.lock().unwrap() = true;
    +    *fake.target_still_creating.lock().unwrap() = true;
    +    fake.dead_jobs
    +        .lock()
    +        .unwrap()
    +        .push(dead_job(RestoreKind::Legacy, "LegacyTarget"));
    +
    +    let started = tokio::time::Instant::now();
    +    // The test's own bound, far past the check's: a cycle that waits on the
    +    // store for longer than it fails here instead of hanging.
    +    let report = tokio::time::timeout(
    +        Duration::from_secs(120),
    +        reconciler::cycle(
    +            &fake,
    +            &runtime,
    +            Duration::from_secs(300),
    +            "test",
    +            &mut reconciler::ReconcilerState::default(),
    +        ),
    +    )
    +    .await
    +    .expect("the cycle returns within the check's bound")
    +    .unwrap();
    +    assert_eq!(
    +        started.elapsed(),
    +        LOCAL_CHECK_TIMEOUT,
    +        "the check's bound is the whole cost"
    +    );
    +    assert_eq!(
    +        report.legacy_restores_removed,
    +        ["LegacyTarget"],
    +        "the duties ran"
    +    );
    +    assert_eq!(report.store_healthy, Some(false));
    +    assert_eq!(
    +        configured.health(),
    +        StoreHealth::Unhealthy(StoreProblem::Transport(
    +            "the check did not finish within 5 s".to_owned()
    +        ))
    +    );
    +    assert_eq!(runtime.store_health().unwrap().reason, Some("unreachable"));
    +    assert_eq!(mem.stalls.load(std::sync::atomic::Ordering::Acquire), 1);
    +    // The abandoned check's call is still in the store (on a filesystem
    +    // root, a thread parked on the mount). A cycle during the stall starts
    +    // no second check: it returns at once, the store stays `unreachable`
    +    // with the reason, and the store sees no second call.
    +    assert!(configured.check_call_in_flight());
    +    let started = tokio::time::Instant::now();
    +    let report = reconciler::cycle(
    +        &fake,
    +        &runtime,
    +        Duration::from_secs(300),
    +        "test",
    +        &mut reconciler::ReconcilerState::default(),
    +    )
    +    .await
    +    .unwrap();
    +    assert_eq!(
    +        started.elapsed(),
    +        Duration::ZERO,
    +        "a cycle that starts no check waits no bound"
    +    );
    +    assert_eq!(report.store_healthy, Some(false));
    +    assert_eq!(
    +        configured.health(),
    +        StoreHealth::Unhealthy(StoreProblem::Transport(
    +            "a call of an earlier check has not returned".to_owned()
    +        ))
    +    );
    +    assert_eq!(
    +        mem.stalls.load(std::sync::atomic::Ordering::Acquire),
    +        1,
    +        "no second call reached the store while the first was in flight"
    +    );
    +    // Checking stores wait the same bound: an S3 store's is 30 s, under the
    +    // SDK's attempt timeout so no retry outlives the check.
    +    let s3 = ConfiguredStore::checking(
    +        StoreKind::S3,
    +        "s3://b/".to_owned(),
    +        Arc::clone(&mem) as Arc,
    +        None,
    +    );
    +    assert_eq!(s3.check_timeout(), S3_CHECK_TIMEOUT);
    +    assert_eq!(S3_CHECK_TIMEOUT, Duration::from_secs(30));
    +    assert!(S3_CHECK_TIMEOUT < extenddb_backup_store::S3_ATTEMPT_TIMEOUT);
    +    // An S3 store's abandoned request is dropped with the check (the SDK
    +    // would otherwise retry it past the bound), so nothing is in flight
    +    // afterwards and the next cycle sends a fresh request.
    +    let s3_runtime = BackupRuntime::new(4).with_store(s3);
    +    let s3 = s3_runtime.configured_store().unwrap();
    +    let calls_before = mem.stalls.load(std::sync::atomic::Ordering::Acquire);
    +    for _ in 0..2 {
    +        let started = tokio::time::Instant::now();
    +        reconciler::cycle(
    +            &fake,
    +            &s3_runtime,
    +            Duration::from_secs(300),
    +            "test",
    +            &mut reconciler::ReconcilerState::default(),
    +        )
    +        .await
    +        .unwrap();
    +        assert_eq!(started.elapsed(), S3_CHECK_TIMEOUT);
    +        assert!(
    +            !s3.check_call_in_flight(),
    +            "an S3 request ends with the check"
    +        );
    +    }
    +    assert_eq!(
    +        mem.stalls.load(std::sync::atomic::Ordering::Acquire),
    +        calls_before + 2,
    +        "each cycle's check reached the endpoint"
    +    );
    +
    +    // The store answers again: the parked call returns (its task clears the
    +    // in-flight mark), and the next cycle's check passes.
    +    mem.set_stalled(false);
    +    tokio::time::sleep(Duration::from_millis(20)).await;
    +    assert!(
    +        !configured.check_call_in_flight(),
    +        "the parked call returned once the store answered"
    +    );
    +    let report = reconciler::cycle(
    +        &fake,
    +        &runtime,
    +        Duration::from_secs(300),
    +        "test",
    +        &mut reconciler::ReconcilerState::default(),
    +    )
    +    .await
    +    .unwrap();
    +    assert_eq!(report.store_healthy, Some(true));
    +    assert!(configured.is_healthy());
    +}
    +
    +/// A shutdown that fires while a cycle waits on the store's check ends the
    +/// cycle at once (the health is left as it was), so the worker drains
    +/// without waiting the check's bound.
    +#[tokio::test(start_paused = true)]
    +async fn a_shutdown_during_the_store_check_ends_the_cycle() {
    +    use extenddb_core::metrics::MetricsCollector;
    +    let mem = Arc::new(MemStore::default());
    +    mem.set_stalled(true);
    +    // An S3 store, whose bound (30 s) is the longer one.
    +    let runtime = Arc::new(runtime().with_store(ConfiguredStore::checking(
    +        StoreKind::S3,
    +        "s3://b/".to_owned(),
    +        Arc::clone(&mem) as Arc,
    +        None,
    +    )));
    +    let fake = Arc::new(Fake::default());
    +    let shutdown = extenddb_storage::CancellationToken::new();
    +    let worker = tokio::spawn(reconciler::run(
    +        Arc::clone(&fake),
    +        Arc::clone(&runtime),
    +        reconciler::ReconcilerSettings {
    +            interval: Duration::from_secs(3600),
    +            stale_job_after: Duration::from_secs(300),
    +        },
    +        "test".to_owned(),
    +        Arc::new(MetricsCollector::new()),
    +        shutdown.clone(),
    +    ));
    +    let deadline = tokio::time::Instant::now() + Duration::from_secs(5);
    +    while mem.stalls.load(std::sync::atomic::Ordering::Acquire) == 0 {
    +        assert!(
    +            tokio::time::Instant::now() < deadline,
    +            "the first cycle's check reached the store"
    +        );
    +        tokio::time::sleep(Duration::from_millis(5)).await;
    +    }
    +    let cancelled = tokio::time::Instant::now();
    +    shutdown.cancel();
    +    tokio::time::timeout(Duration::from_secs(5), worker)
    +        .await
    +        .expect("the worker stops at shutdown without waiting the check's bound")
    +        .unwrap();
    +    assert!(
    +        cancelled.elapsed() < Duration::from_secs(1),
    +        "the drain took {:?}",
    +        cancelled.elapsed()
    +    );
    +    assert_eq!(
    +        runtime.configured_store().unwrap().health(),
    +        StoreHealth::Checking,
    +        "a check shutdown ended records nothing"
    +    );
    +}
    +
    +/// A command's identity read (`backup list`, the purge's dry run) runs under
    +/// the store's check bound as the server's check does: a store that does not
    +/// answer costs the command the bound (5 s for a filesystem root) and
    +/// reports it, where the unbounded read would wait out the store's retries
    +/// or, on a stalled mount, never return; a store that answers is read.
    +#[tokio::test(start_paused = true)]
    +async fn a_commands_identity_read_is_bounded_as_the_check_is() {
    +    let mem = Arc::new(MemStore::default());
    +    mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1"));
    +    let fake = Fake::default();
    +    let configured = ConfiguredStore::checking(
    +        StoreKind::Filesystem,
    +        store().store_root,
    +        Arc::clone(&mem) as Arc,
    +        None,
    +    );
    +    let read = runtime::read_identity_bounded(&configured, &fake)
    +        .await
    +        .unwrap();
    +    assert_eq!(read.map(|i| i.store_id), Some("store-1".to_owned()));
    +
    +    mem.set_stalled(true);
    +    let started = tokio::time::Instant::now();
    +    // The test's own bound, far past the read's: an unbounded read fails
    +    // here instead of hanging.
    +    let problem = tokio::time::timeout(
    +        Duration::from_secs(120),
    +        runtime::read_identity_bounded(&configured, &fake),
    +    )
    +    .await
    +    .expect("the read returns at its bound")
    +    .unwrap_err();
    +    assert_eq!(started.elapsed(), LOCAL_CHECK_TIMEOUT);
    +    assert_eq!(
    +        problem,
    +        StoreProblem::Transport("the identity read did not finish within 5 s".to_owned())
    +    );
    +    assert_eq!(problem.category(), "unreachable");
    +    assert_eq!(mem.stalls.load(std::sync::atomic::Ordering::Acquire), 1);
    +    mem.set_stalled(false);
    +}
    +
    +/// `record_check` logs a transition once per category: it reports a change
    +/// of category, and not a new text within the same one, so a store that
    +/// stays down with a varying error costs one line.
    +#[test]
    +fn record_check_reports_a_transition_only_across_categories() {
    +    let mem = Arc::new(MemStore::default());
    +    let runtime = runtime_on(&mem, store());
    +    let store = runtime.configured_store().unwrap();
    +    assert!(store.record_check(Err(StoreProblem::Transport("refused".to_owned()))));
    +    assert!(
    +        !store.record_check(Err(StoreProblem::Transport("timed out".to_owned()))),
    +        "a new text in the same category is not a transition"
    +    );
    +    assert_eq!(
    +        store.health(),
    +        StoreHealth::Unhealthy(StoreProblem::Transport("timed out".to_owned())),
    +        "the latest text is kept"
    +    );
    +    assert!(store.record_check(Err(StoreProblem::Validate("no bucket".to_owned()))));
    +    assert!(store.record_check(Ok(self::store())));
    +    assert!(!store.record_check(Ok(self::store())));
    +    assert!(
    +        !store.record_check(Err(StoreProblem::Catalog("pool exhausted".to_owned()))),
    +        "a catalog read failure changes nothing"
    +    );
    +}
    +
    +/// A recheck of a root that lost its file while the catalog has no rows for
    +/// it writes the file back with the same id.
    +#[tokio::test]
    +async fn a_recheck_rewrites_a_lost_file_when_no_rows_exist() {
    +    let mem = MemStore::default();
    +    let fake = Fake::default();
    +    store_identity::recheck(&mem, &store(), &fake, "test")
    +        .await
    +        .unwrap();
    +    let file = StoreIdentityFile::parse(&mem.objects.lock().unwrap()[STORE_IDENTITY_KEY]).unwrap();
    +    assert_eq!(file.store_id, "store-1");
    +    assert_eq!(file.catalog_id, "catalog-1");
    +    // A file that changed under the running server (the root replaced) is
    +    // an id mismatch.
    +    mem.insert(STORE_IDENTITY_KEY, &identity_file("store-2", "catalog-1"));
    +    let err = store_identity::recheck(&mem, &store(), &fake, "test")
    +        .await
    +        .unwrap_err();
    +    assert_eq!(
    +        err,
    +        StoreProblem::IdMismatch {
    +            file_id: "store-2".to_owned(),
    +            row_ids: vec!["store-1".to_owned()],
    +        }
    +    );
    +}
    +
    +/// No store configured: `/health` has no component and no operation is
    +/// refused for the store.
    +#[test]
    +fn no_configured_store_reports_no_component() {
    +    assert_eq!(runtime().store_health(), None);
    +    assert_eq!(
    +        runtime_with_store().store_health().unwrap().status,
    +        "healthy"
    +    );
    +}
    +
    +/// The `/health` reason is the problem's category and nothing else: the
    +/// endpoint is unauthenticated, and the ARN (account, table and backup id)
    +/// a manifest probe names, and the catalog and store ids the identity
    +/// problems name, go to the operator's log line only.
    +#[test]
    +fn the_health_reason_is_the_category_only() {
    +    let mem = Arc::new(MemStore::default());
    +    for (problem, category) in [
    +        (
    +            StoreProblem::ManifestMissing {
    +                backup_arn: arn("999988887777"),
    +            },
    +            "manifest probe failed",
    +        ),
    +        (
    +            StoreProblem::ForeignCatalog {
    +                file_catalog_id: "cat-other".to_owned(),
    +                catalog_id: "cat-mine".to_owned(),
    +            },
    +            "owned by another catalog",
    +        ),
    +        (
    +            StoreProblem::IdMismatch {
    +                file_id: "store-9".to_owned(),
    +                row_ids: vec!["store-1".to_owned()],
    +            },
    +            "identity mismatch",
    +        ),
    +        (
    +            StoreProblem::FileMissingWithRows(vec!["store-1".to_owned()]),
    +            "identity file missing",
    +        ),
    +        (
    +            StoreProblem::Transport("connect /var/lib/extenddb: refused".to_owned()),
    +            "unreachable",
    +        ),
    +    ] {
    +        let report = runtime_unhealthy(&mem, problem.clone())
    +            .store_health()
    +            .unwrap();
    +        assert_eq!(report.status, "unhealthy");
    +        assert_eq!(report.reason, Some(category), "{problem:?}");
    +        for secret in ["999988887777", "cat-", "store-", "/var/lib", "arn:"] {
    +            assert!(
    +                !report.reason.unwrap().contains(secret),
    +                "{problem:?}: the reason carries {secret:?}"
    +            );
    +        }
    +    }
    +}
    +
    +/// An identity file over the read bound is reported unreadable without
    +/// being read in full; one under it parses.
    +#[tokio::test]
    +async fn an_oversized_identity_file_is_unreadable() {
    +    let mem = MemStore::default();
    +    let fake = Fake::default();
    +    let mut big = identity_file("store-1", "catalog-1");
    +    big.resize(store_identity::STORE_IDENTITY_MAX_BYTES + 1, b' ');
    +    mem.insert(STORE_IDENTITY_KEY, &big);
    +    let err = store_identity::establish(&mem, StoreKind::Filesystem, "/r", &fake, "test")
    +        .await
    +        .unwrap_err();
    +    assert!(
    +        matches!(&err, StoreProblem::FileUnreadable(m) if m.contains("larger than 65536 bytes")),
    +        "{err:?}"
    +    );
    +    assert_eq!(err.category(), "identity file unreadable");
    +    // Trailing blanks up to the bound are JSON whitespace: the file parses.
    +    let mut padded = identity_file("store-1", "catalog-1");
    +    padded.resize(store_identity::STORE_IDENTITY_MAX_BYTES, b' ');
    +    mem.insert(STORE_IDENTITY_KEY, &padded);
    +    let identity = store_identity::establish(&mem, StoreKind::Filesystem, "/r", &fake, "test")
    +        .await
    +        .unwrap();
    +    assert_eq!(identity.store_id, "store-1");
    +}
    +
    +/// A check that could not read the catalog leaves the store's health as the
    +/// last check left it, healthy or unhealthy, and keeps its id.
    +#[tokio::test]
    +async fn a_catalog_read_failure_leaves_the_health_unchanged() {
    +    let mem = Arc::new(MemStore::default());
    +    let healthy = runtime_on(&mem, store());
    +    let store = healthy.configured_store().unwrap();
    +    store.record_check(Err(StoreProblem::Catalog("pool exhausted".to_owned())));
    +    assert_eq!(store.health(), StoreHealth::Healthy);
    +    assert_eq!(store.identity(), Some(self::store()));
    +    let unhealthy = runtime_unhealthy(&mem, StoreProblem::Transport("refused".to_owned()));
    +    let store = unhealthy.configured_store().unwrap();
    +    store.record_check(Err(StoreProblem::Catalog("pool exhausted".to_owned())));
    +    assert_eq!(
    +        store.health(),
    +        StoreHealth::Unhealthy(StoreProblem::Transport("refused".to_owned()))
    +    );
    +    // And through a cycle: a catalog that fails `catalog_id` leaves a
    +    // healthy store healthy.
    +    let fake = Fake::default();
    +    *fake.catalog_id_error.lock().unwrap() = true;
    +    let report = reconciler::cycle(
    +        &fake,
    +        &healthy,
    +        Duration::from_secs(300),
    +        "test",
    +        &mut reconciler::ReconcilerState::default(),
    +    )
    +    .await
    +    .unwrap();
    +    assert_eq!(report.store_healthy, Some(true));
    +    // A problem of another category replaces the last one; one of the same
    +    // category is kept as the latest text.
    +    let store = healthy.configured_store().unwrap();
    +    store.record_check(Err(StoreProblem::Transport("refused".to_owned())));
    +    store.record_check(Err(StoreProblem::Transport("timed out".to_owned())));
    +    assert_eq!(
    +        store.health(),
    +        StoreHealth::Unhealthy(StoreProblem::Transport("timed out".to_owned()))
    +    );
    +}
    +
    +/// A store whose checks run after the listener is up is `checking` until
    +/// the first check records: `/health` says so with no reason, the
    +/// operations that touch the store refuse naming the pending check, and
    +/// the check runs `validate` first (a store that cannot be reached fails
    +/// there, with its id still unknown), then the identity check, which
    +/// settles the id.
    +#[tokio::test]
    +async fn a_store_checked_after_startup_is_checking_until_the_first_check() {
    +    let mem = Arc::new(MemStore::default());
    +    mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1"));
    +    mem.set_down(true);
    +    let runtime = runtime().with_store(ConfiguredStore::checking(
    +        store().store_kind,
    +        store().store_root,
    +        Arc::clone(&mem) as Arc,
    +        None,
    +    ));
    +    let configured = runtime.configured_store().unwrap();
    +    assert_eq!(configured.health(), StoreHealth::Checking);
    +    assert!(!configured.is_validated());
    +    let report = runtime.store_health().unwrap();
    +    assert_eq!((report.status, report.reason), ("checking", None));
    +    let err = runtime.require_healthy_store().err().unwrap();
    +    assert!(
    +        matches!(&err, DynamoDbError::ValidationException(m) if m.contains("first check is pending")),
    +        "{err:?}"
    +    );
    +    // The restore check of a row of the store refuses the same way, before
    +    // the permit: with no slot free, the refusal is still the store's.
    +    let fake = fake_with_row(BackupStatus::Available);
    +    let full = BackupRuntime::new(0).with_store(ConfiguredStore::checking(
    +        store().store_kind,
    +        store().store_root,
    +        Arc::clone(&mem) as Arc,
    +        None,
    +    ));
    +    let err = restore_on(&fake, &full, &restore_body(json!({})))
    +        .await
    +        .unwrap_err();
    +    assert!(
    +        matches!(&err, DynamoDbError::ValidationException(m) if m.contains("first check is pending")),
    +        "{err:?}"
    +    );
    +
    +    // The first check: `validate` fails on the unreachable store.
    +    let fake = Fake::default();
    +    let report = reconciler::cycle(
    +        &fake,
    +        &runtime,
    +        Duration::from_secs(300),
    +        "test",
    +        &mut reconciler::ReconcilerState::default(),
    +    )
    +    .await
    +    .unwrap();
    +    assert_eq!(report.store_healthy, Some(false));
    +    assert!(!configured.is_validated());
    +    assert_eq!(runtime.store_health().unwrap().reason, Some("unreachable"));
    +    assert_eq!(configured.identity(), None);
    +    // Reachable: `validate` passes once and is not run again, the identity
    +    // check settles the id, and the store is healthy.
    +    mem.set_down(false);
    +    let report = reconciler::cycle(
    +        &fake,
    +        &runtime,
    +        Duration::from_secs(300),
    +        "test",
    +        &mut reconciler::ReconcilerState::default(),
    +    )
    +    .await
    +    .unwrap();
    +    assert_eq!(report.store_healthy, Some(true));
    +    assert!(configured.is_validated());
    +    assert_eq!(configured.identity(), Some(store()));
    +    assert_eq!(runtime.store_health().unwrap().status, "healthy");
    +}
    +
    +/// The refusals while the configured store is unhealthy: DeleteBackup of a
    +/// row of the store, and the restore of such a row, each a
    +/// ValidationException naming the store's kind and the problem's category
    +/// and never the root; the legacy reader is not reached, and the restore's
    +/// refusal comes before the permit (with no slot free it is still the
    +/// store's, not LimitExceeded). CreateBackup is not refused: the in-catalog
    +/// create, the one this release runs, touches no store, so it runs with an
    +/// unhealthy store as it does with none (a server that never configured
    +/// backups keeps CreateBackup through a failed check of the default root),
    +/// and on a backend bound to the in-catalog create.
    +#[tokio::test]
    +async fn an_unhealthy_store_refuses_delete_and_restore_but_not_the_in_catalog_create() {
    +    let mem = Arc::new(MemStore::default());
    +    let unhealthy = || {
    +        runtime_unhealthy(
    +            &mem,
    +            StoreProblem::Transport("connect /var/lib/extenddb/backups: refused".to_owned()),
    +        )
    +    };
    +    let check = |e: DynamoDbError| {
    +        let DynamoDbError::ValidationException(m) = e else {
    +            panic!("expected ValidationException, got {e:?}");
    +        };
    +        assert!(m.contains("filesystem backup store"), "{m}");
    +        assert!(m.contains("unreachable"), "{m}");
    +        assert!(!m.contains("/var/lib"), "{m}");
    +        assert!(!m.contains("connect"), "{m}");
    +    };
    +
    +    // CreateBackup runs the in-catalog create, taking its slot.
    +    let fake = Fake::default();
    +    for runtime in [unhealthy(), unhealthy().with_legacy_create_only(true)] {
    +        let before = fake.creates.lock().unwrap().len();
    +        create_backup(
    +            &fake,
    +            &runtime,
    +            ACCOUNT,
    +            REGION,
    +            "Music",
    +            "nightly",
    +            Ok(active_table("Music")),
    +        )
    +        .await
    +        .unwrap();
    +        assert_eq!(fake.creates.lock().unwrap().len(), before + 1);
    +        assert_eq!(runtime.free_slots(), 3, "the create holds its slot");
    +        fake.permits.lock().unwrap().clear();
    +    }
    +    // With no slot free the create is LimitExceeded, the store's health
    +    // notwithstanding.
    +    let full = BackupRuntime::new(0).with_store(ConfiguredStore::new(
    +        store().store_kind,
    +        store().store_root,
    +        Arc::clone(&mem) as Arc,
    +        None,
    +        Err(StoreProblem::Transport("refused".to_owned())),
    +    ));
    +    let err = create_backup(
    +        &fake,
    +        &full,
    +        ACCOUNT,
    +        REGION,
    +        "Music",
    +        "nightly",
    +        Ok(active_table("Music")),
    +    )
    +    .await
    +    .unwrap_err();
    +    assert!(
    +        matches!(err, DynamoDbError::LimitExceededException(_)),
    +        "{err:?}"
    +    );
    +
    +    // DeleteBackup of a row of the store.
    +    let fake = fake_with_row(BackupStatus::Available);
    +    let err = delete_backup(&fake, &unhealthy(), ACCOUNT, REGION, &own(&arn(ACCOUNT)))
    +        .await
    +        .unwrap_err();
    +    check(err);
    +    assert!(
    +        fake.delete_starts_seen.lock().unwrap().is_empty(),
    +        "no DELETING write"
    +    );
    +
    +    // The restore of a row of the store, with no slot free: the refusal is
    +    // the store's, so it came before the permit.
    +    let full = BackupRuntime::new(0).with_store(ConfiguredStore::new(
    +        store().store_kind,
    +        store().store_root,
    +        Arc::clone(&mem) as Arc,
    +        None,
    +        Err(StoreProblem::Transport(
    +            "connect /var/lib/extenddb/backups: refused".to_owned(),
    +        )),
    +    ));
    +    let err = restore_on(&fake, &full, &restore_body(json!({})))
    +        .await
    +        .unwrap_err();
    +    check(err);
    +    assert!(fake.restores().is_empty(), "the backend was not asked");
    +
    +    // A row of another store is the other-store refusal, whatever the health
    +    // of the configured one.
    +    let mut other = row(&arn(ACCOUNT), BackupStatus::Available);
    +    other.store.store_root = "/elsewhere".to_owned();
    +    let fake = Fake::default();
    +    fake.rows.lock().unwrap().insert(arn(ACCOUNT), other);
    +    let err = delete_backup(&fake, &unhealthy(), ACCOUNT, REGION, &own(&arn(ACCOUNT)))
    +        .await
    +        .unwrap_err();
    +    assert!(
    +        matches!(&err, DynamoDbError::ValidationException(m) if m.contains("other than")),
    +        "{err:?}"
    +    );
    +}
    +
    +/// With a healthy store configured, CreateBackup takes the in-catalog path
    +/// (this release has no store-backed create) and logs that the store is
    +/// configured; the slot is taken and the reader's create runs.
    +#[tokio::test]
    +async fn a_healthy_store_leaves_create_backup_on_the_in_catalog_path() {
    +    let fake = Fake::default();
    +    let runtime = runtime_with_store();
    +    create_backup(
    +        &fake,
    +        &runtime,
    +        ACCOUNT,
    +        REGION,
    +        "Music",
    +        "nightly",
    +        Ok(active_table("Music")),
    +    )
    +    .await
    +    .unwrap();
    +    assert_eq!(fake.creates.lock().unwrap().len(), 1);
    +}
    +
    +/// DeleteBackup of a row of the configured store deletes the manifest first
    +/// (a `delete_prefix` on its exact key), then the prefix, and moves the row
    +/// to DELETED once the prefix lists nothing; a retry on a DELETING row runs
    +/// the same deletion. The identity file and other backups' prefixes stay.
    +#[tokio::test]
    +async fn delete_backup_of_a_configured_store_row_deletes_manifest_prefix_then_marks_deleted() {
    +    let mem = Arc::new(MemStore::default());
    +    let prefix = BackupArn::parse(&arn(ACCOUNT)).unwrap().store_prefix();
    +    mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1"));
    +    mem.insert(&manifest_key(&arn(ACCOUNT)), b"{}");
    +    mem.insert(&format!("{prefix}data/00001.json.gz"), b"gz");
    +    mem.insert(&format!("{prefix}manifest-files.json"), b"{}");
    +    mem.insert(&manifest_key(&newer_arn(ACCOUNT)), b"{}");
    +    let fake = fake_with_rows(&[
    +        (arn(ACCOUNT), BackupStatus::Deleting),
    +        (newer_arn(ACCOUNT), BackupStatus::Available),
    +    ]);
    +    let runtime = runtime_on(&mem, store());
    +    let desc = delete_backup(&fake, &runtime, ACCOUNT, REGION, &own(&arn(ACCOUNT)))
    +        .await
    +        .unwrap();
    +    assert_eq!(desc.backup_details.backup_status, "DELETED");
    +    assert_eq!(
    +        mem.deletes.lock().unwrap().as_slice(),
    +        [manifest_key(&arn(ACCOUNT)), prefix.clone()],
    +        "the manifest first, then the prefix"
    +    );
    +    assert_eq!(
    +        mem.keys(),
    +        [
    +            manifest_key(&newer_arn(ACCOUNT)),
    +            STORE_IDENTITY_KEY.to_owned()
    +        ],
    +        "the other backup and the identity file stay"
    +    );
    +    assert_eq!(fake.set_deleted.lock().unwrap().as_slice(), [arn(ACCOUNT)]);
    +    assert_eq!(
    +        fake.rows.lock().unwrap()[&arn(ACCOUNT)].status,
    +        BackupStatus::Deleted
    +    );
    +}
    +
    +/// A store failure during the deletion leaves the row DELETING and answers
    +/// InternalServerError; the reconciler's next cycle finishes it once the
    +/// store answers, and leaves it alone while the store is unhealthy.
    +#[tokio::test]
    +async fn a_failed_deletion_stays_deleting_and_the_reconciler_finishes_it() {
    +    let mem = Arc::new(MemStore::default());
    +    let prefix = BackupArn::parse(&arn(ACCOUNT)).unwrap().store_prefix();
    +    mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1"));
    +    mem.insert(&manifest_key(&arn(ACCOUNT)), b"{}");
    +    mem.insert(&format!("{prefix}data/00001.json.gz"), b"gz");
    +    let fake = fake_with_rows(&[(arn(ACCOUNT), BackupStatus::Deleting)]);
    +    let runtime = runtime_on(&mem, store());
    +
    +    mem.set_down(true);
    +    let err = delete_backup(&fake, &runtime, ACCOUNT, REGION, &own(&arn(ACCOUNT)))
    +        .await
    +        .unwrap_err();
    +    assert!(
    +        matches!(err, DynamoDbError::InternalServerError(_)),
    +        "{err:?}"
    +    );
    +    assert_eq!(
    +        fake.rows.lock().unwrap()[&arn(ACCOUNT)].status,
    +        BackupStatus::Deleting
    +    );
    +    assert_eq!(mem.keys().len(), 3, "nothing was deleted");
    +
    +    // The reconciler finds the store down: unhealthy, and no row touched.
    +    let report = reconciler::cycle(
    +        &fake,
    +        &runtime,
    +        Duration::from_secs(300),
    +        "test",
    +        &mut reconciler::ReconcilerState::default(),
    +    )
    +    .await
    +    .unwrap();
    +    assert_eq!(report.store_healthy, Some(false));
    +    assert!(report.deletions_finished.is_empty());
    +    assert!(runtime.require_healthy_store().is_err());
    +
    +    // The store answers again: the cycle recovers the health and finishes
    +    // the deletion.
    +    mem.set_down(false);
    +    let report = reconciler::cycle(
    +        &fake,
    +        &runtime,
    +        Duration::from_secs(300),
    +        "test",
    +        &mut reconciler::ReconcilerState::default(),
    +    )
    +    .await
    +    .unwrap();
    +    assert_eq!(report.store_healthy, Some(true));
    +    assert_eq!(report.deletions_finished, [arn(ACCOUNT)]);
    +    assert_eq!(mem.keys(), [STORE_IDENTITY_KEY]);
    +    assert_eq!(
    +        fake.rows.lock().unwrap()[&arn(ACCOUNT)].status,
    +        BackupStatus::Deleted
    +    );
    +    // Nothing left to finish.
    +    let report = reconciler::cycle(
    +        &fake,
    +        &runtime,
    +        Duration::from_secs(300),
    +        "test",
    +        &mut reconciler::ReconcilerState::default(),
    +    )
    +    .await
    +    .unwrap();
    +    assert!(report.deletions_finished.is_empty());
    +}
    +
    +/// The reconciler finishes DELETING rows of the configured store only: a
    +/// DELETING row of another store is left alone, and so is one whose prefix
    +/// still lists an object after the deletion (a put that overlapped it).
    +#[tokio::test]
    +async fn the_reconciler_finishes_only_the_configured_stores_deleting_rows() {
    +    let mem = Arc::new(MemStore::default());
    +    mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1"));
    +    let fake = fake_with_rows(&[(arn(ACCOUNT), BackupStatus::Deleting)]);
    +    let mut other = row(&newer_arn(ACCOUNT), BackupStatus::Deleting);
    +    other.store.store_id = "store-9".to_owned();
    +    other.store.store_root = "/elsewhere".to_owned();
    +    fake.rows.lock().unwrap().insert(newer_arn(ACCOUNT), other);
    +    let runtime = runtime_on(&mem, store());
    +    let report = reconciler::cycle(
    +        &fake,
    +        &runtime,
    +        Duration::from_secs(300),
    +        "test",
    +        &mut reconciler::ReconcilerState::default(),
    +    )
    +    .await
    +    .unwrap();
    +    assert_eq!(report.deletions_finished, [arn(ACCOUNT)]);
    +    assert_eq!(
    +        fake.rows.lock().unwrap()[&newer_arn(ACCOUNT)].status,
    +        BackupStatus::Deleting,
    +        "another store's row is not touched"
    +    );
    +    assert!(
    +        mem.deletes
    +            .lock()
    +            .unwrap()
    +            .iter()
    +            .all(|p| !p.contains("0badf00d")),
    +        "nothing of the other store's prefix was deleted"
    +    );
    +}
    +
    +/// `finish_backup_delete` reports a prefix that still lists an object after
    +/// the deletion and leaves the row DELETING for the next run.
    +#[tokio::test]
    +async fn a_prefix_that_still_lists_an_object_is_left_deleting() {
    +    struct Refilling(MemStore);
    +    impl extenddb_backup_store::BackupStore for Refilling {
    +        fn put<'a>(
    +            &'a self,
    +            key: &'a str,
    +            body: extenddb_backup_store::ByteStream,
    +        ) -> BoxedFuture<
    +            'a,
    +            Result,
    +        > {
    +            self.0.put(key, body)
    +        }
    +        fn put_if_absent<'a>(
    +            &'a self,
    +            key: &'a str,
    +            body: extenddb_backup_store::ByteStream,
    +        ) -> BoxedFuture<
    +            'a,
    +            Result, extenddb_backup_store::StoreError>,
    +        > {
    +            self.0.put_if_absent(key, body)
    +        }
    +        fn get<'a>(
    +            &'a self,
    +            key: &'a str,
    +        ) -> BoxedFuture<
    +            'a,
    +            Result,
    +        > {
    +            self.0.get(key)
    +        }
    +        fn head<'a>(
    +            &'a self,
    +            key: &'a str,
    +        ) -> BoxedFuture<
    +            'a,
    +            Result, extenddb_backup_store::StoreError>,
    +        > {
    +            self.0.head(key)
    +        }
    +        fn list<'a>(
    +            &'a self,
    +            prefix: &'a str,
    +        ) -> BoxStream<
    +            'a,
    +            Result,
    +        > {
    +            self.0.list(prefix)
    +        }
    +        /// The deletion runs, and a late put lands under the prefix after
    +        /// it.
    +        fn delete_prefix<'a>(
    +            &'a self,
    +            prefix: &'a str,
    +        ) -> BoxedFuture<'a, Result> {
    +            Box::pin(async move {
    +                let n = self.0.delete_prefix(prefix).await?;
    +                if !prefix.ends_with(MANIFEST_KEY_SUFFIX) {
    +                    self.0
    +                        .insert(&format!("{prefix}data/late.json.gz"), b"late");
    +                }
    +                Ok(n)
    +            })
    +        }
    +        fn validate(&self) -> BoxedFuture<'_, Result<(), extenddb_backup_store::StoreError>> {
    +            self.0.validate()
    +        }
    +    }
    +    let store_handle = Refilling(MemStore::default());
    +    let fake = fake_with_rows(&[(arn(ACCOUNT), BackupStatus::Deleting)]);
    +    let parsed = BackupArn::parse(&arn(ACCOUNT)).unwrap();
    +    let err = finish_backup_delete(&fake, &store_handle, &parsed)
    +        .await
    +        .unwrap_err();
    +    assert!(err.contains("still lists"), "{err}");
    +    assert_eq!(
    +        fake.rows.lock().unwrap()[&arn(ACCOUNT)].status,
    +        BackupStatus::Deleting
    +    );
    +}
    diff --git a/crates/engine/src/lib.rs b/crates/engine/src/lib.rs
    index a5c6ed77a..2fb8b63a5 100755
    --- a/crates/engine/src/lib.rs
    +++ b/crates/engine/src/lib.rs
    @@ -8,7 +8,11 @@
     //! `DynamoDB`-format error responses.
     
     mod backup;
    -pub use backup::{BackupRuntime, reconciler};
    +pub use backup::{
    +    BackupRuntime, ConfiguredStore, HeldClaim, LOCAL_CHECK_TIMEOUT, S3_CHECK_TIMEOUT,
    +    STORE_IDENTITY_KEY, Spool, StoreHealth, StoreHealthReport, StoreIdentityFile, StoreProblem,
    +    check_configured_store, maintenance, read_identity_bounded, reconciler, store_identity,
    +};
     mod batch_get_item;
     mod batch_write_item;
     pub mod capacity_helpers;
    diff --git a/crates/server/Cargo.toml b/crates/server/Cargo.toml
    index da5da4f74..c6e58acca 100755
    --- a/crates/server/Cargo.toml
    +++ b/crates/server/Cargo.toml
    @@ -16,6 +16,7 @@ extenddb-core = { workspace = true }
     extenddb-config = { workspace = true }
     extenddb-engine = { workspace = true }
     extenddb-storage = { workspace = true }
    +extenddb-backup-store = { workspace = true }
     extenddb-auth = { workspace = true }
     futures = { workspace = true }
     axum = { workspace = true }
    @@ -51,3 +52,5 @@ libc = { workspace = true }
     
     [dev-dependencies]
     extenddb-cache = { workspace = true, features = ["test-util"] }
    +tempfile = "3"
    +tokio = { workspace = true, features = ["test-util"] }
    diff --git a/crates/server/src/backup_store.rs b/crates/server/src/backup_store.rs
    new file mode 100644
    index 000000000..b0962e9b4
    --- /dev/null
    +++ b/crates/server/src/backup_store.rs
    @@ -0,0 +1,1343 @@
    +// Copyright 2026 ExtendDB contributors
    +// SPDX-License-Identifier: Apache-2.0
    +
    +//! Startup of the backup store: the `[backup.store]` resolution, `open`,
    +//! the spool and its lock, and, for a filesystem store, `validate` and the
    +//! identity check; the split between a configuration error, which stops
    +//! startup, and a store condition or identity failure, which starts the
    +//! server with the store unhealthy; and the rule that nothing here waits on
    +//! the network, so a store endpoint that stalls never holds the data plane.
    +//!
    +//! Configuration errors, each named with the path or bucket in the store's
    +//! own message, every one decided without a request to the store: `open`
    +//! refusing the configuration (a missing section, an empty or S3 Express
    +//! bucket name, a part size, prefix or key-length problem, an insecure
    +//! endpoint, a filesystem root that does not exist or fails the owner-only
    +//! rule), a filesystem store's `validate` refusing the root for a reason of
    +//! the same kind (a root another user can reach, a root replaced by a
    +//! symlink), a spool that breaks its placement rule, that another live
    +//! instance holds or that exists with a mode other than owner-only, and an
    +//! S3 store without a spool on a backend with no default. A filesystem
    +//! root's `validate` also writes a probe file, which is a request to the
    +//! store: an I/O failure there or in resolving the root (a full or read-only
    +//! disk, a stale network handle, a root that vanished after `open`) is a
    +//! store condition and degrades the store, as the same failure one cycle
    +//! later does, rather than stopping startup.
    +//!
    +//! The one exception to the configuration errors is the defaulted SQLite
    +//! root (`backups/` beside the database file when `[backup.store]` is
    +//! absent) while the catalog has no backup rows for it: a failure of its
    +//! creation, `open`, `validate` or identity check is a logged warning that
    +//! leaves the store unconfigured, so an upgrade, a fresh deployment or a
    +//! catalog re-created by `destroy` and `init` under the old root never stops
    +//! a server that never configured backups. Once the catalog has rows for
    +//! that root ([`extenddb_storage::BackupCatalog::read_store_ids`]), the root
    +//! has been used and its backups must not vanish behind a warning: the
    +//! defaulted root is then handled as an explicitly configured one, a
    +//! refusal of `open` or `validate` a startup error, and an identity failure
    +//! an unhealthy store (reported on `/health`, its rows' restores and deletes
    +//! refused naming the kind and the category, rechecked each cycle). A
    +//! command that found no store learns why from [`NotConfigured`].
    +//!
    +//! A filesystem store's `validate` and identity check run here, before the
    +//! listener, bounded by [`LOCAL_CHECK_TIMEOUT`]: a root on a local disk
    +//! answers at once, and a mount that stops answering after the root is
    +//! opened leaves the server starting with the store `checking`; a root that
    +//! does not answer at startup holds the server before the listener, as the
    +//! database directory would (the banner's resolution of the root, the
    +//! default root's creation and `open`'s own resolution run unbounded, before
    +//! the checks). The socket is bound before the banner, so a startup held
    +//! that way leaves the port accepting connections it never answers until
    +//! the mount answers or the process is killed. A check abandoned at the
    +//! bound leaves its filesystem call parked on a blocking thread until the
    +//! mount answers (the engine's runtime module says how the later checks
    +//! avoid parking another); the thread costs nothing at exit, where the
    +//! binary ends its runtime with a bounded shutdown. An S3 store's
    +//! `validate` and identity check are requests to the endpoint, and run after
    +//! the listener is up, in the reconciler's first cycle
    +//! ([`extenddb_engine::check_configured_store`]); the store is `checking`
    +//! until that cycle records its result, and the operations that touch the
    +//! store refuse meanwhile as they do while it is unhealthy. A `validate`
    +//! refusal that is not a transport failure (a missing or denied bucket) then
    +//! makes the store unhealthy rather than stopping startup.
    +//!
    +//! Transport failures (the store unreachable: no region, no credentials, a
    +//! refused connection) and failures of the identity check or the manifest
    +//! probe (a denial the store reports while a check runs, say a directory
    +//! under the root that another user can reach, included) leave the server
    +//! running with the store unhealthy: logged once, reported on `/health`,
    +//! the operations that touch the store refused, and the check run again
    +//! each reconciler cycle until it passes.
    +
    +use std::path::{Path, PathBuf};
    +use std::sync::Arc;
    +
    +use extenddb_backup_store::{BackupStore, BackupStoreKind, StoreError};
    +use extenddb_config::backup::{BackupConfig, EffectiveStore, Unconfigured};
    +pub use extenddb_engine::LOCAL_CHECK_TIMEOUT;
    +use extenddb_engine::{BackupRuntime, ConfiguredStore, Spool, StoreProblem, store_identity};
    +use extenddb_storage::backup_catalog::{StoreIdentity, StoreKind};
    +use extenddb_storage::{BackupCatalog, StorageEngine};
    +
    +/// The lock on the spool directory, held for the process's life so a second
    +/// instance on this host refuses the same `spool_path`.
    +pub struct SpoolLock {
    +    _file: std::fs::File,
    +    path: PathBuf,
    +}
    +
    +impl std::fmt::Debug for SpoolLock {
    +    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
    +        f.debug_struct("SpoolLock")
    +            .field("path", &self.path)
    +            .finish()
    +    }
    +}
    +
    +/// The name of the lock file inside the spool.
    +pub const SPOOL_LOCK_FILE: &str = ".lock";
    +
    +/// Who is configuring the store, which decides the spool and the checks.
    +#[derive(Debug, Clone, Copy, PartialEq, Eq)]
    +pub enum Caller {
    +    /// A server: create the spool and hold its lock for the process's life;
    +    /// a filesystem store's checks run here (module doc).
    +    Server,
    +    /// A command that writes (`backup purge`): the spool rule is checked,
    +    /// the directory neither created nor locked; the checks run as a
    +    /// server's do.
    +    Command,
    +    /// A command that reads beside a server (`backup list`, the purge's dry
    +    /// run): as `Command`, and no check of the store runs, so a root is left
    +    /// as it is found (neither the default directory nor an identity file is
    +    /// created; a default root that does not exist yet leaves the store
    +    /// unconfigured, logged); the store is returned `checking` and the
    +    /// command reads what it needs itself.
    +    ReadOnlyCommand,
    +}
    +
    +/// Why [`configure_described`] left the store unconfigured; `Display` is the
    +/// sentence a command prints for it.
    +#[derive(Debug, Clone, PartialEq, Eq)]
    +pub enum NotConfigured {
    +    /// `[backup.store]` is absent and this backend has no default root.
    +    Absent,
    +    /// `kind = "disabled"`.
    +    Disabled,
    +    /// A read-only command found that the default root does not exist yet:
    +    /// no server has run with this database file.
    +    DefaultRootMissing(PathBuf),
    +    /// The defaulted root could not be created, opened, validated or checked
    +    /// while the catalog has no rows for it (module doc): the warning path.
    +    DefaultRootUnusable {
    +        /// The root, as the configuration or `open` spelled it.
    +        root: String,
    +        /// The failure, as the warning logged it.
    +        problem: String,
    +    },
    +}
    +
    +impl std::fmt::Display for NotConfigured {
    +    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
    +        match self {
    +            Self::Absent => f.write_str(
    +                "no backup store is configured: [backup.store] is absent and this backend has \
    +                 no default root, so there are no store-backed backups to list or purge",
    +            ),
    +            Self::Disabled => f.write_str(
    +                "no backup store is configured: [backup.store] is disabled, so there are no \
    +                 store-backed backups to list or purge",
    +            ),
    +            Self::DefaultRootMissing(root) => write!(
    +                f,
    +                "the default root {} does not exist yet: no server has run with this database \
    +                 file, so there are no store-backed backups to list or purge",
    +                root.display()
    +            ),
    +            Self::DefaultRootUnusable { root, problem } => write!(
    +                f,
    +                "the default root {root} failed its check ({problem}) and the catalog has no \
    +                 rows for it, so there are no store-backed backups to list or purge; the \
    +                 warning above says how to fix the root"
    +            ),
    +        }
    +    }
    +}
    +
    +/// What [`configure_described`] built.
    +pub struct StoreSetup {
    +    /// The job slots and, when one is configured, the store with its health.
    +    pub runtime: BackupRuntime,
    +    /// The spool's lock, held by a server for the process's life.
    +    pub spool_lock: Option,
    +    /// Why the runtime has no store, when it has none.
    +    pub not_configured: Option,
    +}
    +
    +impl StoreSetup {
    +    fn without_store(runtime: BackupRuntime, why: NotConfigured) -> Self {
    +        Self {
    +            runtime,
    +            spool_lock: None,
    +            not_configured: Some(why),
    +        }
    +    }
    +
    +    fn with_store(
    +        runtime: BackupRuntime,
    +        store: ConfiguredStore,
    +        spool_lock: Option,
    +    ) -> Self {
    +        Self {
    +            runtime: runtime.with_store(store),
    +            spool_lock,
    +            not_configured: None,
    +        }
    +    }
    +}
    +
    +/// Build the backup runtime for this process: the job slots and, when the
    +/// configuration and the backend select one, the store with its health
    +/// ([`configure_described`] without the reason for an absent store).
    +///
    +/// # Errors
    +///
    +/// A configuration error (module doc), with the key, path or bucket named.
    +pub async fn configure(
    +    backup: &BackupConfig,
    +    database_file: Option<&Path>,
    +    backend: &str,
    +    storage: &Arc,
    +    caller: Caller,
    +) -> anyhow::Result<(BackupRuntime, Option)> {
    +    let setup = configure_described(backup, database_file, backend, &**storage, caller).await?;
    +    Ok((setup.runtime, setup.spool_lock))
    +}
    +
    +/// Build the backup runtime for this process: the job slots and, when the
    +/// configuration and the backend select one, the store with its health;
    +/// otherwise why there is none. `catalog` is the catalog the identity check
    +/// reads and the rows rule of the module doc consults.
    +///
    +/// # Errors
    +///
    +/// A configuration error (module doc), with the key, path or bucket named.
    +pub async fn configure_described(
    +    backup: &BackupConfig,
    +    database_file: Option<&Path>,
    +    backend: &str,
    +    catalog: &C,
    +    caller: Caller,
    +) -> anyhow::Result {
    +    let runtime = BackupRuntime::new(backup.max_concurrent_jobs)
    +        .with_legacy_create_only(backend == "cassandra")
    +        .with_stale_job_after(std::time::Duration::from_secs(backup.stale_job_after_secs));
    +    let resolved = backup
    +        .resolve_store(database_file)
    +        .map_err(|e| anyhow::anyhow!("[backup] configuration: {e}"))?;
    +    let (config, defaulted) = match resolved.store {
    +        EffectiveStore::Unconfigured(Unconfigured::Absent) => {
    +            tracing::info!(
    +                "backup store: none configured ([backup.store] is absent and this backend has \
    +                 no default root); CreateBackup writes in-catalog backups, with a deprecation \
    +                 warning on each call"
    +            );
    +            return Ok(StoreSetup::without_store(runtime, NotConfigured::Absent));
    +        }
    +        EffectiveStore::Unconfigured(Unconfigured::Disabled) => {
    +            tracing::info!(
    +                "backup store: disabled ([backup.store] kind = \"disabled\"); CreateBackup \
    +                 writes in-catalog backups, with a deprecation warning on each call"
    +            );
    +            return Ok(StoreSetup::without_store(runtime, NotConfigured::Disabled));
    +        }
    +        EffectiveStore::Configured { config, defaulted } => (config, defaulted),
    +    };
    +
    +    // The defaulted root beside the database file is created if missing,
    +    // private to this user; an explicitly configured root must exist. A
    +    // read-only command creates nothing: a defaulted root no server has
    +    // created yet is reported as such, and the command has nothing to read.
    +    if defaulted
    +        && caller == Caller::ReadOnlyCommand
    +        && let Some(fs) = &config.filesystem
    +        && !fs.path.exists()
    +    {
    +        tracing::info!(
    +            "backup store: the default root {} does not exist yet (no server has run with this \
    +             database file); there are no store-backed backups to list",
    +            fs.path.display()
    +        );
    +        return Ok(StoreSetup::without_store(
    +            runtime,
    +            NotConfigured::DefaultRootMissing(fs.path.clone()),
    +        ));
    +    }
    +    if defaulted
    +        && let Some(fs) = &config.filesystem
    +        && let Err(e) = create_private_dir_if_missing(&fs.path)
    +    {
    +        // A root that could not be created does not exist, so no row can
    +        // name it: the warning path, whatever the catalog holds.
    +        let root = fs.path.display().to_string();
    +        let problem = format!("it could not be created: {e}");
    +        warn_default_root_unusable(&root, &problem);
    +        return Ok(StoreSetup::without_store(
    +            runtime,
    +            NotConfigured::DefaultRootUnusable { root, problem },
    +        ));
    +    }
    +
    +    let opened = match extenddb_backup_store::open_described(&config).await {
    +        Ok(opened) => opened,
    +        Err(e) => {
    +            // The label rows record is the root's canonical path, which the
    +            // refused root may still have (a parent others can write to).
    +            let label = config
    +                .filesystem
    +                .as_ref()
    +                .and_then(|fs| std::fs::canonicalize(&fs.path).ok())
    +                .map(|p| p.display().to_string());
    +            if defaulted
    +                && let Some(root) = label.as_deref()
    +                && !default_root_has_rows(catalog, root).await
    +            {
    +                let problem = format!("it is not usable: {e}");
    +                warn_default_root_unusable(root, &problem);
    +                return Ok(StoreSetup::without_store(
    +                    runtime,
    +                    NotConfigured::DefaultRootUnusable {
    +                        root: root.to_owned(),
    +                        problem,
    +                    },
    +                ));
    +            }
    +            anyhow::bail!(
    +                "[backup.store] ({}) cannot be opened: {e}{}",
    +                describe_target(&config),
    +                rows_note(defaulted)
    +            );
    +        }
    +    };
    +    let kind = match opened.kind {
    +        BackupStoreKind::Filesystem => StoreKind::Filesystem,
    +        BackupStoreKind::S3 => StoreKind::S3,
    +        BackupStoreKind::Disabled => unreachable!("open_described never returns Disabled"),
    +    };
    +
    +    // The spool and its lock: configuration errors on every backend.
    +    let (spool, spool_lock) = match (resolved.spool_path, caller) {
    +        (Some(path), Caller::Server) => {
    +            let (spool, lock) = take_spool(&path)?;
    +            (Some(spool), Some(lock))
    +        }
    +        (Some(_), Caller::Command | Caller::ReadOnlyCommand) | (None, _) => (None, None),
    +    };
    +    let spool_line = spool
    +        .as_ref()
    +        .map(|s| format!(", spool {}", s.path.display()))
    +        .unwrap_or_default();
    +
    +    // An S3 store's checks are requests to the endpoint: they run once the
    +    // listener is up, in the reconciler's first cycle. A read-only command
    +    // runs no check at all.
    +    if caller == Caller::ReadOnlyCommand {
    +        let configured = ConfiguredStore::checking(kind, opened.root, opened.store, spool);
    +        return Ok(StoreSetup::with_store(runtime, configured, spool_lock));
    +    }
    +    if kind == StoreKind::S3 {
    +        if caller == Caller::Server {
    +            tracing::info!(
    +                "backup store: {kind} {}{spool_line}; its validation and identity check run \
    +                 after the listener is up, and CreateBackup writes in-catalog backups in this \
    +                 release",
    +                opened.root
    +            );
    +        }
    +        let configured = ConfiguredStore::checking(kind, opened.root, opened.store, spool);
    +        return Ok(StoreSetup::with_store(runtime, configured, spool_lock));
    +    }
    +
    +    // A filesystem store: `validate` and the identity check read the root
    +    // here (`run_local_checks`). A `validate` refusal is a configuration
    +    // error; a store condition or the identity check's failure degrades the
    +    // store; on the defaulted root without rows either is the warning.
    +    let outcome = match run_local_checks(&*opened.store, kind, &opened.root, catalog, backend).await
    +    {
    +        LocalChecks::Done(outcome) => outcome,
    +        LocalChecks::Refused(e) => {
    +            if defaulted && !default_root_has_rows(catalog, &opened.root).await {
    +                let problem = format!("it failed its validation: {e}");
    +                warn_default_root_unusable(&opened.root, &problem);
    +                return Ok(StoreSetup::without_store(
    +                    runtime,
    +                    NotConfigured::DefaultRootUnusable {
    +                        root: opened.root,
    +                        problem,
    +                    },
    +                ));
    +            }
    +            anyhow::bail!(
    +                "[backup.store] ({}) failed its startup validation: {e}{}",
    +                describe_target(&config),
    +                rows_note(defaulted)
    +            );
    +        }
    +        LocalChecks::TimedOut => {
    +            tracing::warn!(
    +                "backup store: {kind} {} did not answer its startup checks within {} s (a mount \
    +                 that has stopped answering?); the server starts with the store checking, and \
    +                 the reconciler's first cycle runs the checks again",
    +                opened.root,
    +                LOCAL_CHECK_TIMEOUT.as_secs()
    +            );
    +            let configured = ConfiguredStore::checking(kind, opened.root, opened.store, spool);
    +            return Ok(StoreSetup::with_store(runtime, configured, spool_lock));
    +        }
    +    };
    +    match &outcome {
    +        Ok(identity) => tracing::info!(
    +            "backup store: {kind} {} (store id {}){spool_line}; CreateBackup writes in-catalog \
    +             backups in this release, and the store serves restores and deletes of the backups \
    +             it holds",
    +            opened.root,
    +            identity.store_id
    +        ),
    +        Err(problem) if defaulted && !default_root_has_rows(catalog, &opened.root).await => {
    +            let problem = problem.to_string();
    +            warn_default_root_unusable(&opened.root, &problem);
    +            return Ok(StoreSetup::without_store(
    +                runtime,
    +                NotConfigured::DefaultRootUnusable {
    +                    root: opened.root,
    +                    problem,
    +                },
    +            ));
    +        }
    +        Err(problem) => tracing::error!(
    +            "backup store: {kind} {} is unhealthy: {problem}; the server starts, restore and \
    +             delete of a backup in the store are refused, /health reports the backup_store \
    +             component unhealthy, and the check runs again each reconciler cycle{}",
    +            opened.root,
    +            rows_note(defaulted)
    +        ),
    +    }
    +    let configured = ConfiguredStore::new(kind, opened.root, opened.store, spool, outcome);
    +    Ok(StoreSetup::with_store(runtime, configured, spool_lock))
    +}
    +
    +/// What a filesystem store's startup checks decided.
    +enum LocalChecks {
    +    /// `validate` passed and the identity check ran, or `validate` failed on
    +    /// a store condition (module doc): the identity, or the problem that
    +    /// makes the store unhealthy.
    +    Done(Result),
    +    /// `validate` refused the root for a configuration reason.
    +    Refused(StoreError),
    +    /// Nothing answered within [`LOCAL_CHECK_TIMEOUT`].
    +    TimedOut,
    +}
    +
    +/// A filesystem store's startup checks: `validate`, then
    +/// [`store_identity::establish`], bounded by [`LOCAL_CHECK_TIMEOUT`] so a
    +/// root on a mount that stops answering after `open` does not hold the
    +/// listener. The store's synchronous filesystem calls run on the blocking
    +/// pool, so the bound fires whichever call stalls; a timed-out check leaves
    +/// that thread parked until the mount answers (module doc).
    +async fn run_local_checks(
    +    store: &dyn BackupStore,
    +    kind: StoreKind,
    +    root: &str,
    +    catalog: &C,
    +    backend: &str,
    +) -> LocalChecks {
    +    let checks = async {
    +        match store.validate().await {
    +            Ok(()) => LocalChecks::Done(
    +                store_identity::establish(store, kind, root, catalog, backend).await,
    +            ),
    +            // A store condition, not a configuration error: the probe write
    +            // or the root's resolution failed (module doc).
    +            Err(e @ (StoreError::Transport { .. } | StoreError::Io { .. })) => {
    +                LocalChecks::Done(Err(StoreProblem::from(e)))
    +            }
    +            Err(e) => LocalChecks::Refused(e),
    +        }
    +    };
    +    tokio::time::timeout(LOCAL_CHECK_TIMEOUT, checks)
    +        .await
    +        .unwrap_or(LocalChecks::TimedOut)
    +}
    +
    +/// Whether the catalog has backup rows for the defaulted root at `root`
    +/// (the label rows record), which ends the root's leniency (module doc). A
    +/// catalog that cannot be read is answered as having rows, so a read
    +/// failure never makes a used root's backups vanish behind a warning.
    +async fn default_root_has_rows(catalog: &C, root: &str) -> bool {
    +    match catalog.read_store_ids(StoreKind::Filesystem, root).await {
    +        Ok(ids) => !ids.is_empty(),
    +        Err(e) => {
    +            tracing::warn!(
    +                "backup store: could not read the catalog's rows for the default root {root} \
    +                 ({e}); the root is handled as a configured one"
    +            );
    +            true
    +        }
    +    }
    +}
    +
    +/// The clause a defaulted root's startup error or unhealthy line carries,
    +/// saying why it was not the warning.
    +fn rows_note(defaulted: bool) -> &'static str {
    +    if defaulted {
    +        " (the default root beside the database file; the catalog has backup rows for it, so \
    +         it is handled as a configured root: fix it, or configure [backup.store])"
    +    } else {
    +        ""
    +    }
    +}
    +
    +/// The warning for a defaulted root whose creation, `open`, `validate` or
    +/// identity check failed while the catalog has no rows for it: the store is
    +/// left unconfigured, as the module doc says.
    +fn warn_default_root_unusable(root: &str, problem: &str) {
    +    tracing::warn!(
    +        "backup store: the default root {root} failed its check ({problem}); the catalog has no \
    +         backup rows for it, so the store is unconfigured and CreateBackup writes in-catalog \
    +         backups. Fix the root (it must be owned by this user with mode 0700, under directories \
    +         owned by root or this user that others cannot write to, and its extenddb-store.json, \
    +         if any, must name this catalog: a catalog re-created by `destroy` and `init` needs the \
    +         old file removed or a new root), configure [backup.store] to choose another, or set \
    +         kind = \"disabled\" to silence this"
    +    );
    +}
    +
    +/// The store line of the console banner: what `[backup.store]` and the
    +/// backend select, before anything is opened. The configuration's own
    +/// refusals are not applied here; `configure` reports them.
    +#[must_use]
    +pub fn banner_line(backup: &BackupConfig, database_file: Option<&Path>) -> String {
    +    let resolved = match backup.resolve_store(database_file) {
    +        Ok(resolved) => resolved,
    +        Err(e) => return format!("backup store: configuration error ({e})"),
    +    };
    +    let spool = resolved
    +        .spool_path
    +        .as_ref()
    +        .map(|p| format!(", spool {}", p.display()))
    +        .unwrap_or_default();
    +    match resolved.store {
    +        EffectiveStore::Unconfigured(Unconfigured::Absent) => {
    +            "backup store: none ([backup.store] is absent); CreateBackup writes in-catalog \
    +             backups"
    +                .to_owned()
    +        }
    +        EffectiveStore::Unconfigured(Unconfigured::Disabled) => {
    +            "backup store: disabled; CreateBackup writes in-catalog backups".to_owned()
    +        }
    +        EffectiveStore::Configured { config, defaulted } => format!(
    +            "backup store: {}{}{spool}",
    +            describe_target(&config),
    +            if defaulted {
    +                " (the default beside the database file)"
    +            } else {
    +                ""
    +            }
    +        ),
    +    }
    +}
    +
    +/// The root a configuration names, for a startup error: the bucket of an S3
    +/// store, the path of a filesystem one. Neither is a secret, and an operator
    +/// reading the error wants to know which store it is about.
    +fn describe_target(config: &extenddb_backup_store::BackupStoreConfig) -> String {
    +    match config.kind {
    +        BackupStoreKind::S3 => config.s3.as_ref().map_or_else(
    +            || "s3".to_owned(),
    +            |s3| format!("s3 bucket {:?}", s3.bucket),
    +        ),
    +        BackupStoreKind::Filesystem => config.filesystem.as_ref().map_or_else(
    +            || "filesystem".to_owned(),
    +            |fs| {
    +                // A path that is empty or begins or ends with whitespace is
    +                // quoted, so the message shows where it ends.
    +                let path = fs.path.to_string_lossy();
    +                if path.is_empty() {
    +                    "filesystem path \"\" (empty)".to_owned()
    +                } else if path.trim() != path {
    +                    format!("filesystem path {path:?}")
    +                } else {
    +                    format!("filesystem path {path}")
    +                }
    +            },
    +        ),
    +        BackupStoreKind::Disabled => "disabled".to_owned(),
    +    }
    +}
    +
    +/// Create `path` with mode `0700` unless it exists.
    +fn create_private_dir_if_missing(path: &Path) -> std::io::Result<()> {
    +    if path.exists() {
    +        return Ok(());
    +    }
    +    let mut builder = std::fs::DirBuilder::new();
    +    #[cfg(unix)]
    +    {
    +        use std::os::unix::fs::DirBuilderExt;
    +        builder.mode(0o700);
    +    }
    +    builder.recursive(true).create(path)
    +}
    +
    +/// Create the spool directory (`0700`) if missing and take its `.lock`
    +/// (`0600`, `flock`), refusing a spool another live instance holds. An
    +/// existing spool must meet the root's owner-only rule (owned by this user,
    +/// no group or other permission bits): it will hold downloaded backup
    +/// files, and a readable lock file would let any local user hold the spool.
    +fn take_spool(path: &Path) -> anyhow::Result<(Spool, SpoolLock)> {
    +    create_private_dir_if_missing(path).map_err(|e| {
    +        anyhow::anyhow!(
    +            "backup.spool_path {} cannot be created: {e}",
    +            path.display()
    +        )
    +    })?;
    +    let canonical = path
    +        .canonicalize()
    +        .map_err(|e| anyhow::anyhow!("backup.spool_path {} is not usable: {e}", path.display()))?;
    +    check_spool_mode(&canonical)?;
    +    let lock_path = canonical.join(SPOOL_LOCK_FILE);
    +    let mut options = std::fs::OpenOptions::new();
    +    options.read(true).write(true).create(true).truncate(false);
    +    #[cfg(unix)]
    +    {
    +        use std::os::unix::fs::OpenOptionsExt;
    +        options.mode(0o600);
    +    }
    +    let file = options
    +        .open(&lock_path)
    +        .map_err(|e| anyhow::anyhow!("cannot open the spool lock {}: {e}", lock_path.display()))?;
    +    // `create` honours the umask and the file may predate this rule: flock
    +    // needs only a read handle, so a readable lock file would let any local
    +    // user hold the spool. Set the mode on every open.
    +    #[cfg(unix)]
    +    {
    +        use std::os::unix::fs::PermissionsExt;
    +        file.set_permissions(std::fs::Permissions::from_mode(0o600))
    +            .map_err(|e| anyhow::anyhow!("cannot set the mode of {}: {e}", lock_path.display()))?;
    +    }
    +    try_lock_exclusive(&file).map_err(|e| {
    +        if e.kind() == std::io::ErrorKind::WouldBlock {
    +            anyhow::anyhow!(
    +                "backup.spool_path {} is held by another running extenddb instance (lock {}); \
    +                 every instance on a host needs a spool of its own",
    +                canonical.display(),
    +                lock_path.display()
    +            )
    +        } else {
    +            anyhow::anyhow!("cannot lock {}: {e}", lock_path.display())
    +        }
    +    })?;
    +    let spool = Spool {
    +        spool_host: format!("{}:{}", hostname(), canonical.display()),
    +        path: canonical,
    +    };
    +    Ok((
    +        spool,
    +        SpoolLock {
    +            _file: file,
    +            path: lock_path,
    +        },
    +    ))
    +}
    +
    +/// The owner-only rule on the spool directory, the one the filesystem store
    +/// applies to its root (`extenddb_backup_store::check_private_directory`):
    +/// owned by this user with mode `0700`, under directories owned by root or
    +/// this user that others cannot write to.
    +fn check_spool_mode(spool: &Path) -> anyhow::Result<()> {
    +    if !spool.is_dir() {
    +        anyhow::bail!("backup.spool_path {} is not a directory", spool.display());
    +    }
    +    extenddb_backup_store::check_private_directory(spool, "spool").map_err(|e| {
    +        anyhow::anyhow!(
    +            "backup.spool_path {} is not private: {e}; the spool must have mode 0700 under \
    +             directories others cannot write to",
    +            spool.display()
    +        )
    +    })
    +}
    +
    +/// This host's name, for `spool_host` and a maintenance claim's holder;
    +/// `unknown-host` when the system does not say.
    +#[must_use]
    +pub fn hostname() -> String {
    +    #[cfg(unix)]
    +    {
    +        let mut buf = [0u8; 256];
    +        // SAFETY: `gethostname` writes at most `len` bytes into the buffer it
    +        // is given, which is valid for its whole length.
    +        let rc = unsafe { libc::gethostname(buf.as_mut_ptr().cast(), buf.len()) };
    +        if rc == 0 {
    +            let end = buf.iter().position(|b| *b == 0).unwrap_or(buf.len());
    +            if let Ok(name) = std::str::from_utf8(&buf[..end])
    +                && !name.is_empty()
    +            {
    +                return name.to_owned();
    +            }
    +        }
    +    }
    +    "unknown-host".to_owned()
    +}
    +
    +#[cfg(unix)]
    +fn try_lock_exclusive(file: &std::fs::File) -> std::io::Result<()> {
    +    use std::os::unix::io::AsRawFd;
    +    // SAFETY: `flock` takes a file descriptor and flags and touches no memory
    +    // the caller owns; the descriptor belongs to `file`, borrowed for the
    +    // call.
    +    let rc = unsafe { libc::flock(file.as_raw_fd(), libc::LOCK_EX | libc::LOCK_NB) };
    +    if rc == 0 {
    +        Ok(())
    +    } else {
    +        Err(std::io::Error::last_os_error())
    +    }
    +}
    +
    +#[cfg(not(unix))]
    +fn try_lock_exclusive(_file: &std::fs::File) -> std::io::Result<()> {
    +    tracing::warn!(
    +        "no file lock is taken on the spool on this platform: nothing prevents a second \
    +         extenddb instance from using the same spool_path"
    +    );
    +    Ok(())
    +}
    +
    +#[cfg(all(test, unix))]
    +mod tests {
    +    use super::*;
    +
    +    fn mode_of(path: &Path) -> u32 {
    +        use std::os::unix::fs::PermissionsExt;
    +        std::fs::metadata(path).unwrap().permissions().mode() & 0o777
    +    }
    +
    +    /// The spool is created private with a private lock file, a second take
    +    /// of the same spool is refused while the first lock lives, and a spool
    +    /// elsewhere is not.
    +    #[test]
    +    fn the_spool_lock_refuses_a_second_instance() {
    +        let dir = tempfile::tempdir().unwrap();
    +        let spool = dir.path().join("spool");
    +        let (first, lock) = take_spool(&spool).unwrap();
    +        assert_eq!(mode_of(&spool), 0o700);
    +        assert_eq!(mode_of(&spool.join(SPOOL_LOCK_FILE)), 0o600);
    +        assert!(
    +            first
    +                .spool_host
    +                .ends_with(&format!(":{}", spool.canonicalize().unwrap().display()))
    +        );
    +        assert!(!first.spool_host.starts_with(':'), "the host name is set");
    +        let err = take_spool(&spool).unwrap_err().to_string();
    +        assert!(
    +            err.contains("held by another running extenddb instance"),
    +            "{err}"
    +        );
    +        assert!(err.contains(&spool.display().to_string()), "{err}");
    +        take_spool(&dir.path().join("other")).unwrap();
    +        drop(lock);
    +        take_spool(&spool).expect("free once the holder is gone");
    +    }
    +
    +    /// An existing spool that group or others can reach, or that another
    +    /// user owns, is refused naming the path and the mode; one with mode
    +    /// 0700 is taken.
    +    #[test]
    +    fn an_existing_spool_must_be_owner_only() {
    +        use std::os::unix::fs::PermissionsExt;
    +        let dir = tempfile::tempdir().unwrap();
    +        let spool = dir.path().join("spool");
    +        std::fs::create_dir(&spool).unwrap();
    +        for mode in [0o755u32, 0o750, 0o707, 0o770] {
    +            std::fs::set_permissions(&spool, std::fs::Permissions::from_mode(mode)).unwrap();
    +            let err = take_spool(&spool).unwrap_err().to_string();
    +            assert!(
    +                err.contains(&format!("{mode:04o}")) && err.contains("mode 0700"),
    +                "{mode:04o}: {err}"
    +            );
    +            assert!(err.contains("not private"), "{err}");
    +            assert!(err.contains(&spool.display().to_string()), "{err}");
    +            assert!(
    +                !spool.join(SPOOL_LOCK_FILE).exists(),
    +                "no lock file is created"
    +            );
    +        }
    +        std::fs::set_permissions(&spool, std::fs::Permissions::from_mode(0o700)).unwrap();
    +        take_spool(&spool).expect("an owner-only spool is taken");
    +        // A spool under a parent others can write to is refused as the root
    +        // would be: the parent could be swapped under it.
    +        let parent = dir.path().join("open");
    +        std::fs::create_dir(&parent).unwrap();
    +        std::fs::set_permissions(&parent, std::fs::Permissions::from_mode(0o777)).unwrap();
    +        let under = parent.join("spool");
    +        std::fs::create_dir(&under).unwrap();
    +        std::fs::set_permissions(&under, std::fs::Permissions::from_mode(0o700)).unwrap();
    +        let err = take_spool(&under).unwrap_err().to_string();
    +        assert!(
    +            err.contains("an ancestor of the spool") && err.contains("0777"),
    +            "{err}"
    +        );
    +    }
    +
    +    /// A lock file that predates the mode rule is narrowed to 0600 on open.
    +    #[test]
    +    fn a_permissive_lock_file_is_narrowed() {
    +        use std::os::unix::fs::PermissionsExt;
    +        let dir = tempfile::tempdir().unwrap();
    +        let spool = dir.path().join("spool");
    +        std::fs::create_dir(&spool).unwrap();
    +        std::fs::set_permissions(&spool, std::fs::Permissions::from_mode(0o700)).unwrap();
    +        let lock = spool.join(SPOOL_LOCK_FILE);
    +        std::fs::write(&lock, b"").unwrap();
    +        std::fs::set_permissions(&lock, std::fs::Permissions::from_mode(0o644)).unwrap();
    +        let _held = take_spool(&spool).unwrap();
    +        assert_eq!(mode_of(&lock), 0o600);
    +    }
    +
    +    // --- the startup checks against doubles of the catalog and the store ---
    +
    +    use extenddb_backup_store::{ByteStream, ObjectMeta};
    +    use extenddb_core::backup_format::BackupArn;
    +    use extenddb_storage::BoxedFuture;
    +    use extenddb_storage::backup_catalog::{
    +        BackupKey, BackupRow, BackupStatus, DeleteStart, JobClaim, MaintenanceClaim, RestoreJobRow,
    +        RestoreTargetKey, TimeBounds,
    +    };
    +    use extenddb_storage::error::StorageError;
    +    use futures::stream::BoxStream;
    +
    +    /// A catalog whose rows for every root carry `ids`, or whose rows cannot
    +    /// be read (`rows_unreadable`), with no backup row to list; every other
    +    /// method is an error, since the startup checks never reach it.
    +    #[derive(Default)]
    +    struct StubCatalog {
    +        ids: Vec,
    +        rows_unreadable: bool,
    +    }
    +
    +    fn unused() -> BoxedFuture<'static, Result> {
    +        Box::pin(async {
    +            Err(StorageError::Internal(
    +                "not reached by the startup checks".to_owned(),
    +            ))
    +        })
    +    }
    +
    +    impl BackupCatalog for StubCatalog {
    +        fn catalog_id(&self) -> BoxedFuture<'_, Result> {
    +            Box::pin(async { Ok("catalog-1".to_owned()) })
    +        }
    +        fn claim_backup<'a>(
    +            &'a self,
    +            _row: &'a BackupRow,
    +        ) -> BoxedFuture<'a, Result, StorageError>> {
    +            unused()
    +        }
    +        fn list_backup_rows<'a>(
    +            &'a self,
    +            _status: Option,
    +            _store: &'a StoreIdentity,
    +        ) -> BoxStream<'a, Result> {
    +            Box::pin(futures::stream::empty())
    +        }
    +        fn list_account_backup_rows<'a>(
    +            &'a self,
    +            _account_id: &'a str,
    +            _table_name: Option<&'a str>,
    +            _bounds: &'a TimeBounds,
    +        ) -> BoxStream<'a, Result> {
    +            Box::pin(futures::stream::empty())
    +        }
    +        fn read_backup_row<'a>(
    +            &'a self,
    +            _arn: &'a BackupArn,
    +        ) -> BoxedFuture<'a, Result, StorageError>> {
    +            unused()
    +        }
    +        fn start_backup_delete<'a>(
    +            &'a self,
    +            _key: &'a BackupKey,
    +        ) -> BoxedFuture<'a, Result> {
    +            unused()
    +        }
    +        fn set_backup_deleted<'a>(
    +            &'a self,
    +            _key: &'a BackupKey,
    +        ) -> BoxedFuture<'a, Result> {
    +            unused()
    +        }
    +        fn remove_backup_row<'a>(
    +            &'a self,
    +            _key: &'a BackupKey,
    +        ) -> BoxedFuture<'a, Result> {
    +            unused()
    +        }
    +        fn remove_dead_creating_backup_row<'a>(
    +            &'a self,
    +            _key: &'a BackupKey,
    +            _stale_after: std::time::Duration,
    +        ) -> BoxedFuture<'a, Result> {
    +            unused()
    +        }
    +        fn backup_in_use<'a>(
    +            &'a self,
    +            _key: &'a BackupKey,
    +        ) -> BoxedFuture<'a, Result, StorageError>> {
    +            unused()
    +        }
    +        fn remove_restore_target<'a>(
    +            &'a self,
    +            _key: &'a RestoreTargetKey,
    +        ) -> BoxedFuture<'a, Result> {
    +            unused()
    +        }
    +        fn list_dead_restore_jobs(
    +            &self,
    +            _stale_after: std::time::Duration,
    +        ) -> BoxedFuture<'_, Result, StorageError>> {
    +            unused()
    +        }
    +        fn fail_restore_job<'a>(
    +            &'a self,
    +            _key: &'a RestoreTargetKey,
    +            _reason: &'a str,
    +        ) -> BoxedFuture<'a, Result> {
    +            unused()
    +        }
    +        fn read_store_ids<'a>(
    +            &'a self,
    +            _store_kind: StoreKind,
    +            _store_root: &'a str,
    +        ) -> BoxedFuture<'a, Result, StorageError>> {
    +            if self.rows_unreadable {
    +                return Box::pin(async {
    +                    Err(StorageError::Internal(
    +                        "the catalog is not answering".to_owned(),
    +                    ))
    +                });
    +            }
    +            let ids = self.ids.clone();
    +            Box::pin(async move { Ok(ids) })
    +        }
    +        fn maintenance_claim<'a>(
    +            &'a self,
    +            _holder: &'a str,
    +            _stale_after: std::time::Duration,
    +        ) -> BoxedFuture<'a, Result, StorageError>> {
    +            unused()
    +        }
    +        fn check_maintenance_claim(
    +            &self,
    +            _stale_after: std::time::Duration,
    +        ) -> BoxedFuture<'_, Result> {
    +            unused()
    +        }
    +    }
    +
    +    /// A store whose `validate` answers as told: never (a mount that has
    +    /// stopped answering), or with a given error; nothing else is reached.
    +    enum Validate {
    +        Never,
    +        Fails(fn() -> StoreError),
    +    }
    +
    +    struct FakeStore(Validate);
    +
    +    fn not_reached() -> futures::future::BoxFuture<'static, Result>
    +    {
    +        Box::pin(async { Err(StoreError::Other("not reached".to_owned())) })
    +    }
    +
    +    impl BackupStore for FakeStore {
    +        fn put<'a>(
    +            &'a self,
    +            _key: &'a str,
    +            _body: ByteStream,
    +        ) -> futures::future::BoxFuture<'a, Result> {
    +            not_reached()
    +        }
    +        fn put_if_absent<'a>(
    +            &'a self,
    +            _key: &'a str,
    +            _body: ByteStream,
    +        ) -> futures::future::BoxFuture<'a, Result, StoreError>> {
    +            not_reached()
    +        }
    +        fn get<'a>(
    +            &'a self,
    +            _key: &'a str,
    +        ) -> futures::future::BoxFuture<'a, Result> {
    +            not_reached()
    +        }
    +        fn head<'a>(
    +            &'a self,
    +            _key: &'a str,
    +        ) -> futures::future::BoxFuture<'a, Result, StoreError>> {
    +            not_reached()
    +        }
    +        fn list<'a>(&'a self, _prefix: &'a str) -> BoxStream<'a, Result> {
    +            Box::pin(futures::stream::empty())
    +        }
    +        fn delete_prefix<'a>(
    +            &'a self,
    +            _prefix: &'a str,
    +        ) -> futures::future::BoxFuture<'a, Result> {
    +            not_reached()
    +        }
    +        fn validate(&self) -> futures::future::BoxFuture<'_, Result<(), StoreError>> {
    +            match &self.0 {
    +                Validate::Never => Box::pin(std::future::pending()),
    +                Validate::Fails(make) => Box::pin(std::future::ready(Err(make()))),
    +            }
    +        }
    +    }
    +
    +    /// A SQLite deployment's `[backup]` without a store, and the database
    +    /// file whose directory the default root is resolved against.
    +    fn defaulted(dir: &Path) -> (BackupConfig, PathBuf) {
    +        (BackupConfig::default(), dir.join("extenddb.sqlite"))
    +    }
    +
    +    /// The defaulted root's leniency ends with the catalog's first row for
    +    /// it. The root is created empty (no identity file) each time: with no
    +    /// rows for it, a fresh root gets its file and the store is healthy;
    +    /// with rows for it and no file, an explicit root would be unhealthy
    +    /// (`identity file missing`), and so is the defaulted one, configured
    +    /// and reported rather than left unconfigured behind a warning.
    +    #[tokio::test]
    +    async fn a_defaulted_root_with_rows_is_handled_as_a_configured_one() {
    +        let dir = tempfile::tempdir().unwrap();
    +        let (backup, database) = defaulted(dir.path());
    +        let root = dir.path().join("backups");
    +
    +        // Rows for the root (seeded, as a server that used the root leaves
    +        // them), no identity file: an explicit root's `identity file
    +        // missing`, on `/health` and in the refusals.
    +        let with_rows = StubCatalog {
    +            ids: vec!["store-1".to_owned()],
    +            ..StubCatalog::default()
    +        };
    +        let setup = configure_described(
    +            &backup,
    +            Some(&database),
    +            "sqlite",
    +            &with_rows,
    +            Caller::Server,
    +        )
    +        .await
    +        .unwrap();
    +        assert_eq!(setup.not_configured, None, "the store is configured");
    +        let store = setup
    +            .runtime
    +            .configured_store()
    +            .expect("a defaulted root with rows is a configured store");
    +        assert_eq!(store.kind, StoreKind::Filesystem);
    +        assert_eq!(
    +            store.root,
    +            root.canonicalize().unwrap().display().to_string()
    +        );
    +        assert_eq!(
    +            store.health().category(),
    +            Some("identity file missing"),
    +            "{:?}",
    +            store.health()
    +        );
    +        let report = setup.runtime.store_health().unwrap();
    +        assert_eq!(
    +            (report.status, report.reason),
    +            ("unhealthy", Some("identity file missing"))
    +        );
    +        assert!(
    +            !root.join(store_identity::STORE_IDENTITY_KEY).exists(),
    +            "no file is created over rows that name another id"
    +        );
    +
    +        // No rows: the fresh-root case, the file created, the store healthy.
    +        let without_rows = StubCatalog::default();
    +        let setup = configure_described(
    +            &backup,
    +            Some(&database),
    +            "sqlite",
    +            &without_rows,
    +            Caller::Server,
    +        )
    +        .await
    +        .unwrap();
    +        assert_eq!(setup.not_configured, None);
    +        assert!(setup.runtime.configured_store().unwrap().is_healthy());
    +        assert!(root.join(store_identity::STORE_IDENTITY_KEY).exists());
    +
    +        // No rows and a failed check (the file names another catalog, as a
    +        // catalog re-created by `destroy` and `init` leaves it): the
    +        // warning path, with the reason carried out for the commands.
    +        let foreign = store_identity::StoreIdentityFile::parse(
    +            br#"{"format_version":1,"store_id":"s-9","catalog_id":"other","backend":"sqlite","created_at_ms":0}"#,
    +        )
    +        .unwrap();
    +        std::fs::write(
    +            root.join(store_identity::STORE_IDENTITY_KEY),
    +            foreign.to_bytes(),
    +        )
    +        .unwrap();
    +        let setup = configure_described(
    +            &backup,
    +            Some(&database),
    +            "sqlite",
    +            &without_rows,
    +            Caller::Server,
    +        )
    +        .await
    +        .unwrap();
    +        assert!(setup.runtime.configured_store().is_none());
    +        let why = setup.not_configured.expect("the reason is carried out");
    +        assert!(
    +            matches!(&why, NotConfigured::DefaultRootUnusable { root, problem }
    +                if root.contains("backups") && problem.contains("names catalog other")),
    +            "{why:?}"
    +        );
    +        assert!(why.to_string().contains("no rows for it"), "{why}");
    +        // The same file over rows for the root: a configured, unhealthy
    +        // store (`owned by another catalog`), not the warning.
    +        let setup = configure_described(
    +            &backup,
    +            Some(&database),
    +            "sqlite",
    +            &with_rows,
    +            Caller::Server,
    +        )
    +        .await
    +        .unwrap();
    +        assert_eq!(
    +            setup
    +                .runtime
    +                .configured_store()
    +                .unwrap()
    +                .health()
    +                .category(),
    +            Some("owned by another catalog")
    +        );
    +    }
    +
    +    /// A defaulted root `open` refuses (a parent others can write to) is the
    +    /// warning while the catalog has no rows for it, and a startup error
    +    /// naming the root once it has; an explicitly configured root is the
    +    /// startup error either way.
    +    #[tokio::test]
    +    async fn a_defaulted_root_open_refuses_is_a_warning_only_without_rows() {
    +        use std::os::unix::fs::PermissionsExt;
    +        let dir = tempfile::tempdir().unwrap();
    +        std::fs::set_permissions(dir.path(), std::fs::Permissions::from_mode(0o777)).unwrap();
    +        let (backup, database) = defaulted(dir.path());
    +        let setup = configure_described(
    +            &backup,
    +            Some(&database),
    +            "sqlite",
    +            &StubCatalog::default(),
    +            Caller::Server,
    +        )
    +        .await
    +        .unwrap();
    +        let why = setup.not_configured.expect("the warning path");
    +        assert!(
    +            matches!(&why, NotConfigured::DefaultRootUnusable { problem, .. }
    +                if problem.contains("writable by group or others")),
    +            "{why:?}"
    +        );
    +        let err = configure_described(
    +            &backup,
    +            Some(&database),
    +            "sqlite",
    +            &StubCatalog {
    +                ids: vec!["store-1".to_owned()],
    +                ..StubCatalog::default()
    +            },
    +            Caller::Server,
    +        )
    +        .await
    +        .err()
    +        .expect("with rows, the refusal stops startup")
    +        .to_string();
    +        assert!(err.contains("cannot be opened"), "{err}");
    +        assert!(err.contains("backup rows for it"), "{err}");
    +        assert!(err.contains(&dir.path().display().to_string()), "{err}");
    +    }
    +
    +    /// A catalog whose rows cannot be read is answered as having rows, so a
    +    /// defaulted root whose check then fails is a configured, unhealthy
    +    /// store and not the warning that leaves the store unconfigured: the
    +    /// root may hold backups the read would have shown. Here the root
    +    /// carries a foreign identity file; the read failure reaches `establish`
    +    /// first, so the problem is the catalog's (`catalog unreadable`), and
    +    /// the next cycle's check reads again.
    +    #[tokio::test]
    +    async fn an_unreadable_catalog_keeps_a_failing_default_root_configured() {
    +        use std::os::unix::fs::PermissionsExt;
    +        let dir = tempfile::tempdir().unwrap();
    +        let (backup, database) = defaulted(dir.path());
    +        let root = dir.path().join("backups");
    +        std::fs::create_dir(&root).unwrap();
    +        std::fs::set_permissions(&root, std::fs::Permissions::from_mode(0o700)).unwrap();
    +        let foreign = store_identity::StoreIdentityFile::parse(
    +            br#"{"format_version":1,"store_id":"s-9","catalog_id":"other","backend":"sqlite","created_at_ms":0}"#,
    +        )
    +        .unwrap();
    +        std::fs::write(
    +            root.join(store_identity::STORE_IDENTITY_KEY),
    +            foreign.to_bytes(),
    +        )
    +        .unwrap();
    +        let unreadable = StubCatalog {
    +            rows_unreadable: true,
    +            ..StubCatalog::default()
    +        };
    +        let setup = configure_described(
    +            &backup,
    +            Some(&database),
    +            "sqlite",
    +            &unreadable,
    +            Caller::Server,
    +        )
    +        .await
    +        .unwrap();
    +        assert_eq!(setup.not_configured, None, "the store is configured");
    +        let store = setup
    +            .runtime
    +            .configured_store()
    +            .expect("a root whose rows cannot be read is a configured store");
    +        assert_eq!(
    +            store.health().category(),
    +            Some("catalog unreadable"),
    +            "{:?}",
    +            store.health()
    +        );
    +    }
    +
    +    /// The timed startup checks: a store that never answers leaves the
    +    /// server starting with the store `checking` after `LOCAL_CHECK_TIMEOUT`
    +    /// (not sooner, since a store that answers must get its outcome; not
    +    /// later, since the listener waits on this); `validate`'s I/O failure
    +    /// is a store condition that degrades (`failed validation`), its denial
    +    /// a configuration refusal.
    +    #[tokio::test(start_paused = true)]
    +    async fn the_startup_checks_are_bounded_and_split_store_conditions_from_refusals() {
    +        let catalog = StubCatalog::default();
    +        // The bound's value is part of the rule (a local disk answers in
    +        // milliseconds), so it is pinned here rather than read back.
    +        assert_eq!(LOCAL_CHECK_TIMEOUT, std::time::Duration::from_secs(5));
    +        let started = tokio::time::Instant::now();
    +        let stalled = FakeStore(Validate::Never);
    +        // The test's own guard, far past the bound: a check with no bound
    +        // fails here instead of hanging.
    +        let outcome = tokio::time::timeout(
    +            std::time::Duration::from_secs(120),
    +            run_local_checks(&stalled, StoreKind::Filesystem, "/r", &catalog, "test"),
    +        )
    +        .await
    +        .expect("the checks return at their bound");
    +        assert!(matches!(outcome, LocalChecks::TimedOut));
    +        assert_eq!(
    +            started.elapsed(),
    +            LOCAL_CHECK_TIMEOUT,
    +            "the bound is the whole wait, no more and no less"
    +        );
    +
    +        let io = FakeStore(Validate::Fails(|| StoreError::Io {
    +            context: "write probe /r/.put-validate.tmp".to_owned(),
    +            source: std::io::Error::from_raw_os_error(28),
    +        }));
    +        let outcome = run_local_checks(&io, StoreKind::Filesystem, "/r", &catalog, "test").await;
    +        let LocalChecks::Done(Err(problem)) = outcome else {
    +            panic!("an I/O failure of the probe degrades the store");
    +        };
    +        assert_eq!(problem.category(), "failed validation");
    +        assert!(problem.to_string().contains("write probe"), "{problem}");
    +
    +        let denied = FakeStore(Validate::Fails(|| {
    +            StoreError::PermissionDenied("/r has mode 0755".to_owned())
    +        }));
    +        let outcome =
    +            run_local_checks(&denied, StoreKind::Filesystem, "/r", &catalog, "test").await;
    +        assert!(matches!(
    +            outcome,
    +            LocalChecks::Refused(StoreError::PermissionDenied(_))
    +        ));
    +    }
    +
    +    /// `NotConfigured`'s sentences: each says why there are no store-backed
    +    /// backups, and the unusable root's names the root and the failure.
    +    #[test]
    +    fn not_configured_reads_as_the_commands_print_it() {
    +        assert!(
    +            NotConfigured::Absent
    +                .to_string()
    +                .contains("[backup.store] is absent")
    +        );
    +        assert!(NotConfigured::Disabled.to_string().contains("is disabled"));
    +        let missing = NotConfigured::DefaultRootMissing(PathBuf::from("/srv/db/backups"));
    +        assert!(
    +            missing
    +                .to_string()
    +                .contains("/srv/db/backups does not exist yet")
    +        );
    +        let unusable = NotConfigured::DefaultRootUnusable {
    +            root: "/srv/db/backups".to_owned(),
    +            problem: "extenddb-store.json names catalog other".to_owned(),
    +        }
    +        .to_string();
    +        assert!(
    +            unusable.contains("/srv/db/backups failed its check"),
    +            "{unusable}"
    +        );
    +        assert!(unusable.contains("names catalog other"), "{unusable}");
    +        assert!(unusable.contains("no rows for it"), "{unusable}");
    +    }
    +
    +    /// The banner and the startup errors spell a root as written, quoting
    +    /// one that is empty or begins or ends with whitespace so the message
    +    /// shows where the path ends.
    +    #[test]
    +    fn a_root_with_edge_whitespace_is_quoted_in_the_banner() {
    +        let line = |path: &str| {
    +            let backup = BackupConfig {
    +                store: Some(extenddb_backup_store::BackupStoreConfig {
    +                    kind: BackupStoreKind::Filesystem,
    +                    filesystem: Some(extenddb_backup_store::FilesystemStoreConfig {
    +                        path: PathBuf::from(path),
    +                    }),
    +                    s3: None,
    +                }),
    +                ..BackupConfig::default()
    +            };
    +            banner_line(&backup, None)
    +        };
    +        assert!(
    +            line("/srv/backups ").contains("filesystem path \"/srv/backups \""),
    +            "{}",
    +            line("/srv/backups ")
    +        );
    +        assert!(
    +            line(" /srv/backups").contains("filesystem path \" /srv/backups\""),
    +            "{}",
    +            line(" /srv/backups")
    +        );
    +        assert!(
    +            line("/srv/backups").contains("filesystem path /srv/backups"),
    +            "{}",
    +            line("/srv/backups")
    +        );
    +        assert!(
    +            line("").contains("filesystem path \"\" (empty)"),
    +            "{}",
    +            line("")
    +        );
    +    }
    +}
    diff --git a/crates/server/src/lib.rs b/crates/server/src/lib.rs
    index a14715cb6..3b798214a 100755
    --- a/crates/server/src/lib.rs
    +++ b/crates/server/src/lib.rs
    @@ -11,6 +11,7 @@
     pub(crate) mod authorization;
     pub mod authz_cache;
     pub(crate) mod authz_resource;
    +pub mod backup_store;
     pub mod console;
     pub mod dev_credentials;
     mod handler;
    @@ -281,9 +282,39 @@ pub async fn start_server(
         Ok(())
     }
     
    -/// GET /health — REQ-OBS-006
    -async fn health_check() -> impl IntoResponse {
    -    (StatusCode::OK, axum::Json(json!({"status": "healthy"})))
    +/// GET /health — REQ-OBS-006.
    +///
    +/// Always 200 while the server answers: the data plane is up. `status` is
    +/// `healthy`, or `degraded` when a component is unhealthy, and `components`
    +/// carries each component the server has. Today the one component is
    +/// `backup_store`, present when a store is configured: `status` `healthy`,
    +/// `checking` (an S3 store's first check has not returned; the server's
    +/// `status` stays `healthy`) or `unhealthy`, `kind` (`filesystem` or `s3`)
    +/// and, when unhealthy, `reason`, the problem's category in one or two
    +/// words. The endpoint is unauthenticated, so the reason carries no id, ARN,
    +/// table name or root; the operator's log line does. A store problem never
    +/// takes the data plane out of rotation, which is why the HTTP status stays
    +/// 200.
    +async fn health_check(
    +    axum::extract::State(state): axum::extract::State>,
    +) -> impl IntoResponse {
    +    let mut components = serde_json::Map::new();
    +    let mut status = "healthy";
    +    if let Some(store) = state.backup_runtime.store_health() {
    +        let mut component = json!({
    +            "status": store.status,
    +            "kind": store.kind,
    +        });
    +        if let Some(reason) = store.reason {
    +            component["reason"] = serde_json::Value::String(reason.to_owned());
    +            status = "degraded";
    +        }
    +        components.insert("backup_store".to_owned(), component);
    +    }
    +    (
    +        StatusCode::OK,
    +        axum::Json(json!({ "status": status, "components": components })),
    +    )
     }
     
     async fn graceful_shutdown(pid_file: Option) {
    diff --git a/crates/server/src/serve.rs b/crates/server/src/serve.rs
    index a9dce5254..96d951dcc 100644
    --- a/crates/server/src/serve.rs
    +++ b/crates/server/src/serve.rs
    @@ -628,9 +628,21 @@ async fn serve_inner(params: ServeParams, port: u16) -> anyhow::Result<()> {
             initial_throttling,
         ));
     
    -    let backup_runtime = Arc::new(extenddb_engine::BackupRuntime::new(
    -        app_config.backup.max_concurrent_jobs,
    -    ));
    +    // The backup store: resolved per backend, opened, validated, its identity
    +    // checked. A configuration error stops startup here; a transport or
    +    // identity failure starts the server with the store unhealthy
    +    // (`backup_store` module doc). The spool lock lives until shutdown.
    +    let backend_name = app_config.storage.backend.clone();
    +    let (backup_runtime, _spool_lock) = crate::backup_store::configure(
    +        &app_config.backup,
    +        app_config.storage.database_file().as_deref(),
    +        &backend_name,
    +        &storage,
    +        crate::backup_store::Caller::Server,
    +    )
    +    .await?;
    +    let backup_runtime = Arc::new(backup_runtime);
    +    let backup_runtime_for_reconciler = Arc::clone(&backup_runtime);
         let state = AppState {
             storage: storage.clone(),
             backup_runtime,
    @@ -714,10 +726,12 @@ async fn serve_inner(params: ServeParams, port: u16) -> anyhow::Result<()> {
         // backends' workers rather than in each backend's `spawn_workers`.
         worker_handles.push(tokio::spawn(extenddb_engine::reconciler::run(
             storage.clone(),
    +        Arc::clone(&backup_runtime_for_reconciler),
             extenddb_engine::reconciler::ReconcilerSettings {
                 interval: std::time::Duration::from_secs(app_config.backup.reconcile_interval_secs),
                 stale_job_after: std::time::Duration::from_secs(app_config.backup.stale_job_after_secs),
             },
    +        backend_name,
             metrics.clone(),
             shutdown.clone(),
         )));
    diff --git a/crates/storage-cassandra/src/backup_catalog.rs b/crates/storage-cassandra/src/backup_catalog.rs
    index bfd2a2d2c..1cfab090f 100644
    --- a/crates/storage-cassandra/src/backup_catalog.rs
    +++ b/crates/storage-cassandra/src/backup_catalog.rs
    @@ -41,6 +41,20 @@
     //! server's `WriteTimeout` or `WriteFailure`, has an unknown outcome
     //! (`WriteOutcome::Unknown`: the Paxos round may still commit); every other
     //! error is `Err`.
    +//!
    +//! The maintenance claim is the one row of `backup_maintenance`, taken with
    +//! `INSERT ... IF NOT EXISTS`; when a row exists, one whose heartbeat is older
    +//! than the stale window on the coordinator's clock is taken over with
    +//! `IF owner_generation = `, so two commands racing for a dead holder's
    +//! row see one win. An insert whose answer was lost is settled by a re-read:
    +//! a row carrying the call's own generation is its claim. Heartbeats and the
    +//! release are lightweight transactions on the generation. The two reads of
    +//! the claim row that decide whether another process's claim is live
    +//! (`check_maintenance_claim`, the row before a takeover) run at
    +//! `LOCAL_SERIAL`, so a heartbeat an LWT just wrote is seen; the RUNNING
    +//! restore records `list_dead_restore_jobs` scans are a range read, where a
    +//! serial level is a local quorum with no Paxos read, so the scan reads at
    +//! quorum and the coordinator clock after it.
     
     use std::time::Duration;
     
    @@ -55,8 +69,8 @@ use cdrs_tokio::types::value::Value;
     use extenddb_core::backup_format::BackupArn;
     use extenddb_storage::backup_catalog::{
         BackupCatalog, BackupKey, BackupRow, BackupStatus, DeleteStart, Heartbeat, JobClaim, JobRow,
    -    JobStatus, RestoreJobRow, RestoreKind, RestoreTargetKey, StatusFields, StoreIdentity,
    -    StoreKind, TimeBounds, WriteOutcome, billing_mode_name, new_generation,
    +    JobStatus, MaintenanceClaim, RestoreJobRow, RestoreKind, RestoreTargetKey, StatusFields,
    +    StoreIdentity, StoreKind, TimeBounds, WriteOutcome, billing_mode_name, new_generation,
     };
     use extenddb_storage::error::StorageError;
     use futures::future::BoxFuture;
    @@ -387,6 +401,59 @@ impl JobClaim for CassandraBackupClaim {
     impl CassandraEngine {
         /// The index row `backup_arn` names in `account_id`, if there is one; an
         /// in-catalog row is `None`.
    +    /// Remove the three index entries of a `backups_by_arn` row this process
    +    /// has deleted: its status partition, its account listing and its table
    +    /// listing, in one logged batch.
    +    async fn remove_index_entries(&self, row: &BackupRow) -> Result<(), StorageError> {
    +        let catalog = self.catalog_keyspace();
    +        let created_at = seconds_to_millis(row.created_at);
    +        let batch = BatchQueryBuilder::new()
    +            .with_consistency(Consistency::LocalQuorum)
    +            .add_query(
    +                format!(
    +                    "DELETE FROM {catalog}.backup_index_by_status \
    +                     WHERE store_root = ? AND backup_status = ? AND backup_arn = ?"
    +                ),
    +                cdrs_tokio::query_values!(
    +                    row.store.store_root.as_str(),
    +                    row.status.as_str(),
    +                    row.backup_arn.as_str()
    +                ),
    +            )
    +            .add_query(
    +                format!(
    +                    "DELETE FROM {catalog}.backups_by_account \
    +                     WHERE account_id = ? AND created_at = ? AND backup_arn = ?"
    +                ),
    +                cdrs_tokio::query_values!(
    +                    row.account_id.as_str(),
    +                    created_at,
    +                    row.backup_arn.as_str()
    +                ),
    +            )
    +            .add_query(
    +                format!(
    +                    "DELETE FROM {catalog}.backups_by_table \
    +                     WHERE account_id = ? AND table_name = ? AND created_at = ? AND backup_arn = ?"
    +                ),
    +                cdrs_tokio::query_values!(
    +                    row.account_id.as_str(),
    +                    row.table_name.as_str(),
    +                    created_at,
    +                    row.backup_arn.as_str()
    +                ),
    +            );
    +        self.session
    +            .batch(
    +                batch
    +                    .build()
    +                    .map_err(|e| StorageError::Internal(e.to_string()))?,
    +            )
    +            .await
    +            .map_err(|e| StorageError::Internal(format!("Remove backup index rows: {e}")))?;
    +        Ok(())
    +    }
    +
         async fn read_index_row(
             &self,
             account_id: &str,
    @@ -934,12 +1001,12 @@ impl BackupCatalog for CassandraEngine {
                 if !matches!(row.status, BackupStatus::Deleted | BackupStatus::Failed) {
                     return Ok(false);
                 }
    -            let catalog = self.catalog_keyspace();
                 // The authoritative row first, conditioned on the tombstone
                 // status read; then its index entries.
                 let delete = format!(
    -                "DELETE FROM {catalog}.backups_by_arn WHERE account_id = ? AND backup_arn = ? \
    -                 IF backup_status = ?"
    +                "DELETE FROM {}.backups_by_arn WHERE account_id = ? AND backup_arn = ? \
    +                 IF backup_status = ?",
    +                self.catalog_keyspace()
                 );
                 let result = crate::cassandra_util::query_lwt(
                     &self.session,
    @@ -954,51 +1021,53 @@ impl BackupCatalog for CassandraEngine {
                 if !crate::cassandra_util::lwt_applied(&result)? {
                     return Ok(false);
                 }
    -            let created_at = seconds_to_millis(row.created_at);
    -            let batch = BatchQueryBuilder::new()
    -                .with_consistency(Consistency::LocalQuorum)
    -                .add_query(
    -                    format!(
    -                        "DELETE FROM {catalog}.backup_index_by_status \
    -                         WHERE store_root = ? AND backup_status = ? AND backup_arn = ?"
    -                    ),
    -                    cdrs_tokio::query_values!(
    -                        row.store.store_root.as_str(),
    -                        row.status.as_str(),
    -                        key.backup_arn.as_str()
    -                    ),
    -                )
    -                .add_query(
    -                    format!(
    -                        "DELETE FROM {catalog}.backups_by_account \
    -                         WHERE account_id = ? AND created_at = ? AND backup_arn = ?"
    -                    ),
    -                    cdrs_tokio::query_values!(
    -                        key.account_id.as_str(),
    -                        created_at,
    -                        key.backup_arn.as_str()
    -                    ),
    -                )
    -                .add_query(
    -                    format!(
    -                        "DELETE FROM {catalog}.backups_by_table \
    -                         WHERE account_id = ? AND table_name = ? AND created_at = ? AND backup_arn = ?"
    -                    ),
    -                    cdrs_tokio::query_values!(
    -                        key.account_id.as_str(),
    -                        row.table_name.as_str(),
    -                        created_at,
    -                        key.backup_arn.as_str()
    -                    ),
    -                );
    -            self.session
    -                .batch(
    -                    batch
    -                        .build()
    -                        .map_err(|e| StorageError::Internal(e.to_string()))?,
    -                )
    -                .await
    -                .map_err(|e| StorageError::Internal(format!("Remove backup index rows: {e}")))?;
    +            self.remove_index_entries(&row).await?;
    +            Ok(true)
    +        })
    +    }
    +
    +    fn remove_dead_creating_backup_row<'a>(
    +        &'a self,
    +        key: &'a BackupKey,
    +        stale_after: Duration,
    +    ) -> BoxFuture<'a, Result> {
    +        Box::pin(async move {
    +            let Some(row) = self
    +                .read_index_row(&key.account_id, &key.backup_arn)
    +                .await?
    +            else {
    +                return Ok(false);
    +            };
    +            if row.status != BackupStatus::Creating {
    +                return Ok(false);
    +            }
    +            let now = coordinator_now_millis(self).await?;
    +            let stale_ms = i64::try_from(stale_after.as_millis()).unwrap_or(i64::MAX);
    +            let heartbeat_at = seconds_to_millis(row.heartbeat_at);
    +            if heartbeat_at >= now.saturating_sub(stale_ms) {
    +                return Ok(false);
    +            }
    +            // Conditioned on the status and the heartbeat read: a heartbeat
    +            // written since (the owner is alive after all) leaves the row.
    +            let delete = format!(
    +                "DELETE FROM {}.backups_by_arn WHERE account_id = ? AND backup_arn = ? \
    +                 IF backup_status = 'CREATING' AND heartbeat_at = ?",
    +                self.catalog_keyspace()
    +            );
    +            let result = crate::cassandra_util::query_lwt(
    +                &self.session,
    +                &delete,
    +                cdrs_tokio::query_values!(
    +                    key.account_id.as_str(),
    +                    key.backup_arn.as_str(),
    +                    heartbeat_at
    +                ),
    +            )
    +            .await?;
    +            if !crate::cassandra_util::lwt_applied(&result)? {
    +                return Ok(false);
    +            }
    +            self.remove_index_entries(&row).await?;
                 Ok(true)
             })
         }
    @@ -1076,9 +1145,6 @@ impl BackupCatalog for CassandraEngine {
             stale_after: Duration,
         ) -> BoxFuture<'_, Result, StorageError>> {
             Box::pin(async move {
    -            let now = coordinator_now_millis(self).await?;
    -            #[allow(clippy::cast_possible_truncation, clippy::cast_possible_wrap)]
    -            let cutoff = now - stale_after.as_millis() as i64;
                 // Two filtered scans over the tables (CQL takes one equality on a
                 // non-key column per scan): RUNNING records, dead when their
                 // heartbeat is older than the cutoff; and FAILED records whose
    @@ -1087,11 +1153,15 @@ impl BackupCatalog for CassandraEngine {
                 // DELETING with its registration: the removal's claim landed and
                 // its drop did not), dead when their `restore_completed_at` (or,
                 // where none was stamped, their heartbeat) is that old. Both
    -            // kinds are few.
    +            // kinds are few. The clock is read after the scans, so no
    +            // heartbeat a scan returned is later than `now` (with a zero
    +            // window, every RUNNING record the scans saw is listed). A range
    +            // scan has no Paxos read (a serial level resolves to a local
    +            // quorum there), so it stays at quorum.
                 let columns = "account_id, table_name, table_id, table_status, \
                                restore_source_backup_arn, restore_kind, restore_status, \
                                restore_owner_generation, restore_heartbeat_at, restore_completed_at";
    -            let mut dead = Vec::new();
    +            let mut scanned = Vec::new();
                 for status in ["RUNNING", "FAILED"] {
                     let query = format!(
                         "SELECT {columns} FROM {}.tables WHERE restore_status = ? ALLOW FILTERING",
    @@ -1104,6 +1174,13 @@ impl BackupCatalog for CassandraEngine {
                         "dead restore jobs",
                     )
                     .await?;
    +                scanned.push((status, rows));
    +            }
    +            let now = coordinator_now_millis(self).await?;
    +            #[allow(clippy::cast_possible_truncation, clippy::cast_possible_wrap)]
    +            let cutoff = now - stale_after.as_millis() as i64;
    +            let mut dead = Vec::new();
    +            for (status, rows) in scanned {
                     for row in rows {
                         let heartbeat_at: Option =
                             row.get_by_name("restore_heartbeat_at").ok().flatten();
    @@ -1184,20 +1261,237 @@ impl BackupCatalog for CassandraEngine {
             })
         }
     
    -    fn check_maintenance_claim(&self) -> BoxFuture<'_, Result> {
    +    fn read_store_ids<'a>(
    +        &'a self,
    +        store_kind: StoreKind,
    +        store_root: &'a str,
    +    ) -> BoxFuture<'a, Result, StorageError>> {
             Box::pin(async move {
    +            // `backup_index_by_status` is partitioned by (root, status), so a
    +            // root's rows are its five status partitions; the entry carries
    +            // the kind and id, which is all this read needs (an orphan entry
    +            // left by a crash names the same store its row did).
                 let query = format!(
    -                "SELECT claim FROM {}.backup_maintenance LIMIT 1",
    +                "SELECT store_kind, store_id FROM {}.backup_index_by_status \
    +                 WHERE store_root = ? AND backup_status = ?",
                     self.catalog_keyspace()
                 );
    -            let rows = crate::cassandra_util::query_rows_quorum::(
    +            let mut ids = Vec::new();
    +            for status in [
    +                BackupStatus::Creating,
    +                BackupStatus::Available,
    +                BackupStatus::Deleting,
    +                BackupStatus::Deleted,
    +                BackupStatus::Failed,
    +            ] {
    +                let rows = crate::cassandra_util::query_rows_quorum::(
    +                    &self.session,
    +                    &query,
    +                    cdrs_tokio::query_values!(store_root, status.as_str()),
    +                    "status index",
    +                )
    +                .await?;
    +                for row in rows {
    +                    let kind: Option = row.get_by_name("store_kind").ok().flatten();
    +                    let id: Option = row.get_by_name("store_id").ok().flatten();
    +                    if kind.as_deref() == Some(store_kind.as_str())
    +                        && let Some(id) = id
    +                        && !ids.contains(&id)
    +                    {
    +                        ids.push(id);
    +                    }
    +                }
    +            }
    +            ids.sort();
    +            Ok(ids)
    +        })
    +    }
    +
    +    fn maintenance_claim<'a>(
    +        &'a self,
    +        holder: &'a str,
    +        stale_after: Duration,
    +    ) -> BoxFuture<'a, Result, StorageError>> {
    +        Box::pin(async move {
    +            // The row is the claim: `INSERT ... IF NOT EXISTS` takes it when
    +            // there is none; otherwise the row is read, and one whose
    +            // heartbeat is older than `stale_after` on the coordinator's clock
    +            // is taken over with `IF owner_generation = `, so two
    +            // commands racing for a dead holder's row see one win.
    +            let catalog = self.catalog_keyspace();
    +            let generation = new_generation();
    +            let insert = format!(
    +                "INSERT INTO {catalog}.backup_maintenance \
    +                 (claim, holder, owner_generation, claimed_at, heartbeat_at) \
    +                 VALUES (?, ?, ?, toTimestamp(now()), toTimestamp(now())) IF NOT EXISTS"
    +            );
    +            let inserted = conditional_write(
    +                &self.session,
    +                &insert,
    +                cdrs_tokio::query_values!(
    +                    MAINTENANCE_CLAIM_KEY,
    +                    holder,
    +                    generation_column(generation)
    +                ),
    +            )
    +            .await?;
    +            if inserted != WriteOutcome::Matched {
    +                let read = format!(
    +                    "SELECT holder, owner_generation, heartbeat_at FROM \
    +                     {catalog}.backup_maintenance WHERE claim = ?"
    +                );
    +                let rows = crate::cassandra_util::query_rows_serial::(
    +                    &self.session,
    +                    &read,
    +                    cdrs_tokio::query_values!(MAINTENANCE_CLAIM_KEY),
    +                    "maintenance claim",
    +                )
    +                .await?;
    +                let row = rows.first();
    +                let current_holder: Option =
    +                    row.and_then(|r| r.get_by_name("holder").ok().flatten());
    +                let current_generation: Option =
    +                    row.and_then(|r| r.get_by_name("owner_generation").ok().flatten());
    +                // An insert whose answer was lost (`Unknown`) may have
    +                // committed: a row carrying this call's generation is this
    +                // call's claim.
    +                if inserted == WriteOutcome::Unknown
    +                    && current_generation == Some(generation_column(generation))
    +                {
    +                    return Ok(Box::new(CassandraMaintenanceClaim {
    +                        engine: self.clone(),
    +                        generation,
    +                    }) as Box);
    +                }
    +                let heartbeat_at: Option =
    +                    row.and_then(|r| r.get_by_name("heartbeat_at").ok().flatten());
    +                let now = coordinator_now_millis(self).await?;
    +                let stale_ms = i64::try_from(stale_after.as_millis()).unwrap_or(i64::MAX);
    +                let stale = heartbeat_at.is_none_or(|at| at < now.saturating_sub(stale_ms));
    +                let refused = || {
    +                    StorageError::AlreadyExists(format!(
    +                        "the backup maintenance claim is held by {}",
    +                        current_holder
    +                            .as_deref()
    +                            .unwrap_or("another maintenance command")
    +                    ))
    +                };
    +                if !stale {
    +                    return Err(refused());
    +                }
    +                let take_over = format!(
    +                    "UPDATE {catalog}.backup_maintenance SET holder = ?, owner_generation = ?, \
    +                     claimed_at = toTimestamp(now()), heartbeat_at = toTimestamp(now()) \
    +                     WHERE claim = ? IF owner_generation = ?"
    +                );
    +                let taken = conditional_write(
    +                    &self.session,
    +                    &take_over,
    +                    cdrs_tokio::query_values!(
    +                        holder,
    +                        generation_column(generation),
    +                        MAINTENANCE_CLAIM_KEY,
    +                        current_generation
    +                    ),
    +                )
    +                .await?;
    +                if taken != WriteOutcome::Matched {
    +                    return Err(refused());
    +                }
    +            }
    +            Ok(Box::new(CassandraMaintenanceClaim {
    +                engine: self.clone(),
    +                generation,
    +            }) as Box)
    +        })
    +    }
    +
    +    fn check_maintenance_claim(
    +        &self,
    +        stale_after: Duration,
    +    ) -> BoxFuture<'_, Result> {
    +        Box::pin(async move {
    +            let query = format!(
    +                "SELECT heartbeat_at FROM {}.backup_maintenance WHERE claim = ?",
    +                self.catalog_keyspace()
    +            );
    +            let rows = crate::cassandra_util::query_rows_serial::(
                     &self.session,
                     &query,
    -                cdrs_tokio::query_values!(),
    +                cdrs_tokio::query_values!(MAINTENANCE_CLAIM_KEY),
                     "maintenance claim",
                 )
                 .await?;
    -            Ok(!rows.is_empty())
    +            let Some(row) = rows.first() else {
    +                return Ok(false);
    +            };
    +            let heartbeat_at: Option = row.get_by_name("heartbeat_at").ok().flatten();
    +            let Some(heartbeat_at) = heartbeat_at else {
    +                return Ok(false);
    +            };
    +            let now = coordinator_now_millis(self).await?;
    +            let stale_ms = i64::try_from(stale_after.as_millis()).unwrap_or(i64::MAX);
    +            Ok(heartbeat_at >= now.saturating_sub(stale_ms))
    +        })
    +    }
    +}
    +
    +/// The partition key of the one maintenance claim row.
    +const MAINTENANCE_CLAIM_KEY: &str = "claim";
    +
    +/// The holder's handle on the `backup_maintenance` row.
    +struct CassandraMaintenanceClaim {
    +    engine: CassandraEngine,
    +    generation: u64,
    +}
    +
    +impl MaintenanceClaim for CassandraMaintenanceClaim {
    +    fn generation(&self) -> u64 {
    +        self.generation
    +    }
    +
    +    fn heartbeat(&self) -> BoxFuture<'_, Result> {
    +        Box::pin(async move {
    +            let query = format!(
    +                "UPDATE {}.backup_maintenance SET heartbeat_at = toTimestamp(now()) \
    +                 WHERE claim = ? IF owner_generation = ?",
    +                self.engine.catalog_keyspace()
    +            );
    +            let result = crate::cassandra_util::query_lwt(
    +                &self.engine.session,
    +                &query,
    +                cdrs_tokio::query_values!(
    +                    MAINTENANCE_CLAIM_KEY,
    +                    generation_column(self.generation)
    +                ),
    +            )
    +            .await?;
    +            Ok(if crate::cassandra_util::lwt_applied(&result)? {
    +                Heartbeat::Confirmed
    +            } else {
    +                Heartbeat::Lost
    +            })
    +        })
    +    }
    +
    +    fn release(&self) -> BoxFuture<'_, Result<(), StorageError>> {
    +        Box::pin(async move {
    +            let query = format!(
    +                "DELETE FROM {}.backup_maintenance WHERE claim = ? IF owner_generation = ?",
    +                self.engine.catalog_keyspace()
    +            );
    +            // Not applied means another holder took the row over; it is
    +            // theirs to release.
    +            conditional_write(
    +                &self.engine.session,
    +                &query,
    +                cdrs_tokio::query_values!(
    +                    MAINTENANCE_CLAIM_KEY,
    +                    generation_column(self.generation)
    +                ),
    +            )
    +            .await?;
    +            Ok(())
             })
         }
     }
    diff --git a/crates/storage-cassandra/src/cassandra_util.rs b/crates/storage-cassandra/src/cassandra_util.rs
    index a4ebf5c0c..630f95bb7 100644
    --- a/crates/storage-cassandra/src/cassandra_util.rs
    +++ b/crates/storage-cassandra/src/cassandra_util.rs
    @@ -147,6 +147,39 @@ pub async fn query_rows_quorum(
         Ok(body.into_rows().unwrap_or_default())
     }
     
    +/// Execute a query at `LOCAL_SERIAL` and return all rows: the read sees
    +/// every Paxos round that committed, so a row an LWT wrote (a claim taken,
    +/// a heartbeat refreshed) is read as written. For the reads whose answer
    +/// decides whether another process's claim is live.
    +pub async fn query_rows_serial(
    +    session: &Arc,
    +    query: &str,
    +    values: QueryValues,
    +    context: &str,
    +) -> Result, E> {
    +    use cdrs_tokio::consistency::Consistency;
    +    use cdrs_tokio::statement::StatementParamsBuilder;
    +
    +    let params = StatementParamsBuilder::new()
    +        .with_consistency(Consistency::LocalSerial)
    +        .with_values(values)
    +        .build();
    +    let result = session
    +        .query_with_params(query, params)
    +        .await
    +        .map_err(|e| {
    +            tracing::error!("{context} serial query failed: {e}");
    +            E::db_error(format!("{context}: {e}"))
    +        })?;
    +
    +    let body = result.response_body().map_err(|e| {
    +        tracing::error!("{context} response_body failed: {e}");
    +        E::db_error(format!("{context} response_body: {e}"))
    +    })?;
    +
    +    Ok(body.into_rows().unwrap_or_default())
    +}
    +
     /// Execute a query and return the first row, if any.
     ///
     /// Similar to `sqlx::query().fetch_optional()`.
    diff --git a/crates/storage-cassandra/src/lib.rs b/crates/storage-cassandra/src/lib.rs
    index 33c4c8bdb..54b038946 100644
    --- a/crates/storage-cassandra/src/lib.rs
    +++ b/crates/storage-cassandra/src/lib.rs
    @@ -300,7 +300,7 @@ async fn make_catalog_store_from_connection_string(
     fn server_components_factory(
         config: &dyn extenddb_storage::config::StorageConfig,
         region: &str,
    -    _options: extenddb_storage::server_components::ServerComponentsOptions,
    +    options: extenddb_storage::server_components::ServerComponentsOptions,
     ) -> std::pin::Pin<
         Box> + Send>,
     > {
    @@ -332,14 +332,18 @@ fn server_components_factory(
             let control_plane_notify = engine.control_plane_notify.clone();
             let engine = Arc::new(engine);
     
    -        match engine.process_control_plane_transitions().await {
    -            Ok(ref t) if t.is_empty() => {}
    -            Ok(transitions) => {
    -                for (name, transition) in &transitions {
    -                    tracing::info!("Recovered table '{name}': {transition}");
    +        // The startup recovery is a server's; a command reading beside one
    +        // leaves it to the server.
    +        if !options.beside_server {
    +            match engine.process_control_plane_transitions().await {
    +                Ok(ref t) if t.is_empty() => {}
    +                Ok(transitions) => {
    +                    for (name, transition) in &transitions {
    +                        tracing::info!("Recovered table '{name}': {transition}");
    +                    }
                     }
    +                Err(e) => tracing::error!("Failed to recover control plane transitions: {e}"),
                 }
    -            Err(e) => tracing::error!("Failed to recover control plane transitions: {e}"),
             }
     
             // Load encryption key from catalog.
    diff --git a/crates/storage-cassandra/tests/direct/backup_engine.rs b/crates/storage-cassandra/tests/direct/backup_engine.rs
    index b7f2933ef..3daa5be2c 100644
    --- a/crates/storage-cassandra/tests/direct/backup_engine.rs
    +++ b/crates/storage-cassandra/tests/direct/backup_engine.rs
    @@ -2233,7 +2233,6 @@ async fn test_claim_backup_excludes_a_second_owner_and_fences_its_writes() {
             uuid::Uuid::parse_str(&engine.catalog_id().await.expect("id")).is_ok(),
             "the catalog has an id from the migration"
         );
    -    assert!(!engine.check_maintenance_claim().await.expect("check"));
         let store = unique_store();
         let row = index_row(&account_id, "Music", "nightly", &store);
         let claim = engine.claim_backup(&row).await.expect("claim");
    @@ -4291,3 +4290,255 @@ async fn test_a_migration_that_fails_in_its_step_leaves_the_version_behind_and_r
             "the step wrote the catalog id"
         );
     }
    +
    +/// `read_store_ids`: the distinct ids of the rows for a root, of every
    +/// status, scoped to the kind and the root, read from the status partitions.
    +#[tokio::test]
    +async fn test_read_store_ids_lists_the_distinct_ids_of_a_root() {
    +    if crate::helpers::skip_without_cassandra() {
    +        return;
    +    }
    +    let engine = setup_engine().await;
    +    let account = TestAccount::new(&engine, "extenddb_ttl_test").await;
    +    let account_id = account.account_id.clone();
    +    let store = unique_store();
    +    let root = store.store_root.clone();
    +    let ids = |kind: StoreKind| {
    +        let engine = &engine;
    +        let root = root.clone();
    +        async move { engine.read_store_ids(kind, &root).await.expect("ids") }
    +    };
    +    assert!(ids(StoreKind::Filesystem).await.is_empty());
    +    let first = index_row(&account_id, "Music", "a", &store);
    +    let _c1 = engine.claim_backup(&first).await.expect("claim");
    +    let second = index_row(&account_id, "Music", "b", &store);
    +    let c2 = engine.claim_backup(&second).await.expect("claim");
    +    assert_eq!(
    +        c2.set_status(
    +            JobStatus::Creating,
    +            JobStatus::Failed,
    +            &StatusFields::default()
    +        )
    +        .await
    +        .expect("fail"),
    +        WriteOutcome::Matched
    +    );
    +    assert_eq!(
    +        ids(StoreKind::Filesystem).await,
    +        ["store-1"],
    +        "two rows of one store, of two statuses, are one id"
    +    );
    +    let mut other_store = store.clone();
    +    other_store.store_id = "store-2".to_owned();
    +    let other = index_row(&account_id, "Songs", "c", &other_store);
    +    let _c3 = engine.claim_backup(&other).await.expect("claim");
    +    assert_eq!(ids(StoreKind::Filesystem).await, ["store-1", "store-2"]);
    +    assert!(
    +        ids(StoreKind::S3).await.is_empty(),
    +        "another kind at the same root is another store"
    +    );
    +    assert!(
    +        engine
    +            .read_store_ids(StoreKind::Filesystem, "/elsewhere")
    +            .await
    +            .expect("ids")
    +            .is_empty()
    +    );
    +}
    +
    +/// `remove_dead_creating_backup_row`: a CREATING row whose heartbeat is older
    +/// than the window is removed outright, its status-index entry with it; one
    +/// with a live owner, one that is not CREATING, and one that is gone are left,
    +/// answering `false`.
    +#[tokio::test]
    +async fn test_a_dead_creating_row_is_removed_and_a_live_or_moved_one_is_kept() {
    +    if crate::helpers::skip_without_cassandra() {
    +        return;
    +    }
    +    let engine = setup_engine().await;
    +    let account = TestAccount::new(&engine, "extenddb_ttl_test").await;
    +    let account_id = account.account_id.clone();
    +    let store = unique_store();
    +    let stale = Duration::from_secs(300);
    +    let ten_minutes_ago = i64::try_from(
    +        std::time::SystemTime::now()
    +            .duration_since(std::time::UNIX_EPOCH)
    +            .unwrap()
    +            .as_millis(),
    +    )
    +    .unwrap()
    +        - 600_000;
    +    let age = |backup_arn: String, generation: u64| {
    +        let engine = &engine;
    +        let account_id = account_id.clone();
    +        async move {
    +            apply(
    +                engine,
    +                &format!(
    +                    "UPDATE {}.backups_by_arn SET heartbeat_at = ? \
    +                     WHERE account_id = ? AND backup_arn = ? IF owner_generation = ?",
    +                    engine.catalog_keyspace()
    +                ),
    +                cdrs_tokio::query_values!(
    +                    ten_minutes_ago,
    +                    account_id.as_str(),
    +                    backup_arn.as_str(),
    +                    i64::try_from(generation).unwrap()
    +                ),
    +            )
    +            .await;
    +        }
    +    };
    +    let creating = index_row(&account_id, "Music", "dead-a", &store);
    +    let key = creating.key();
    +    let claim = engine.claim_backup(&creating).await.expect("claim");
    +    assert!(
    +        !engine
    +            .remove_dead_creating_backup_row(&key, stale)
    +            .await
    +            .unwrap(),
    +        "a live owner keeps its row"
    +    );
    +    age(creating.backup_arn.clone(), claim.generation()).await;
    +    assert!(
    +        engine
    +            .remove_dead_creating_backup_row(&key, stale)
    +            .await
    +            .unwrap()
    +    );
    +    assert!(
    +        engine
    +            .read_backup_row(&arn_of(&creating))
    +            .await
    +            .unwrap()
    +            .is_none()
    +    );
    +    let creating_rows: Vec = engine
    +        .list_backup_rows(Some(BackupStatus::Creating), &store)
    +        .try_collect()
    +        .await
    +        .unwrap();
    +    assert!(
    +        creating_rows.is_empty(),
    +        "the status index entry went with the row: {creating_rows:?}"
    +    );
    +    assert!(
    +        !engine
    +            .remove_dead_creating_backup_row(&key, stale)
    +            .await
    +            .unwrap(),
    +        "gone"
    +    );
    +    drop(claim);
    +
    +    let available = index_row(&account_id, "Music", "dead-b", &store);
    +    let claim = engine.claim_backup(&available).await.expect("claim");
    +    assert_eq!(
    +        claim
    +            .set_status(
    +                JobStatus::Creating,
    +                JobStatus::Available,
    +                &StatusFields::default()
    +            )
    +            .await
    +            .expect("available"),
    +        WriteOutcome::Matched
    +    );
    +    age(available.backup_arn.clone(), claim.generation()).await;
    +    assert!(
    +        !engine
    +            .remove_dead_creating_backup_row(&available.key(), stale)
    +            .await
    +            .unwrap(),
    +        "only a CREATING row"
    +    );
    +    assert_eq!(
    +        engine
    +            .read_backup_row(&arn_of(&available))
    +            .await
    +            .unwrap()
    +            .unwrap()
    +            .status,
    +        BackupStatus::Available
    +    );
    +    drop(claim);
    +}
    +
    +/// The maintenance claim: taken once, refused to a second holder naming the
    +/// first, heartbeated on the coordinator's clock, released; a holder whose
    +/// heartbeat is older than the stale window is dead, so its claim is not
    +/// reported held and is taken over with a lightweight transaction on its
    +/// generation, and its handle finds the claim lost. The catalog is shared by
    +/// this binary's tests, so the test releases what it takes.
    +#[tokio::test]
    +async fn test_the_maintenance_claim_excludes_a_second_holder_and_expires_when_stale() {
    +    if crate::helpers::skip_without_cassandra() {
    +        return;
    +    }
    +    let engine = setup_engine().await;
    +    let stale = Duration::from_secs(300);
    +    let check = || engine.check_maintenance_claim(stale);
    +    // Another test of this binary never holds the claim; a stale row an
    +    // interrupted run left is taken over below like any dead holder's.
    +    let first = engine
    +        .maintenance_claim("purge on host-a", stale)
    +        .await
    +        .expect("claim");
    +    assert_ne!(first.generation(), UNOWNED_GENERATION);
    +    assert!(check().await.expect("check"), "a live claim is held");
    +    match engine.maintenance_claim("purge on host-b", stale).await {
    +        Err(StorageError::AlreadyExists(m)) => assert!(m.contains("purge on host-a"), "{m}"),
    +        Err(other) => panic!("a second holder is refused with AlreadyExists: {other:?}"),
    +        Ok(_) => panic!("a second holder is refused"),
    +    }
    +    assert_eq!(
    +        first.heartbeat().await.expect("heartbeat"),
    +        Heartbeat::Confirmed
    +    );
    +    first.release().await.expect("release");
    +    assert!(!check().await.expect("check"), "released");
    +    first.release().await.expect("a second release is a no-op");
    +
    +    // A dead holder: its heartbeat aged past the window.
    +    let dead = engine
    +        .maintenance_claim("purge on host-c", stale)
    +        .await
    +        .expect("claim");
    +    let aged = format!(
    +        "UPDATE {}.backup_maintenance SET heartbeat_at = ? WHERE claim = 'claim' \
    +         IF owner_generation = ?",
    +        engine.catalog_keyspace()
    +    );
    +    let ten_minutes_ago = i64::try_from(
    +        std::time::SystemTime::now()
    +            .duration_since(std::time::UNIX_EPOCH)
    +            .unwrap()
    +            .as_millis(),
    +    )
    +    .unwrap()
    +        - 600_000;
    +    apply(
    +        &engine,
    +        &aged,
    +        cdrs_tokio::query_values!(ten_minutes_ago, i64::try_from(dead.generation()).unwrap()),
    +    )
    +    .await;
    +    assert!(!check().await.expect("check"), "a stale claim is not held");
    +    let taker = engine
    +        .maintenance_claim("purge on host-d", stale)
    +        .await
    +        .expect("a dead holder's claim is taken over");
    +    assert_ne!(taker.generation(), dead.generation());
    +    assert!(check().await.expect("check"));
    +    assert_eq!(
    +        dead.heartbeat().await.expect("heartbeat"),
    +        Heartbeat::Lost,
    +        "the dead holder's handle finds the claim lost"
    +    );
    +    dead.release()
    +        .await
    +        .expect("a release by a lost holder changes nothing");
    +    assert!(check().await.expect("check"), "the taker still holds it");
    +    taker.release().await.expect("release");
    +    assert!(!check().await.expect("check"));
    +}
    diff --git a/crates/storage-mongodb/src/backup_catalog.rs b/crates/storage-mongodb/src/backup_catalog.rs
    index 6633f8486..d65b5dfef 100644
    --- a/crates/storage-mongodb/src/backup_catalog.rs
    +++ b/crates/storage-mongodb/src/backup_catalog.rs
    @@ -24,14 +24,22 @@
     //! (`WriteOutcome::Unknown`: the statement may have reached the server);
     //! a server error (a command or write error) is `Err`, as a statement error
     //! is on PostgreSQL.
    +//!
    +//! The maintenance claim is one document (`_id: "claim"`) in
    +//! `backup_maintenance`: a record whose heartbeat is older than the stale
    +//! window is taken over by a conditional update, a missing one is inserted by
    +//! an upsert whose filter a live record never matches (its insert then trips
    +//! the `_id` index and the claim is refused), and heartbeats and the release
    +//! are conditioned on the generation. `$expr` is not allowed in an upsert's
    +//! filter, which is why the takeover and the insert are two writes.
     
     use std::time::Duration;
     
     use extenddb_core::backup_format::BackupArn;
     use extenddb_storage::backup_catalog::{
         BackupCatalog, BackupKey, BackupRow, BackupStatus, DeleteStart, Heartbeat, JobClaim, JobRow,
    -    JobStatus, RestoreJobRow, RestoreKind, RestoreTargetKey, StatusFields, StoreIdentity,
    -    StoreKind, TimeBounds, WriteOutcome, new_generation,
    +    JobStatus, MaintenanceClaim, RestoreJobRow, RestoreKind, RestoreTargetKey, StatusFields,
    +    StoreIdentity, StoreKind, TimeBounds, WriteOutcome, new_generation,
     };
     use extenddb_storage::error::StorageError;
     use futures::TryStreamExt;
    @@ -648,6 +656,24 @@ impl BackupCatalog for MongoEngine {
             })
         }
     
    +    fn remove_dead_creating_backup_row<'a>(
    +        &'a self,
    +        key: &'a BackupKey,
    +        stale_after: Duration,
    +    ) -> BoxFuture<'a, Result> {
    +        Box::pin(async move {
    +            let mut filter = key_filter(key);
    +            filter.insert("status", BackupStatus::Creating.as_str());
    +            filter.insert("$expr", age_is_stale(&["heartbeat_at"], stale_after));
    +            let result = self
    +                .backup_index()
    +                .delete_one(filter)
    +                .await
    +                .map_err(internal)?;
    +            Ok(result.deleted_count == 1)
    +        })
    +    }
    +
         fn backup_in_use<'a>(
             &'a self,
             key: &'a BackupKey,
    @@ -777,12 +803,116 @@ impl BackupCatalog for MongoEngine {
             })
         }
     
    -    fn check_maintenance_claim(&self) -> BoxFuture<'_, Result> {
    +    fn read_store_ids<'a>(
    +        &'a self,
    +        store_kind: StoreKind,
    +        store_root: &'a str,
    +    ) -> BoxFuture<'a, Result, StorageError>> {
    +        Box::pin(async move {
    +            let ids = self
    +                .backup_index()
    +                .distinct(
    +                    "store_id",
    +                    doc! { "store_kind": store_kind.as_str(), "store_root": store_root },
    +                )
    +                .await
    +                .map_err(internal)?;
    +            let mut ids: Vec = ids
    +                .into_iter()
    +                .filter_map(|v| match v {
    +                    Bson::String(s) => Some(s),
    +                    _ => None,
    +                })
    +                .collect();
    +            ids.sort();
    +            Ok(ids)
    +        })
    +    }
    +
    +    fn maintenance_claim<'a>(
    +        &'a self,
    +        holder: &'a str,
    +        stale_after: Duration,
    +    ) -> BoxFuture<'a, Result, StorageError>> {
    +        Box::pin(async move {
    +            // The record is the claim, as a backup row is: one document under
    +            // a fixed `_id`. Two writes: the first takes over a record whose
    +            // heartbeat is stale (a dead holder's), the second inserts the
    +            // record when there is none. A live record matches neither
    +            // filter, so the second write's upsert trips the `_id` index and
    +            // the claim is refused; two commands racing for a stale record
    +            // both run the first write, and the one whose filter no longer
    +            // matches falls to the second and is refused the same way.
    +            // (`$expr` is not allowed in an upsert's filter, which is why
    +            // the two are not one write.)
    +            let generation = new_generation();
    +            let collection = self.maintenance();
    +            let set = doc! { "$set": {
    +                "holder": { "$literal": holder },
    +                "owner_generation": { "$literal": generation_field(generation) },
    +                "claimed_at": "$$NOW",
    +                "heartbeat_at": "$$NOW",
    +            } };
    +            let refused = |current: Option| {
    +                StorageError::AlreadyExists(format!(
    +                    "the backup maintenance claim is held by {}",
    +                    current.as_deref().unwrap_or("another maintenance command")
    +                ))
    +            };
    +            let taken_over = collection
    +                .update_one(
    +                    doc! {
    +                        "_id": MAINTENANCE_CLAIM_ID,
    +                        "$expr": age_is_stale(&["heartbeat_at"], stale_after),
    +                    },
    +                    vec![set.clone()],
    +                )
    +                .await
    +                .map_err(internal)?
    +                .matched_count
    +                == 1;
    +            if !taken_over {
    +                match collection
    +                    .update_one(
    +                        doc! {
    +                            "_id": MAINTENANCE_CLAIM_ID,
    +                            "owner_generation": { "$exists": false },
    +                        },
    +                        vec![set],
    +                    )
    +                    .upsert(true)
    +                    .await
    +                {
    +                    Ok(result) if result.upserted_id.is_some() => {}
    +                    Ok(_) => return Err(refused(None)),
    +                    Err(e) if is_duplicate_key(&e) => {
    +                        let current = collection
    +                            .find_one(doc! { "_id": MAINTENANCE_CLAIM_ID })
    +                            .await
    +                            .map_err(internal)?
    +                            .and_then(|d| d.get_str("holder").ok().map(str::to_owned));
    +                        return Err(refused(current));
    +                    }
    +                    Err(e) => return Err(internal(e)),
    +                }
    +            }
    +            Ok(Box::new(MongoMaintenanceClaim {
    +                collection,
    +                generation,
    +            }) as Box)
    +        })
    +    }
    +
    +    fn check_maintenance_claim(
    +        &self,
    +        stale_after: Duration,
    +    ) -> BoxFuture<'_, Result> {
             Box::pin(async move {
                 let held = self
    -                .catalog_db
    -                .collection::(BACKUP_MAINTENANCE)
    -                .find_one(doc! {})
    +                .maintenance()
    +                .find_one(doc! {
    +                    "$nor": [ { "$expr": age_is_stale(&["heartbeat_at"], stale_after) } ],
    +                })
                     .await
                     .map_err(internal)?;
                 Ok(held.is_some())
    @@ -790,6 +920,64 @@ impl BackupCatalog for MongoEngine {
         }
     }
     
    +/// The `_id` of the one maintenance claim document.
    +const MAINTENANCE_CLAIM_ID: &str = "claim";
    +
    +impl MongoEngine {
    +    fn maintenance(&self) -> mongodb::Collection {
    +        self.catalog_db.collection::(BACKUP_MAINTENANCE)
    +    }
    +}
    +
    +/// The holder's handle on the maintenance claim document.
    +struct MongoMaintenanceClaim {
    +    collection: mongodb::Collection,
    +    generation: u64,
    +}
    +
    +impl MongoMaintenanceClaim {
    +    fn owned_filter(&self) -> Document {
    +        doc! {
    +            "_id": MAINTENANCE_CLAIM_ID,
    +            "owner_generation": generation_field(self.generation),
    +        }
    +    }
    +}
    +
    +impl MaintenanceClaim for MongoMaintenanceClaim {
    +    fn generation(&self) -> u64 {
    +        self.generation
    +    }
    +
    +    fn heartbeat(&self) -> BoxFuture<'_, Result> {
    +        Box::pin(async move {
    +            let result = self
    +                .collection
    +                .update_one(
    +                    self.owned_filter(),
    +                    vec![doc! { "$set": { "heartbeat_at": "$$NOW" } }],
    +                )
    +                .await
    +                .map_err(|e| StorageError::Internal(format!("maintenance heartbeat: {e}")))?;
    +            Ok(if result.matched_count == 1 {
    +                Heartbeat::Confirmed
    +            } else {
    +                Heartbeat::Lost
    +            })
    +        })
    +    }
    +
    +    fn release(&self) -> BoxFuture<'_, Result<(), StorageError>> {
    +        Box::pin(async move {
    +            self.collection
    +                .delete_one(self.owned_filter())
    +                .await
    +                .map_err(internal)?;
    +            Ok(())
    +        })
    +    }
    +}
    +
     #[cfg(test)]
     mod tests {
         use super::*;
    diff --git a/crates/storage-mongodb/src/backup_live_tests.rs b/crates/storage-mongodb/src/backup_live_tests.rs
    index b34afe988..2bf7fe17f 100644
    --- a/crates/storage-mongodb/src/backup_live_tests.rs
    +++ b/crates/storage-mongodb/src/backup_live_tests.rs
    @@ -520,7 +520,12 @@ async fn live_claim_backup_excludes_a_second_owner_and_fences_its_writes() {
             uuid::Uuid::parse_str(&fx.engine.catalog_id().await.expect("id")).is_ok(),
             "the catalog has an id from the migration step"
         );
    -    assert!(!fx.engine.check_maintenance_claim().await.expect("check"));
    +    assert!(
    +        !fx.engine
    +            .check_maintenance_claim(std::time::Duration::from_secs(300))
    +            .await
    +            .expect("check")
    +    );
         let row = row(ACCOUNT, "Music", "nightly");
         let claim = fx.engine.claim_backup(&row).await.expect("claim");
         assert_ne!(claim.generation(), UNOWNED_GENERATION);
    @@ -664,14 +669,236 @@ async fn live_claim_backup_excludes_a_second_owner_and_fences_its_writes() {
             "{third:?}"
         );
     
    -    // A maintenance record, once written, is seen by every job.
    -    fx.engine
    +    fx.finish().await;
    +}
    +
    +/// `read_store_ids`: the distinct ids of the rows for a root, of every
    +/// status, scoped to the kind and the root.
    +#[tokio::test(flavor = "multi_thread")]
    +#[ignore = "needs a MongoDB replica set in EXTENDDB_TEST_MONGODB_URI"]
    +async fn live_read_store_ids_lists_the_distinct_ids_of_a_root() {
    +    let fx = Fixture::new().await;
    +    let root = store().store_root.clone();
    +    let ids = |kind: StoreKind, root: String| {
    +        let engine = &fx.engine;
    +        async move { engine.read_store_ids(kind, &root).await.expect("ids") }
    +    };
    +    assert!(ids(StoreKind::Filesystem, root.clone()).await.is_empty());
    +    let first = row(ACCOUNT, "Music", "a");
    +    let _c1 = fx.engine.claim_backup(&first).await.expect("claim");
    +    let second = row(ACCOUNT, "Music", "b");
    +    let c2 = fx.engine.claim_backup(&second).await.expect("claim");
    +    c2.set_status(
    +        JobStatus::Creating,
    +        JobStatus::Failed,
    +        &StatusFields::default(),
    +    )
    +    .await
    +    .expect("fail");
    +    assert_eq!(
    +        ids(StoreKind::Filesystem, root.clone()).await,
    +        ["store-1"],
    +        "two rows of one store, of two statuses, are one id"
    +    );
    +    let mut other = row(ACCOUNT, "Songs", "c");
    +    other.store.store_id = "store-2".to_owned();
    +    let _c3 = fx.engine.claim_backup(&other).await.expect("claim");
    +    assert_eq!(
    +        ids(StoreKind::Filesystem, root.clone()).await,
    +        ["store-1", "store-2"]
    +    );
    +    assert!(
    +        ids(StoreKind::S3, root.clone()).await.is_empty(),
    +        "another kind at the same root is another store"
    +    );
    +    assert!(
    +        ids(StoreKind::Filesystem, "/elsewhere".to_owned())
    +            .await
    +            .is_empty()
    +    );
    +    fx.finish().await;
    +}
    +
    +/// `remove_dead_creating_backup_row`: a CREATING row whose heartbeat is older
    +/// than the window is removed outright; one with a live owner, one that is
    +/// not CREATING, and one that is gone are left, answering `false`.
    +#[tokio::test(flavor = "multi_thread")]
    +#[ignore = "needs a MongoDB replica set in EXTENDDB_TEST_MONGODB_URI"]
    +async fn live_a_dead_creating_row_is_removed_and_a_live_or_moved_one_is_kept() {
    +    let fx = Fixture::new().await;
    +    let stale = std::time::Duration::from_secs(300);
    +    let index = fx.engine.backup_index();
    +    let age = |backup_id: String| {
    +        let index = index.clone();
    +        async move {
    +            index
    +                .update_one(
    +                    doc! { "backup_id": backup_id },
    +                    vec![doc! { "$set": { "heartbeat_at": { "$subtract": [ "$$NOW", 600_000_i64 ] } } }],
    +                )
    +                .await
    +                .expect("age");
    +        }
    +    };
    +    let creating = row(ACCOUNT, "Music", "a");
    +    let key = creating.key();
    +    let claim = fx.engine.claim_backup(&creating).await.expect("claim");
    +    assert!(
    +        !fx.engine
    +            .remove_dead_creating_backup_row(&key, stale)
    +            .await
    +            .unwrap(),
    +        "a live owner keeps its row"
    +    );
    +    age(creating.backup_id.clone()).await;
    +    assert!(
    +        fx.engine
    +            .remove_dead_creating_backup_row(&key, stale)
    +            .await
    +            .unwrap()
    +    );
    +    assert!(
    +        fx.engine
    +            .read_backup_row(&arn_of(&creating))
    +            .await
    +            .unwrap()
    +            .is_none()
    +    );
    +    assert!(
    +        !fx.engine
    +            .remove_dead_creating_backup_row(&key, stale)
    +            .await
    +            .unwrap(),
    +        "gone"
    +    );
    +    drop(claim);
    +
    +    let available = row(ACCOUNT, "Music", "b");
    +    let claim = fx.engine.claim_backup(&available).await.expect("claim");
    +    claim
    +        .set_status(
    +            JobStatus::Creating,
    +            JobStatus::Available,
    +            &StatusFields::default(),
    +        )
    +        .await
    +        .expect("available");
    +    age(available.backup_id.clone()).await;
    +    assert!(
    +        !fx.engine
    +            .remove_dead_creating_backup_row(&available.key(), stale)
    +            .await
    +            .unwrap(),
    +        "only a CREATING row"
    +    );
    +    assert_eq!(
    +        fx.engine
    +            .read_backup_row(&arn_of(&available))
    +            .await
    +            .unwrap()
    +            .unwrap()
    +            .status,
    +        BackupStatus::Available
    +    );
    +    drop(claim);
    +    fx.finish().await;
    +}
    +
    +/// The maintenance claim: taken once, refused to a second holder naming the
    +/// first, heartbeated on the server's clock, released; a holder whose
    +/// heartbeat is older than the stale window is dead, so its claim is not
    +/// reported held and is taken over, and its handle finds the claim lost. A
    +/// record without a heartbeat (no release of this server wrote one) is
    +/// stale.
    +#[tokio::test(flavor = "multi_thread")]
    +#[ignore = "needs a MongoDB replica set in EXTENDDB_TEST_MONGODB_URI"]
    +async fn live_the_maintenance_claim_excludes_a_second_holder_and_expires_when_stale() {
    +    let fx = Fixture::new().await;
    +    let stale = std::time::Duration::from_secs(300);
    +    let check = || fx.engine.check_maintenance_claim(stale);
    +    let maintenance = fx
    +        .engine
             .catalog_db
    -        .collection::(crate::backup_catalog::BACKUP_MAINTENANCE)
    -        .insert_one(doc! { "holder": "backup purge", "owner_generation": 1_i64 })
    +        .collection::(crate::backup_catalog::BACKUP_MAINTENANCE);
    +    assert!(!check().await.expect("check"));
    +    let first = fx
    +        .engine
    +        .maintenance_claim("purge on host-a", stale)
    +        .await
    +        .expect("claim");
    +    assert_ne!(first.generation(), UNOWNED_GENERATION);
    +    assert!(check().await.expect("check"), "a live claim is held");
    +    match fx.engine.maintenance_claim("purge on host-b", stale).await {
    +        Err(StorageError::AlreadyExists(m)) => assert!(m.contains("purge on host-a"), "{m}"),
    +        Err(other) => panic!("a second holder is refused with AlreadyExists: {other:?}"),
    +        Ok(_) => panic!("a second holder is refused"),
    +    }
    +    assert_eq!(
    +        first.heartbeat().await.expect("heartbeat"),
    +        Heartbeat::Confirmed
    +    );
    +    first.release().await.expect("release");
    +    assert!(!check().await.expect("check"), "released");
    +    assert_eq!(
    +        maintenance.count_documents(doc! {}).await.expect("count"),
    +        0
    +    );
    +    first.release().await.expect("a second release is a no-op");
    +
    +    // A dead holder: its heartbeat aged past the window.
    +    let dead = fx
    +        .engine
    +        .maintenance_claim("purge on host-c", stale)
    +        .await
    +        .expect("claim");
    +    maintenance
    +        .update_one(
    +            doc! {},
    +            vec![doc! { "$set": { "heartbeat_at": { "$subtract": [ "$$NOW", 600_000_i64 ] } } }],
    +        )
    +        .await
    +        .expect("age");
    +    assert!(!check().await.expect("check"), "a stale claim is not held");
    +    let taker = fx
    +        .engine
    +        .maintenance_claim("purge on host-d", stale)
    +        .await
    +        .expect("a dead holder's claim is taken over");
    +    assert_ne!(taker.generation(), dead.generation());
    +    assert!(check().await.expect("check"));
    +    assert_eq!(
    +        dead.heartbeat().await.expect("heartbeat"),
    +        Heartbeat::Lost,
    +        "the dead holder's handle finds the claim lost"
    +    );
    +    dead.release()
    +        .await
    +        .expect("a release by a lost holder changes nothing");
    +    assert!(check().await.expect("check"), "the taker still holds it");
    +    let holder = maintenance
    +        .find_one(doc! {})
    +        .await
    +        .expect("find")
    +        .expect("record");
    +    assert_eq!(holder.get_str("holder").unwrap(), "purge on host-d");
    +    taker.release().await.expect("release");
    +    assert!(!check().await.expect("check"));
    +
    +    // A record with no heartbeat at all is stale: taken over, never held.
    +    maintenance
    +        .insert_one(
    +            doc! { "_id": "claim", "holder": "an older release", "owner_generation": 1_i64 },
    +        )
    +        .await
    +        .expect("record");
    +    assert!(!check().await.expect("check"));
    +    let over = fx
    +        .engine
    +        .maintenance_claim("purge on host-e", stale)
             .await
    -        .expect("maintenance record");
    -    assert!(fx.engine.check_maintenance_claim().await.expect("check"));
    +        .expect("taken over");
    +    assert!(check().await.expect("check"));
    +    over.release().await.expect("release");
         fx.finish().await;
     }
     
    diff --git a/crates/storage-postgres/Cargo.toml b/crates/storage-postgres/Cargo.toml
    index 6bb3fafa8..531a0f4c1 100755
    --- a/crates/storage-postgres/Cargo.toml
    +++ b/crates/storage-postgres/Cargo.toml
    @@ -26,6 +26,11 @@ extenddb-storage = { workspace = true, features = ["test-util"] }
     # The crate's own tests need `test-hooks` and `test-util`; a dev-dependency on
     # itself turns both features on for test builds only.
     extenddb-storage-postgres = { path = ".", features = ["test-hooks", "test-util"] }
    +# The store identity tests run the engine's checks over two engines on one
    +# scratch catalog and two filesystem stores.
    +extenddb-engine = { workspace = true }
    +extenddb-backup-store = { workspace = true }
    +tempfile = "3"
     
     [dependencies]
     anyhow = { workspace = true }
    diff --git a/crates/storage-postgres/src/backup_catalog.rs b/crates/storage-postgres/src/backup_catalog.rs
    index 4477c823b..468f63eb7 100644
    --- a/crates/storage-postgres/src/backup_catalog.rs
    +++ b/crates/storage-postgres/src/backup_catalog.rs
    @@ -17,6 +17,19 @@
     //! registration is `FOR UPDATE` on the `backup_index` row. Staleness is
     //! measured on the catalog's clock.
     //!
    +//! The maintenance claim is the same mechanism on the `backup_maintenance`
    +//! row: an advisory lock (`backup_maintenance` in the same key space) on a
    +//! dedicated connection is the proof of a live holder, so a second command
    +//! finds it taken and a holder whose session ended has released it; the row
    +//! carries the holder, the generation and the heartbeat.
    +//! `check_maintenance_claim` reports the claim held while a session holds
    +//! the lock (a transaction-scoped shared try-lock in one autocommit
    +//! statement fails while the exclusive lock is held, and holds nothing past
    +//! its statement, so a request dropped mid-check leaves no lock on the
    +//! pool), whatever the row's heartbeat says, and otherwise while the row's
    +//! heartbeat is younger than the stale window, so a dead holder's claim
    +//! expires for the jobs that read it without anyone taking it over. `release` deletes the row and ends the session.
    +//!
     //! A conditional write whose error is of the connection class (an I/O or TLS
     //! error, a pool or worker failure, SQLSTATE class 08, or 57P01 to 57P03)
     //! has an unknown outcome (`WriteOutcome::Unknown`: the statement may have
    @@ -31,8 +44,8 @@ use std::time::Duration;
     use extenddb_core::backup_format::BackupArn;
     use extenddb_storage::backup_catalog::{
         BackupCatalog, BackupKey, BackupRow, BackupStatus, DeleteStart, Heartbeat, JobClaim, JobRow,
    -    JobStatus, RestoreJobRow, RestoreKind, RestoreTargetKey, StatusFields, StoreIdentity,
    -    StoreKind, TimeBounds, WriteOutcome,
    +    JobStatus, MaintenanceClaim, RestoreJobRow, RestoreKind, RestoreTargetKey, StatusFields,
    +    StoreIdentity, StoreKind, TimeBounds, WriteOutcome,
     };
     use extenddb_storage::error::StorageError;
     use futures::future::BoxFuture;
    @@ -499,6 +512,28 @@ impl BackupCatalog for PostgresEngine {
             })
         }
     
    +    fn remove_dead_creating_backup_row<'a>(
    +        &'a self,
    +        key: &'a BackupKey,
    +        stale_after: Duration,
    +    ) -> BoxFuture<'a, Result> {
    +        Box::pin(async move {
    +            let affected = sqlx::query(
    +                "DELETE FROM backup_index WHERE account_id = $1 AND backup_id = $2 \
    +                 AND status = 'CREATING' \
    +                 AND heartbeat_at < now() - make_interval(secs => $3)",
    +            )
    +            .bind(&key.account_id)
    +            .bind(&key.backup_id)
    +            .bind(stale_after.as_secs_f64())
    +            .execute(&self.pool)
    +            .await
    +            .map_err(db_err)?
    +            .rows_affected();
    +            Ok(affected == 1)
    +        })
    +    }
    +
         fn backup_in_use<'a>(
             &'a self,
             key: &'a BackupKey,
    @@ -631,12 +666,160 @@ impl BackupCatalog for PostgresEngine {
             })
         }
     
    -    fn check_maintenance_claim(&self) -> BoxFuture<'_, Result> {
    +    fn read_store_ids<'a>(
    +        &'a self,
    +        store_kind: StoreKind,
    +        store_root: &'a str,
    +    ) -> BoxFuture<'a, Result, StorageError>> {
    +        Box::pin(async move {
    +            sqlx::query_scalar(
    +                "SELECT DISTINCT store_id FROM backup_index \
    +                 WHERE store_kind = $1 AND store_root = $2 ORDER BY store_id",
    +            )
    +            .bind(store_kind.as_str())
    +            .bind(store_root)
    +            .fetch_all(&self.pool)
    +            .await
    +            .map_err(db_err)
    +        })
    +    }
    +
    +    fn maintenance_claim<'a>(
    +        &'a self,
    +        holder: &'a str,
    +        stale_after: Duration,
    +    ) -> BoxFuture<'a, Result, StorageError>> {
             Box::pin(async move {
    -            sqlx::query_scalar("SELECT EXISTS(SELECT 1 FROM backup_maintenance)")
    -                .fetch_one(&self.pool)
    +            // The advisory lock is the live-holder proof, as a job's is: a
    +            // second command on any instance finds it taken while the first
    +            // runs, and a holder whose session ended has released it. The
    +            // row carries the generation the heartbeats and the release are
    +            // conditioned on, and the heartbeat age jobs read.
    +            let Some(mut session) = try_claim(&self.pool, MAINTENANCE_LOCK_KEY, None).await? else {
    +                let current: Option =
    +                    sqlx::query_scalar("SELECT holder FROM backup_maintenance")
    +                        .fetch_optional(&self.pool)
    +                        .await
    +                        .map_err(db_err)?;
    +                return Err(StorageError::AlreadyExists(format!(
    +                    "the backup maintenance claim is held by {}",
    +                    current.as_deref().unwrap_or("another maintenance command")
    +                )));
    +            };
    +            // With the lock held, a row whose heartbeat is still young is a
    +            // holder whose lock session ended within the stale window: dead
    +            // by this backend's rule (4(e)), so the row is taken over
    +            // whatever its age. `stale_after` is the other backends' rule and
    +            // the one `check_maintenance_claim` applies on every backend.
    +            let _ = stale_after;
    +            let generation = session.generation();
    +            sqlx::query(
    +                "INSERT INTO backup_maintenance (singleton, holder, owner_generation) \
    +                 VALUES (TRUE, $1, $2) \
    +                 ON CONFLICT (singleton) DO UPDATE SET holder = EXCLUDED.holder, \
    +                 owner_generation = EXCLUDED.owner_generation, claimed_at = now(), \
    +                 heartbeat_at = now()",
    +            )
    +            .bind(holder)
    +            .bind(generation_column(generation))
    +            .execute(session.connection())
    +            .await
    +            .map_err(db_err)?;
    +            Ok(Box::new(PostgresMaintenanceClaim {
    +                session: Mutex::new(Some(session)),
    +                generation,
    +            }) as Box)
    +        })
    +    }
    +
    +    fn check_maintenance_claim(
    +        &self,
    +        stale_after: Duration,
    +    ) -> BoxFuture<'_, Result> {
    +        Box::pin(async move {
    +            // A live holder's session holds the exclusive lock, so a shared
    +            // try-lock fails; it succeeds when no session does. The lock is
    +            // taken at transaction scope in one autocommit statement, so it
    +            // ends with the statement and nothing is held on the pooled
    +            // connection when the caller's future is dropped between awaits
    +            // (this runs inside every CreateBackup and RestoreTableFromBackup
    +            // request). Then the row a dead session left is held until its
    +            // heartbeat ages past the window.
    +            let unlocked: bool = sqlx::query_scalar(
    +                "SELECT pg_try_advisory_xact_lock_shared(hashtextextended($1, $2))",
    +            )
    +            .bind(MAINTENANCE_LOCK_KEY)
    +            .bind(crate::backup_engine::RESTORE_LOCK_SEED)
    +            .fetch_one(&self.pool)
    +            .await
    +            .map_err(db_err)?;
    +            if !unlocked {
    +                return Ok(true);
    +            }
    +            sqlx::query_scalar(
    +                "SELECT EXISTS(SELECT 1 FROM backup_maintenance \
    +                 WHERE heartbeat_at >= now() - make_interval(secs => $1))",
    +            )
    +            .bind(stale_after.as_secs_f64())
    +            .fetch_one(&self.pool)
    +            .await
    +            .map_err(db_err)
    +        })
    +    }
    +}
    +
    +/// The advisory-lock key of the maintenance claim, in the one lock key space
    +/// (`RESTORE_LOCK_SEED`) the job claims use.
    +const MAINTENANCE_LOCK_KEY: &str = "backup_maintenance";
    +
    +/// The holder's handle on the `backup_maintenance` row: the lock session
    +/// (`None` once released) and the generation its writes are conditioned on.
    +struct PostgresMaintenanceClaim {
    +    session: Mutex>,
    +    generation: u64,
    +}
    +
    +impl MaintenanceClaim for PostgresMaintenanceClaim {
    +    fn generation(&self) -> u64 {
    +        self.generation
    +    }
    +
    +    fn heartbeat(&self) -> BoxFuture<'_, Result> {
    +        Box::pin(async move {
    +            let mut session = self.session.lock().await;
    +            let Some(session) = session.as_mut() else {
    +                return Ok(Heartbeat::Lost);
    +            };
    +            let confirmed = session
    +                .confirm(
    +                    "UPDATE backup_maintenance SET heartbeat_at = now() WHERE owner_generation = $1",
    +                    &[],
    +                )
                     .await
    -                .map_err(db_err)
    +                .map_err(|e| StorageError::Internal(format!("claim session: {e}")))?;
    +            Ok(if confirmed {
    +                Heartbeat::Confirmed
    +            } else {
    +                Heartbeat::Lost
    +            })
    +        })
    +    }
    +
    +    /// Delete the row and end the lock session, which releases the advisory
    +    /// lock; a second release finds no session and does nothing.
    +    fn release(&self) -> BoxFuture<'_, Result<(), StorageError>> {
    +        Box::pin(async move {
    +            let mut slot = self.session.lock().await;
    +            let Some(mut session) = slot.take() else {
    +                return Ok(());
    +            };
    +            sqlx::query("DELETE FROM backup_maintenance WHERE owner_generation = $1")
    +                .bind(generation_column(self.generation))
    +                .execute(session.connection())
    +                .await
    +                .map_err(db_err)?;
    +            session.close().await;
    +            Ok(())
             })
         }
     }
    diff --git a/crates/storage-postgres/src/backup_engine.rs b/crates/storage-postgres/src/backup_engine.rs
    index 0adceaa5a..b501810a1 100755
    --- a/crates/storage-postgres/src/backup_engine.rs
    +++ b/crates/storage-postgres/src/backup_engine.rs
    @@ -112,6 +112,14 @@ impl ClaimSession {
             &mut self.conn
         }
     
    +    /// End the session, releasing its advisory lock; a close that fails is
    +    /// logged (the server reaps the connection either way).
    +    pub(crate) async fn close(self) {
    +        if let Err(e) = ::close(self.conn).await {
    +            tracing::warn!("closing a claim session: {e}");
    +        }
    +    }
    +
         /// Run `statement`, bound with the record's key and this generation, on
         /// the claim connection and report whether it refreshed exactly one row.
         pub(crate) async fn confirm(
    diff --git a/crates/storage-postgres/src/lib.rs b/crates/storage-postgres/src/lib.rs
    index 9f2849477..757eee26a 100755
    --- a/crates/storage-postgres/src/lib.rs
    +++ b/crates/storage-postgres/src/lib.rs
    @@ -909,23 +909,35 @@ fn server_components_factory(
                 })?;
             engine.serving = Some(Arc::new(registration));
     
    -        // Rebuild any vector index a crash left CREATING, before serving: an index
    -        // in that state is not searchable, and nothing else will repair it.
    -        match crate::data::vector_index::reconcile_incomplete_vector_indexes(&engine).await {
    -            Ok(n) if n > 0 => {
    -                tracing::info!("Reconciled {n} incomplete vector index(es) at startup");
    +        // The startup recovery is a server's; a command reading beside one
    +        // leaves it to the server.
    +        if !options.beside_server {
    +            // Rebuild any vector index a crash left CREATING, before serving:
    +            // an index in that state is not searchable, and nothing else will
    +            // repair it.
    +            match crate::data::vector_index::reconcile_incomplete_vector_indexes(&engine).await {
    +                Ok(n) if n > 0 => {
    +                    tracing::info!("Reconciled {n} incomplete vector index(es) at startup");
    +                }
    +                Ok(_) => {}
    +                Err(e) => tracing::error!("Failed to reconcile incomplete vector indexes: {e}"),
                 }
    -            Ok(_) => {}
    -            Err(e) => tracing::error!("Failed to reconcile incomplete vector indexes: {e}"),
    -        }
     
    -        // Recover control plane transitions (errors are logged, not fatal).
    -        // A drop deferred by a session still holding the table wakes the
    -        // poller, which `spawn_workers` starts later.
    -        workers::recover_control_plane_at_startup(&engine).await;
    +            // Recover control plane transitions (errors are logged, not fatal).
    +            // A drop deferred by a session still holding the table wakes the
    +            // poller, which `spawn_workers` starts later.
    +            workers::recover_control_plane_at_startup(&engine).await;
    +        }
     
    -        // Start GSI workers
    -        let engine = engine.start_gsi_workers();
    +        // Start GSI workers: a server's. A command reading beside one
    +        // (`backup list`, the purge's dry run) leaves the queue to the
    +        // server that owns the catalog, so the command applies no pending
    +        // index write and exits with none half-applied.
    +        let engine = if options.beside_server {
    +            engine
    +        } else {
    +            engine.start_gsi_workers()
    +        };
     
             // Get data database name for logging (before wrapping in Arc)
             let data_db_name = engine
    diff --git a/crates/storage-postgres/tests/backup_restore.rs b/crates/storage-postgres/tests/backup_restore.rs
    index f654f702f..af3c8fa46 100644
    --- a/crates/storage-postgres/tests/backup_restore.rs
    +++ b/crates/storage-postgres/tests/backup_restore.rs
    @@ -57,6 +57,8 @@ struct Scratch {
         db: PgPool,
         catalog: PgPool,
         admin: PgPool,
    +    /// The catalog's URL, for a second engine on the same catalog.
    +    catalog_url: String,
         db_names: [String; 2],
         _guard: DbGuard,
         _serial: tokio::sync::MutexGuard<'static, ()>,
    @@ -216,7 +218,7 @@ async fn scratch_with_pool(pool_size: u32) -> Scratch {
         }
         let engine = PostgresEngine::new(
             &PostgresConfig {
    -            connection_string: catalog_url,
    +            connection_string: catalog_url.clone(),
                 pool_size,
                 max_item_size_bytes: 400_000,
             },
    @@ -229,6 +231,7 @@ async fn scratch_with_pool(pool_size: u32) -> Scratch {
             db,
             catalog,
             admin,
    +        catalog_url,
             db_names: [catalog_name, data_name],
             _guard: guard,
             _serial: serial,
    @@ -718,6 +721,14 @@ async fn indexed_source_with(s: &Scratch, name: &str, items: usize) -> String {
             .create_table(ACCOUNT, input)
             .await
             .expect("create the source table");
    +    put_indexed_items(s, name, &desc, items).await;
    +    desc.table_id
    +}
    +
    +/// Write `items` items through PutItem into the table `indexed_source`
    +/// created (`desc` its description): every item has `pk` and `sk`, every
    +/// second one `g` (the GSI's key), every third one `l` (the LSI's).
    +async fn put_indexed_items(s: &Scratch, name: &str, desc: &TableDescription, items: usize) {
         let key_info = TableKeyInfo {
             table_name: name.to_owned(),
             account_id: ACCOUNT.to_owned(),
    @@ -749,7 +760,6 @@ async fn indexed_source_with(s: &Scratch, name: &str, items: usize) -> String {
                 .await
                 .expect("put");
         }
    -    desc.table_id
     }
     
     /// `(index_name, index_id)` of a table's secondary indexes, by name.
    @@ -2705,6 +2715,16 @@ async fn a_definition_read_that_loses_its_connection_is_transient() {
     
     // --- the catalog index -----------------------------------------------------------
     
    +/// A catalog row for a backup of `table` in `account` minted now, with the
    +/// manifest fixture's source table, in the `catalog_index` module's store.
    +fn catalog_index_row(
    +    account: &str,
    +    table: &str,
    +    name: &str,
    +) -> extenddb_storage::backup_catalog::BackupRow {
    +    catalog_index::row(account, table, name)
    +}
    +
     mod catalog_index {
         use extenddb_core::backup_format::BackupArn;
         use extenddb_storage::BackupCatalog;
    @@ -2726,7 +2746,7 @@ mod catalog_index {
     
         /// A row for a backup of `table` in `account` minted now, with the
         /// manifest fixture's source table.
    -    fn row(account: &str, table: &str, name: &str) -> BackupRow {
    +    pub(super) fn row(account: &str, table: &str, name: &str) -> BackupRow {
             let backup_id = extenddb_storage::util::new_backup_id();
             let backup_arn = extenddb_storage::util::backup_arn(REGION, account, table, &backup_id);
             let manifest: serde_json::Value = serde_json::from_str(include_str!(
    @@ -2773,7 +2793,12 @@ mod catalog_index {
                 uuid::Uuid::parse_str(&s.engine.catalog_id().await.expect("id")).is_ok(),
                 "the catalog has an id from the migration"
             );
    -        assert!(!s.engine.check_maintenance_claim().await.expect("check"));
    +        assert!(
    +            !s.engine
    +                .check_maintenance_claim(std::time::Duration::from_secs(300))
    +                .await
    +                .expect("check")
    +        );
             let row = row(ACCOUNT, "Music", "nightly");
             assert_eq!(backup_lock_holder(&s, &row.backup_id).await, None);
             let claim = s.engine.claim_backup(&row).await.expect("claim");
    @@ -3823,4 +3848,820 @@ mod catalog_index {
             assert!(record.1.is_some());
             s.cleanup().await;
         }
    +
    +    /// `read_store_ids`: the distinct ids of the rows for a root, of every
    +    /// status, scoped to the kind and the root.
    +    #[tokio::test]
    +    async fn read_store_ids_lists_the_distinct_ids_of_a_root() {
    +        if base_conn().is_none() {
    +            eprintln!("SKIP read_store_ids_lists_the_distinct_ids_of_a_root: no PostgreSQL");
    +            return;
    +        }
    +        let s = scratch().await;
    +        let root = &store().store_root;
    +        assert!(
    +            s.engine
    +                .read_store_ids(StoreKind::Filesystem, root)
    +                .await
    +                .expect("ids")
    +                .is_empty()
    +        );
    +        let first = row(ACCOUNT, "Music", "a");
    +        let _c1 = s.engine.claim_backup(&first).await.expect("claim");
    +        let second = row(ACCOUNT, "Music", "b");
    +        let c2 = s.engine.claim_backup(&second).await.expect("claim");
    +        c2.set_status(
    +            JobStatus::Creating,
    +            JobStatus::Failed,
    +            &StatusFields::default(),
    +        )
    +        .await
    +        .expect("fail");
    +        assert_eq!(
    +            s.engine
    +                .read_store_ids(StoreKind::Filesystem, root)
    +                .await
    +                .expect("ids"),
    +            ["store-1"],
    +            "two rows of one store, of two statuses, are one id"
    +        );
    +        let mut other = row(ACCOUNT, "Songs", "c");
    +        other.store.store_id = "store-2".to_owned();
    +        let _c3 = s.engine.claim_backup(&other).await.expect("claim");
    +        assert_eq!(
    +            s.engine
    +                .read_store_ids(StoreKind::Filesystem, root)
    +                .await
    +                .expect("ids"),
    +            ["store-1", "store-2"]
    +        );
    +        assert!(
    +            s.engine
    +                .read_store_ids(StoreKind::S3, root)
    +                .await
    +                .expect("ids")
    +                .is_empty(),
    +            "another kind at the same root is another store"
    +        );
    +        assert!(
    +            s.engine
    +                .read_store_ids(StoreKind::Filesystem, "/elsewhere")
    +                .await
    +                .expect("ids")
    +                .is_empty()
    +        );
    +        s.cleanup().await;
    +    }
    +
    +    /// The maintenance claim: taken once, refused to a second holder naming
    +    /// the first, heartbeated, released; a holder whose session ended and
    +    /// whose heartbeat is older than the stale window is dead, so its claim
    +    /// is not reported held and is taken over, and its handle finds the claim
    +    /// lost.
    +    /// `remove_dead_creating_backup_row`: a CREATING row whose heartbeat is
    +    /// older than the window is removed outright; one with a live owner, one
    +    /// that is not CREATING, and one that is gone are left, answering
    +    /// `false`.
    +    #[tokio::test]
    +    async fn a_dead_creating_row_is_removed_and_a_live_or_moved_one_is_kept() {
    +        if base_conn().is_none() {
    +            eprintln!(
    +                "SKIP a_dead_creating_row_is_removed_and_a_live_or_moved_one_is_kept: no PostgreSQL"
    +            );
    +            return;
    +        }
    +        let s = scratch().await;
    +        let stale = std::time::Duration::from_secs(300);
    +        let age = |backup_id: String| {
    +            let catalog = s.catalog.clone();
    +            async move {
    +                sqlx::query(
    +                    "UPDATE backup_index SET heartbeat_at = now() - interval '10 minutes' \
    +                     WHERE backup_id = $1",
    +                )
    +                .bind(backup_id)
    +                .execute(&catalog)
    +                .await
    +                .expect("age");
    +            }
    +        };
    +        let creating = row(ACCOUNT, "Music", "a");
    +        let key = creating.key();
    +        let claim = s.engine.claim_backup(&creating).await.expect("claim");
    +        assert!(
    +            !s.engine
    +                .remove_dead_creating_backup_row(&key, stale)
    +                .await
    +                .unwrap(),
    +            "a live owner keeps its row"
    +        );
    +        age(creating.backup_id.clone()).await;
    +        assert!(
    +            s.engine
    +                .remove_dead_creating_backup_row(&key, stale)
    +                .await
    +                .unwrap()
    +        );
    +        assert!(
    +            s.engine
    +                .read_backup_row(&arn_of(&creating))
    +                .await
    +                .unwrap()
    +                .is_none()
    +        );
    +        assert!(
    +            !s.engine
    +                .remove_dead_creating_backup_row(&key, stale)
    +                .await
    +                .unwrap(),
    +            "gone"
    +        );
    +        drop(claim);
    +
    +        let available = row(ACCOUNT, "Music", "b");
    +        let claim = s.engine.claim_backup(&available).await.expect("claim");
    +        claim
    +            .set_status(
    +                JobStatus::Creating,
    +                JobStatus::Available,
    +                &StatusFields::default(),
    +            )
    +            .await
    +            .expect("available");
    +        age(available.backup_id.clone()).await;
    +        assert!(
    +            !s.engine
    +                .remove_dead_creating_backup_row(&available.key(), stale)
    +                .await
    +                .unwrap(),
    +            "only a CREATING row"
    +        );
    +        assert_eq!(
    +            s.engine
    +                .read_backup_row(&arn_of(&available))
    +                .await
    +                .unwrap()
    +                .unwrap()
    +                .status,
    +            BackupStatus::Available
    +        );
    +        drop(claim);
    +        s.cleanup().await;
    +    }
    +
    +    #[tokio::test]
    +    async fn the_maintenance_claim_excludes_a_second_holder_and_expires_when_stale() {
    +        if base_conn().is_none() {
    +            eprintln!(
    +                "SKIP the_maintenance_claim_excludes_a_second_holder_and_expires_when_stale: \
    +                 no PostgreSQL"
    +            );
    +            return;
    +        }
    +        let s = scratch().await;
    +        let stale = std::time::Duration::from_secs(300);
    +        let check = || s.engine.check_maintenance_claim(stale);
    +        assert!(!check().await.expect("check"));
    +        let first = s
    +            .engine
    +            .maintenance_claim("purge on host-a", stale)
    +            .await
    +            .expect("claim");
    +        assert_ne!(first.generation(), UNOWNED_GENERATION);
    +        assert!(check().await.expect("check"), "a live claim is held");
    +        match s.engine.maintenance_claim("purge on host-b", stale).await {
    +            Err(StorageError::AlreadyExists(m)) => {
    +                assert!(m.contains("purge on host-a"), "{m}");
    +            }
    +            Err(other) => panic!("a second holder is refused with AlreadyExists: {other:?}"),
    +            Ok(_) => panic!("a second holder is refused"),
    +        }
    +        assert_eq!(
    +            first.heartbeat().await.expect("heartbeat"),
    +            Heartbeat::Confirmed
    +        );
    +        first.release().await.expect("release");
    +        assert!(!check().await.expect("check"), "released");
    +        first.release().await.expect("a second release is a no-op");
    +
    +        // A dead holder: its session ended (the handle dropped) and its
    +        // heartbeat aged past the window.
    +        let dead = s
    +            .engine
    +            .maintenance_claim("purge on host-c", stale)
    +            .await
    +            .expect("claim");
    +        let dead_generation = dead.generation();
    +        drop(dead);
    +        // The drop closes the holder's session; the backend reaps it a
    +        // moment later. Wait for the exclusive lock to be gone, so the check
    +        // below is answered by the heartbeat arm alone and not by a lock
    +        // still being released.
    +        let deadline = std::time::Instant::now() + std::time::Duration::from_secs(10);
    +        loop {
    +            let exclusive: i64 = sqlx::query_scalar(
    +                "SELECT count(*) FROM pg_locks WHERE locktype = 'advisory' AND mode = \
    +                 'ExclusiveLock' AND database = (SELECT oid FROM pg_database WHERE datname = \
    +                 current_database())",
    +            )
    +            .fetch_one(&s.catalog)
    +            .await
    +            .expect("pg_locks");
    +            if exclusive == 0 {
    +                break;
    +            }
    +            assert!(
    +                std::time::Instant::now() < deadline,
    +                "the dropped holder's session still holds the lock"
    +            );
    +            tokio::time::sleep(std::time::Duration::from_millis(20)).await;
    +        }
    +        // The heartbeat arm: the lock is free (the session ended with the
    +        // handle) and the row's heartbeat is young, so the jobs still see
    +        // the claim held, as they must for a command that died a moment ago
    +        // or lost its claim while its row is young.
    +        assert!(
    +            check().await.expect("check"),
    +            "a dead session's row is held by its heartbeat until it ages"
    +        );
    +        sqlx::query("UPDATE backup_maintenance SET heartbeat_at = now() - interval '10 minutes'")
    +            .execute(&s.catalog)
    +            .await
    +            .expect("age");
    +        assert!(!check().await.expect("check"), "a stale claim is not held");
    +        let taker = s
    +            .engine
    +            .maintenance_claim("purge on host-d", stale)
    +            .await
    +            .expect("a dead holder's claim is taken over");
    +        assert_ne!(taker.generation(), dead_generation);
    +        assert!(check().await.expect("check"));
    +        let holder: String = sqlx::query_scalar("SELECT holder FROM backup_maintenance")
    +            .fetch_one(&s.catalog)
    +            .await
    +            .expect("holder");
    +        assert_eq!(holder, "purge on host-d");
    +        // A holder whose session is still open is alive whatever its
    +        // heartbeat says: the lock is the proof of life on this backend.
    +        sqlx::query("UPDATE backup_maintenance SET heartbeat_at = now() - interval '10 minutes'")
    +            .execute(&s.catalog)
    +            .await
    +            .expect("age");
    +        assert!(
    +            matches!(
    +                s.engine.maintenance_claim("purge on host-e", stale).await,
    +                Err(StorageError::AlreadyExists(_))
    +            ),
    +            "a live session keeps the claim"
    +        );
    +        assert!(
    +            check().await.expect("check"),
    +            "the jobs' check sees a live session's claim held through the lock, whatever the \
    +             row's heartbeat says"
    +        );
    +        // The check holds no lock past its statement: after it, the only
    +        // advisory lock in the catalog is the holder's exclusive one, so no
    +        // pooled connection carries a shared lock a dropped request left.
    +        let modes: Vec = sqlx::query_scalar(
    +            "SELECT mode FROM pg_locks WHERE locktype = 'advisory' AND database = \
    +             (SELECT oid FROM pg_database WHERE datname = current_database()) ORDER BY mode",
    +        )
    +        .fetch_all(&s.catalog)
    +        .await
    +        .expect("pg_locks");
    +        assert_eq!(
    +            modes,
    +            ["ExclusiveLock"],
    +            "the holder's lock and nothing else"
    +        );
    +        assert_eq!(
    +            taker.heartbeat().await.expect("heartbeat"),
    +            Heartbeat::Confirmed
    +        );
    +        assert!(check().await.expect("check"), "the heartbeat refreshed it");
    +        // The row's generation moved under the live holder (a takeover by
    +        // another writer): the holder's next heartbeat matches no row and
    +        // reports the claim lost, and its release leaves the other's row.
    +        sqlx::query("UPDATE backup_maintenance SET owner_generation = owner_generation + 1")
    +            .execute(&s.catalog)
    +            .await
    +            .expect("move the generation");
    +        assert_eq!(
    +            taker.heartbeat().await.expect("heartbeat"),
    +            Heartbeat::Lost,
    +            "a heartbeat under a moved generation reports the claim lost"
    +        );
    +        taker.release().await.expect("release");
    +        let rows: i64 = sqlx::query_scalar("SELECT COUNT(*) FROM backup_maintenance")
    +            .fetch_one(&s.catalog)
    +            .await
    +            .expect("count");
    +        assert_eq!(
    +            rows, 1,
    +            "a release by a holder that lost the claim leaves the row"
    +        );
    +        assert!(
    +            check().await.expect("check"),
    +            "the lock is free and the row young: held by the heartbeat arm"
    +        );
    +        sqlx::query("DELETE FROM backup_maintenance")
    +            .execute(&s.catalog)
    +            .await
    +            .expect("clear");
    +        assert!(!check().await.expect("check"));
    +        s.cleanup().await;
    +    }
    +
    +    /// Purge's wait (`wait_for_quiet`) over this catalog, on a real claim.
    +    /// A FAILED restore record whose target is still CREATING (the state a
    +    /// stop between the FAILED write and the target's removal leaves) is
    +    /// returned by the zero-window dead-job listing with the RUNNING
    +    /// records; it is the reconciler's duty, not a running job, so the wait
    +    /// is quiet at its first read and runs no duty itself: the target is
    +    /// left CREATING for the server. A FAILED backup row whose heartbeat is
    +    /// within the window (a job that has not yet seen its claim lost may
    +    /// still write under its prefix) holds the wait to its limit, which
    +    /// names it; once the heartbeat has aged past the window it does not.
    +    #[tokio::test]
    +    async fn the_wait_passes_a_failed_record_and_holds_on_a_failed_row_with_a_live_heartbeat() {
    +        use extenddb_engine::HeldClaim;
    +        use extenddb_engine::maintenance::{Interrupt, Wait, WaitOutcome, wait_for_quiet};
    +        if base_conn().is_none() {
    +            eprintln!(
    +                "SKIP the_wait_passes_a_failed_record_and_holds_on_a_failed_row_with_a_live_heartbeat: \
    +                 no PostgreSQL"
    +            );
    +            return;
    +        }
    +        let stale = std::time::Duration::from_secs(300);
    +        let s = scratch().await;
    +        indexed_source(&s, "wq_src").await;
    +        let backup = s
    +            .engine
    +            .create(ACCOUNT, "wq_src", "b", JobPermit::detached())
    +            .await
    +            .expect("backup");
    +        let desc = restore(&s, "wq_dst", &backup.backup_arn)
    +            .await
    +            .expect("restore");
    +        sqlx::query(
    +            "UPDATE tables SET table_status = 'CREATING', status_transition_at = NULL \
    +             WHERE table_id = $1",
    +        )
    +        .bind(&desc.table_id)
    +        .execute(&s.catalog)
    +        .await
    +        .expect("the target back to CREATING");
    +        sqlx::query(
    +            "UPDATE table_restores SET status = 'FAILED', completed_at = NOW() - INTERVAL '1 hour' \
    +             WHERE table_id = $1",
    +        )
    +        .bind(&desc.table_id)
    +        .execute(&s.catalog)
    +        .await
    +        .expect("the record FAILED an hour ago");
    +        let listed = s
    +            .engine
    +            .list_dead_restore_jobs(std::time::Duration::ZERO)
    +            .await
    +            .expect("list");
    +        assert_eq!(listed.len(), 1, "{listed:?}");
    +        assert_eq!(
    +            listed[0].status,
    +            JobStatus::Failed,
    +            "the zero-window listing returns the FAILED record the wait reads"
    +        );
    +
    +        let claim = s
    +            .engine
    +            .maintenance_claim("purge on host-a", stale)
    +            .await
    +            .expect("claim");
    +        let held = HeldClaim::start(claim, std::time::Duration::from_secs(1));
    +        let limit = std::time::Duration::from_secs(3);
    +        let wait = Wait {
    +            owner_window: stale,
    +            limit,
    +            poll: std::time::Duration::from_millis(200),
    +        };
    +        let announced: std::sync::Mutex>> = std::sync::Mutex::new(Vec::new());
    +        let announce = |jobs: &[String]| announced.lock().unwrap().push(jobs.to_vec());
    +        let (_tx, mut interrupt) = Interrupt::new();
    +        let started = std::time::Instant::now();
    +        assert_eq!(
    +            wait_for_quiet(&s.engine, &store(), &held, wait, &mut interrupt, announce).await,
    +            WaitOutcome::Quiet,
    +            "a FAILED restore record is the reconciler's duty, not a running job"
    +        );
    +        assert!(
    +            started.elapsed() < std::time::Duration::from_secs(1),
    +            "quiet at the first read: {:?}",
    +            started.elapsed()
    +        );
    +        assert!(announced.lock().unwrap().is_empty());
    +        assert_eq!(
    +            table_status(&s, "wq_dst").await.as_deref(),
    +            Some("CREATING"),
    +            "the wait runs no duty: the target is left for the server's reconciler"
    +        );
    +        assert_eq!(
    +            job_status(&s, &desc.table_id).await.as_deref(),
    +            Some("FAILED")
    +        );
    +
    +        // A FAILED backup row of the store whose heartbeat is this moment's.
    +        let failed = row(ACCOUNT, "Music", "f");
    +        let job = s.engine.claim_backup(&failed).await.expect("claim");
    +        job.set_status(
    +            JobStatus::Creating,
    +            JobStatus::Failed,
    +            &StatusFields {
    +                failure_reason: Some("the test's".to_owned()),
    +                ..StatusFields::default()
    +            },
    +        )
    +        .await
    +        .expect("failed");
    +        drop(job);
    +        let started = std::time::Instant::now();
    +        let expected = vec![format!(
    +            "the FAILED backup {} has a live owner (heartbeat within stale_job_after_secs)",
    +            failed.backup_arn
    +        )];
    +        assert_eq!(
    +            wait_for_quiet(&s.engine, &store(), &held, wait, &mut interrupt, announce).await,
    +            WaitOutcome::Busy(expected.clone()),
    +            "a FAILED row with a live heartbeat holds the wait and is named"
    +        );
    +        assert!(started.elapsed() >= limit, "{:?}", started.elapsed());
    +        assert_eq!(*announced.lock().unwrap(), vec![expected]);
    +        // Aged past the window on the catalog's clock: a dead owner.
    +        sqlx::query(
    +            "UPDATE backup_index SET heartbeat_at = now() - interval '10 minutes' \
    +             WHERE backup_id = $1",
    +        )
    +        .bind(&failed.backup_id)
    +        .execute(&s.catalog)
    +        .await
    +        .expect("age");
    +        assert_eq!(
    +            wait_for_quiet(&s.engine, &store(), &held, wait, &mut interrupt, announce).await,
    +            WaitOutcome::Quiet,
    +            "a FAILED row whose heartbeat has aged is not waited on"
    +        );
    +        held.release().await.expect("release");
    +        assert!(
    +            !s.engine
    +                .check_maintenance_claim(stale)
    +                .await
    +                .expect("check")
    +        );
    +        s.cleanup().await;
    +    }
    +
    +    /// The factory (`create_server_components`) starts the GSI queue workers
    +    /// for a server only. Components opened `beside_server` (`backup list`,
    +    /// the purge's dry run) leave a pending index write where it is, for
    +    /// the server that owns the catalog; components opened as a server
    +    /// apply it.
    +    #[tokio::test]
    +    async fn components_opened_beside_a_server_apply_no_pending_index_write() {
    +        use extenddb_storage::server_components::ServerComponentsOptions;
    +        if base_conn().is_none() {
    +            eprintln!(
    +                "SKIP components_opened_beside_a_server_apply_no_pending_index_write: no PostgreSQL"
    +            );
    +            return;
    +        }
    +        let s = scratch().await;
    +        indexed_source_with(&s, "gq_src", 0).await;
    +        // An asynchronous index: from here a write queues its index update
    +        // in `gsi_pending` for a worker, instead of applying it in its own
    +        // transaction.
    +        sqlx::query("UPDATE settings SET value = '50' WHERE key = 'index_propagation_delay_ms'")
    +            .execute(&s.catalog)
    +            .await
    +            .expect("an asynchronous index");
    +        let pending = || async {
    +            sqlx::query_scalar::<_, i64>("SELECT count(*) FROM gsi_pending")
    +                .fetch_one(&s.db)
    +                .await
    +                .expect("count gsi_pending")
    +        };
    +        assert_eq!(pending().await, 0);
    +        let desc = s
    +            .engine
    +            .describe_table(
    +                ACCOUNT,
    +                DescribeTableInput {
    +                    table_name: "gq_src".to_owned(),
    +                },
    +            )
    +            .await
    +            .expect("describe");
    +        put_indexed_items(&s, "gq_src", &desc, 1).await;
    +        assert_eq!(pending().await, 1, "the write queued its index update");
    +
    +        // The factory reads the encryption key `init` writes.
    +        sqlx::query(
    +            "INSERT INTO settings (key, value) VALUES ('encryption_key', 'test-key') \
    +             ON CONFLICT (key) DO NOTHING",
    +        )
    +        .execute(&s.catalog)
    +        .await
    +        .expect("encryption key");
    +        // Installed once per process; a second install is refused and that
    +        // is fine.
    +        let _ = extenddb_storage::set_backend(extenddb_storage_postgres::backend());
    +        let config = extenddb_storage_postgres::PostgresStorageConfig {
    +            connection_string: s.catalog_url.clone(),
    +            pool_size: 10,
    +            catalog_pool_size: Some(10),
    +        };
    +        let open = |beside_server: bool| {
    +            let mut options = ServerComponentsOptions::default();
    +            options.release = "test";
    +            options.beside_server = beside_server;
    +            extenddb_storage::create_server_components(&config, REGION, options)
    +        };
    +
    +        let beside = open(true).await.expect("open beside a server");
    +        tokio::time::sleep(std::time::Duration::from_secs(3)).await;
    +        assert_eq!(
    +            pending().await,
    +            1,
    +            "components opened beside a server start no worker: the row is left for the server"
    +        );
    +        drop(beside);
    +
    +        let server = open(false).await.expect("open as a server");
    +        let deadline = std::time::Instant::now() + std::time::Duration::from_secs(10);
    +        while pending().await != 0 {
    +            assert!(
    +                std::time::Instant::now() < deadline,
    +                "a server's workers apply the pending row"
    +            );
    +            tokio::time::sleep(std::time::Duration::from_millis(100)).await;
    +        }
    +        drop(server);
    +        s.cleanup().await;
    +    }
    +}
    +
    +// --- the store identity file, over two engines on one catalog ----------------------
    +
    +/// The identity checks of a configured store (`extenddb_engine::store_identity`)
    +/// run against this catalog and real filesystem stores: two instances of one
    +/// catalog starting against one empty root, and two instances with
    +/// host-local roots under one `store_root` label.
    +mod store_identity {
    +    use std::sync::Arc;
    +
    +    use extenddb_backup_store::{BackupStore, FilesystemStore, byte_stream_from};
    +    use extenddb_core::backup_format::BackupArn;
    +    use extenddb_engine::StoreIdentityFile;
    +    use extenddb_engine::store_identity::{STORE_IDENTITY_KEY, StoreProblem, establish, recheck};
    +    use extenddb_storage::BackupCatalog;
    +    use extenddb_storage::backup_catalog::{JobStatus, StatusFields, StoreKind};
    +
    +    use super::*;
    +
    +    /// A private root: `tempfile` creates directories with the umask applied
    +    /// to 0777, which the store refuses.
    +    fn private_root() -> tempfile::TempDir {
    +        use std::os::unix::fs::PermissionsExt;
    +        let dir = tempfile::tempdir().expect("tempdir");
    +        std::fs::set_permissions(dir.path(), std::fs::Permissions::from_mode(0o700))
    +            .expect("chmod");
    +        dir
    +    }
    +
    +    async fn second_engine(s: &Scratch) -> PostgresEngine {
    +        PostgresEngine::new(
    +            &PostgresConfig {
    +                connection_string: s.catalog_url.clone(),
    +                pool_size: 4,
    +                max_item_size_bytes: 400_000,
    +            },
    +            REGION,
    +        )
    +        .await
    +        .expect("a second engine on the same catalog")
    +    }
    +
    +    async fn read_file(store: &dyn BackupStore) -> StoreIdentityFile {
    +        use futures::TryStreamExt;
    +        let bytes: Vec = store
    +            .get(STORE_IDENTITY_KEY)
    +            .await
    +            .expect("the file")
    +            .try_fold(Vec::new(), |mut acc, chunk| async move {
    +                acc.extend_from_slice(&chunk);
    +                Ok(acc)
    +            })
    +            .await
    +            .expect("read");
    +        StoreIdentityFile::parse(&bytes).expect("parses")
    +    }
    +
    +    /// Two instances of one catalog started against one empty root end with
    +    /// one id in the file, and a row either writes carries it.
    +    #[tokio::test(flavor = "multi_thread", worker_threads = 4)]
    +    async fn two_instances_on_one_empty_root_end_with_one_id() {
    +        if base_conn().is_none() {
    +            eprintln!("SKIP two_instances_on_one_empty_root_end_with_one_id: no PostgreSQL");
    +            return;
    +        }
    +        let s = scratch().await;
    +        let other = second_engine(&s).await;
    +        let root = private_root();
    +        let store_a: Arc =
    +            Arc::new(FilesystemStore::open(root.path()).await.unwrap());
    +        let store_b: Arc =
    +            Arc::new(FilesystemStore::open(root.path()).await.unwrap());
    +        let label = root.path().canonicalize().unwrap().display().to_string();
    +        let (a, b) = tokio::join!(
    +            establish(
    +                &*store_a,
    +                StoreKind::Filesystem,
    +                &label,
    +                &s.engine,
    +                "postgres"
    +            ),
    +            establish(&*store_b, StoreKind::Filesystem, &label, &other, "postgres"),
    +        );
    +        let (a, b) = (a.expect("instance a"), b.expect("instance b"));
    +        assert_eq!(a, b, "both instances use one id");
    +        let file = read_file(&*store_a).await;
    +        assert_eq!(file.store_id, a.store_id);
    +        assert_eq!(file.catalog_id, s.engine.catalog_id().await.unwrap());
    +        assert_eq!(file.backend, "postgres");
    +        // A row written under the identity carries it, and both instances
    +        // read it back as the root's one id.
    +        let mut row = super::catalog_index_row(ACCOUNT, "Music", "nightly");
    +        row.store = a.clone();
    +        let claim = s.engine.claim_backup(&row).await.expect("claim");
    +        drop(claim);
    +        for engine in [&s.engine, &other] {
    +            assert_eq!(
    +                engine
    +                    .read_store_ids(StoreKind::Filesystem, &label)
    +                    .await
    +                    .unwrap(),
    +                std::slice::from_ref(&a.store_id)
    +            );
    +            recheck(&*store_a, &a, engine, "postgres")
    +                .await
    +                .expect("still healthy");
    +        }
    +        s.cleanup().await;
    +    }
    +
    +    /// Two instances of one catalog with host-local filesystem roots under
    +    /// one `store_root` label: the second, whose root has no file while the
    +    /// catalog has rows for the label, is unhealthy; with the first's file
    +    /// copied to it the manifest probe marks it unhealthy, because the
    +    /// newest AVAILABLE backup's manifest is not in its root.
    +    #[tokio::test]
    +    async fn host_local_roots_under_one_label_leave_the_second_instance_unhealthy() {
    +        if base_conn().is_none() {
    +            eprintln!(
    +                "SKIP host_local_roots_under_one_label_leave_the_second_instance_unhealthy: \
    +                 no PostgreSQL"
    +            );
    +            return;
    +        }
    +        let s = scratch().await;
    +        let other = second_engine(&s).await;
    +        let root_a = private_root();
    +        let root_b = private_root();
    +        let store_a: Arc =
    +            Arc::new(FilesystemStore::open(root_a.path()).await.unwrap());
    +        let store_b: Arc =
    +            Arc::new(FilesystemStore::open(root_b.path()).await.unwrap());
    +        // One label for both: the path both hosts configured.
    +        let label = "/var/lib/extenddb/backups";
    +
    +        // The first instance creates the file and an AVAILABLE backup with its
    +        // manifest in its own root.
    +        let a = establish(
    +            &*store_a,
    +            StoreKind::Filesystem,
    +            label,
    +            &s.engine,
    +            "postgres",
    +        )
    +        .await
    +        .expect("instance a");
    +        let mut row = super::catalog_index_row(ACCOUNT, "Music", "nightly");
    +        row.store = a.clone();
    +        let claim = s.engine.claim_backup(&row).await.expect("claim");
    +        let arn = BackupArn::parse(&row.backup_arn).unwrap();
    +        store_a
    +            .put(
    +                &format!("{}extenddb-manifest.json", arn.store_prefix()),
    +                byte_stream_from(b"{}".to_vec()),
    +            )
    +            .await
    +            .expect("manifest");
    +        claim
    +            .set_status(
    +                JobStatus::Creating,
    +                JobStatus::Available,
    +                &StatusFields::default(),
    +            )
    +            .await
    +            .expect("available");
    +        recheck(&*store_a, &a, &s.engine, "postgres")
    +            .await
    +            .expect("a is healthy");
    +
    +        // The second instance: its root has no file and the catalog has rows
    +        // for the label.
    +        let err = establish(&*store_b, StoreKind::Filesystem, label, &other, "postgres")
    +            .await
    +            .unwrap_err();
    +        assert_eq!(
    +            err,
    +            StoreProblem::FileMissingWithRows(vec![a.store_id.clone()])
    +        );
    +        assert!(
    +            store_b.head(STORE_IDENTITY_KEY).await.unwrap().is_none(),
    +            "nothing was created over the rows"
    +        );
    +
    +        // The operator copies the first's file to the second root: the ids
    +        // agree and the probe finds the manifest missing.
    +        let file = read_file(&*store_a).await;
    +        store_b
    +            .put(STORE_IDENTITY_KEY, byte_stream_from(file.to_bytes()))
    +            .await
    +            .unwrap();
    +        let err = establish(&*store_b, StoreKind::Filesystem, label, &other, "postgres")
    +            .await
    +            .unwrap_err();
    +        assert_eq!(
    +            err,
    +            StoreProblem::ManifestMissing {
    +                backup_arn: row.backup_arn.clone(),
    +            }
    +        );
    +        // The first instance is unaffected.
    +        recheck(&*store_a, &a, &s.engine, "postgres")
    +            .await
    +            .expect("a stays healthy");
    +        // A shared root (the second instance pointed at the first's
    +        // directory) passes.
    +        let shared: Arc =
    +            Arc::new(FilesystemStore::open(root_a.path()).await.unwrap());
    +        let b = establish(&*shared, StoreKind::Filesystem, label, &other, "postgres")
    +            .await
    +            .expect("a shared root is healthy");
    +        assert_eq!(b, a);
    +        s.cleanup().await;
    +    }
    +
    +    /// A root whose file names another catalog is unhealthy for this one
    +    /// until an operator changes the file; the first catalog's instance is
    +    /// unaffected.
    +    #[tokio::test]
    +    async fn a_root_owned_by_another_catalog_is_unhealthy() {
    +        if base_conn().is_none() {
    +            eprintln!("SKIP a_root_owned_by_another_catalog_is_unhealthy: no PostgreSQL");
    +            return;
    +        }
    +        let s = scratch().await;
    +        let root = private_root();
    +        let store: Arc =
    +            Arc::new(FilesystemStore::open(root.path()).await.unwrap());
    +        let label = root.path().canonicalize().unwrap().display().to_string();
    +        let foreign = StoreIdentityFile {
    +            format_version: 1,
    +            store_id: "s-foreign".to_owned(),
    +            catalog_id: "another-catalog".to_owned(),
    +            backend: "sqlite".to_owned(),
    +            created_at_ms: 0,
    +        };
    +        store
    +            .put(STORE_IDENTITY_KEY, byte_stream_from(foreign.to_bytes()))
    +            .await
    +            .unwrap();
    +        let err = establish(
    +            &*store,
    +            StoreKind::Filesystem,
    +            &label,
    +            &s.engine,
    +            "postgres",
    +        )
    +        .await
    +        .unwrap_err();
    +        assert_eq!(
    +            err,
    +            StoreProblem::ForeignCatalog {
    +                file_catalog_id: "another-catalog".to_owned(),
    +                catalog_id: s.engine.catalog_id().await.unwrap(),
    +            }
    +        );
    +        s.cleanup().await;
    +    }
     }
    diff --git a/crates/storage-sqlite/src/backup.rs b/crates/storage-sqlite/src/backup.rs
    index 6c3d9ebce..2dfea06a5 100644
    --- a/crates/storage-sqlite/src/backup.rs
    +++ b/crates/storage-sqlite/src/backup.rs
    @@ -4231,7 +4231,12 @@ mod restore_tests {
                     uuid::Uuid::parse_str(&engine.catalog_id().await.expect("id")).is_ok(),
                     "the catalog has an id from the migration"
                 );
    -            assert!(!engine.check_maintenance_claim().await.expect("check"));
    +            assert!(
    +                !engine
    +                    .check_maintenance_claim(std::time::Duration::from_secs(300))
    +                    .await
    +                    .expect("check")
    +            );
                 let row = row(ACCOUNT, "Music", "nightly");
                 let claim = engine.claim_backup(&row).await.expect("claim");
                 assert_ne!(claim.generation(), UNOWNED_GENERATION);
    @@ -4832,5 +4837,216 @@ mod restore_tests {
                     .await
                     .expect("the backup is free");
             }
    +
    +        /// `read_store_ids`: the distinct ids of the rows for a root, of
    +        /// every status, scoped to the kind and the root.
    +        #[tokio::test]
    +        async fn read_store_ids_lists_the_distinct_ids_of_a_root() {
    +            let engine = engine().await;
    +            let root = store().store_root.clone();
    +            assert!(
    +                engine
    +                    .read_store_ids(StoreKind::Filesystem, &root)
    +                    .await
    +                    .expect("ids")
    +                    .is_empty()
    +            );
    +            let first = row(ACCOUNT, "Music", "a");
    +            let _c1 = engine.claim_backup(&first).await.expect("claim");
    +            let second = row(ACCOUNT, "Music", "b");
    +            let c2 = engine.claim_backup(&second).await.expect("claim");
    +            c2.set_status(
    +                JobStatus::Creating,
    +                JobStatus::Failed,
    +                &StatusFields::default(),
    +            )
    +            .await
    +            .expect("fail");
    +            assert_eq!(
    +                engine
    +                    .read_store_ids(StoreKind::Filesystem, &root)
    +                    .await
    +                    .expect("ids"),
    +                ["store-1"],
    +                "two rows of one store, of two statuses, are one id"
    +            );
    +            let mut other = row(ACCOUNT, "Songs", "c");
    +            other.store.store_id = "store-2".to_owned();
    +            let _c3 = engine.claim_backup(&other).await.expect("claim");
    +            assert_eq!(
    +                engine
    +                    .read_store_ids(StoreKind::Filesystem, &root)
    +                    .await
    +                    .expect("ids"),
    +                ["store-1", "store-2"]
    +            );
    +            assert!(
    +                engine
    +                    .read_store_ids(StoreKind::S3, &root)
    +                    .await
    +                    .expect("ids")
    +                    .is_empty(),
    +                "another kind at the same root is another store"
    +            );
    +            assert!(
    +                engine
    +                    .read_store_ids(StoreKind::Filesystem, "/elsewhere")
    +                    .await
    +                    .expect("ids")
    +                    .is_empty()
    +            );
    +        }
    +
    +        /// `remove_dead_creating_backup_row`: a CREATING row whose heartbeat
    +        /// is older than the window is removed outright; one with a live
    +        /// owner, one that is not CREATING, and one that is gone are left,
    +        /// answering `false`.
    +        #[tokio::test]
    +        async fn a_dead_creating_row_is_removed_and_a_live_or_moved_one_is_kept() {
    +            let engine = engine().await;
    +            let stale = Duration::from_secs(300);
    +            let age = |backup_id: String| {
    +                let pool = engine.pool.clone();
    +                async move {
    +                    sqlx::query(
    +                        "UPDATE backup_index SET heartbeat_at = \
    +                         strftime('%Y-%m-%dT%H:%M:%fZ', 'now', '-10 minutes') WHERE backup_id = ?",
    +                    )
    +                    .bind(backup_id)
    +                    .execute(&pool)
    +                    .await
    +                    .expect("age");
    +                }
    +            };
    +            let creating = row(ACCOUNT, "Music", "a");
    +            let key = creating.key();
    +            let claim = engine.claim_backup(&creating).await.expect("claim");
    +            assert!(
    +                !engine
    +                    .remove_dead_creating_backup_row(&key, stale)
    +                    .await
    +                    .unwrap(),
    +                "a live owner keeps its row"
    +            );
    +            age(creating.backup_id.clone()).await;
    +            assert!(
    +                engine
    +                    .remove_dead_creating_backup_row(&key, stale)
    +                    .await
    +                    .unwrap()
    +            );
    +            assert!(
    +                engine
    +                    .read_backup_row(&arn_of(&creating))
    +                    .await
    +                    .unwrap()
    +                    .is_none()
    +            );
    +            assert!(
    +                !engine
    +                    .remove_dead_creating_backup_row(&key, stale)
    +                    .await
    +                    .unwrap(),
    +                "gone"
    +            );
    +            drop(claim);
    +
    +            let available = row(ACCOUNT, "Music", "b");
    +            let claim = engine.claim_backup(&available).await.expect("claim");
    +            claim
    +                .set_status(
    +                    JobStatus::Creating,
    +                    JobStatus::Available,
    +                    &StatusFields::default(),
    +                )
    +                .await
    +                .expect("available");
    +            age(available.backup_id.clone()).await;
    +            assert!(
    +                !engine
    +                    .remove_dead_creating_backup_row(&available.key(), stale)
    +                    .await
    +                    .unwrap(),
    +                "only a CREATING row"
    +            );
    +            assert_eq!(
    +                engine
    +                    .read_backup_row(&arn_of(&available))
    +                    .await
    +                    .unwrap()
    +                    .unwrap()
    +                    .status,
    +                BackupStatus::Available
    +            );
    +        }
    +
    +        /// The maintenance claim: taken once, refused to a second holder
    +        /// naming the first, heartbeated, released; a holder whose heartbeat
    +        /// is older than the stale window is dead, so its claim is not
    +        /// reported held and is taken over, and its handle finds the claim
    +        /// lost.
    +        #[tokio::test]
    +        async fn the_maintenance_claim_excludes_a_second_holder_and_expires_when_stale() {
    +            let engine = engine().await;
    +            let stale = Duration::from_secs(300);
    +            let check = || engine.check_maintenance_claim(stale);
    +            assert!(!check().await.expect("check"));
    +            let first = engine
    +                .maintenance_claim("purge on host-a", stale)
    +                .await
    +                .expect("claim");
    +            assert_ne!(first.generation(), UNOWNED_GENERATION);
    +            assert!(check().await.expect("check"), "a live claim is held");
    +            match engine.maintenance_claim("purge on host-b", stale).await {
    +                Err(StorageError::AlreadyExists(m)) => {
    +                    assert!(m.contains("purge on host-a"), "{m}");
    +                }
    +                Err(other) => panic!("a second holder is refused with AlreadyExists: {other:?}"),
    +                Ok(_) => panic!("a second holder is refused"),
    +            }
    +            assert_eq!(
    +                first.heartbeat().await.expect("heartbeat"),
    +                Heartbeat::Confirmed
    +            );
    +            first.release().await.expect("release");
    +            assert!(!check().await.expect("check"), "released");
    +            first.release().await.expect("a second release is a no-op");
    +
    +            // A dead holder: its heartbeat aged past the window.
    +            let dead = engine
    +                .maintenance_claim("purge on host-c", stale)
    +                .await
    +                .expect("claim");
    +            sqlx::query(
    +                "UPDATE backup_maintenance SET heartbeat_at = \
    +                 strftime('%Y-%m-%dT%H:%M:%fZ', 'now', '-10 minutes')",
    +            )
    +            .execute(&engine.pool)
    +            .await
    +            .expect("age");
    +            assert!(!check().await.expect("check"), "a stale claim is not held");
    +            let taker = engine
    +                .maintenance_claim("purge on host-d", stale)
    +                .await
    +                .expect("a dead holder's claim is taken over");
    +            assert_ne!(taker.generation(), dead.generation());
    +            assert!(check().await.expect("check"));
    +            assert_eq!(
    +                dead.heartbeat().await.expect("heartbeat"),
    +                Heartbeat::Lost,
    +                "the dead holder's handle finds the claim lost"
    +            );
    +            dead.release()
    +                .await
    +                .expect("a release by a lost holder changes nothing");
    +            assert!(check().await.expect("check"), "the taker still holds it");
    +            let holder: String = sqlx::query_scalar("SELECT holder FROM backup_maintenance")
    +                .fetch_one(&engine.pool)
    +                .await
    +                .expect("holder");
    +            assert_eq!(holder, "purge on host-d");
    +            taker.release().await.expect("release");
    +            assert!(!check().await.expect("check"));
    +        }
         }
     }
    diff --git a/crates/storage-sqlite/src/backup_catalog.rs b/crates/storage-sqlite/src/backup_catalog.rs
    index 6f6db92a0..a22931da5 100644
    --- a/crates/storage-sqlite/src/backup_catalog.rs
    +++ b/crates/storage-sqlite/src/backup_catalog.rs
    @@ -22,6 +22,12 @@
     //! A conditional write never has an unknown outcome here (one process, one
     //! file: a statement either ran or errored), so `WriteOutcome::Unknown` is
     //! never returned.
    +//!
    +//! The maintenance claim is the `backup_maintenance` row, taken under the
    +//! engine write lock: refused while a row with a heartbeat younger than the
    +//! stale window exists, taken over (replaced) from a dead holder, heartbeated
    +//! and released by generation. One process serves a file, so a holder is a
    +//! command run while the server is stopped.
     
     use std::collections::HashSet;
     use std::sync::{Arc, Mutex};
    @@ -31,8 +37,8 @@ use extenddb_core::backup_format::BackupArn;
     use extenddb_storage::JobPermit;
     use extenddb_storage::backup_catalog::{
         BackupCatalog, BackupKey, BackupRow, BackupStatus, DeleteStart, Heartbeat, JobClaim, JobRow,
    -    JobStatus, RestoreJobRow, RestoreKind, RestoreTargetKey, StatusFields, StoreIdentity,
    -    StoreKind, TimeBounds, WriteOutcome, new_generation,
    +    JobStatus, MaintenanceClaim, RestoreJobRow, RestoreKind, RestoreTargetKey, StatusFields,
    +    StoreIdentity, StoreKind, TimeBounds, WriteOutcome, new_generation,
     };
     use extenddb_storage::error::StorageError;
     use futures::future::BoxFuture;
    @@ -468,6 +474,20 @@ impl SqliteEngine {
             tx.commit().await.map_err(db_err)?;
             Ok(names)
         }
    +
    +    /// Remove the maintenance claim, whoever holds it: the server's startup
    +    /// step under the serve lock, where one process has the file and so no
    +    /// holder can be alive (a `backup purge` that died, or was killed, left
    +    /// the row with a heartbeat the jobs' check would otherwise honour for
    +    /// the whole stale window). Returns the holder's label when a row was
    +    /// removed.
    +    pub(crate) async fn clear_maintenance_claim(&self) -> Result, StorageError> {
    +        let _writer = self.write_lock.lock().await;
    +        sqlx::query_scalar("DELETE FROM backup_maintenance RETURNING holder")
    +            .fetch_optional(&self.pool)
    +            .await
    +            .map_err(db_err)
    +    }
     }
     
     impl BackupCatalog for SqliteEngine {
    @@ -700,6 +720,29 @@ impl BackupCatalog for SqliteEngine {
             })
         }
     
    +    fn remove_dead_creating_backup_row<'a>(
    +        &'a self,
    +        key: &'a BackupKey,
    +        stale_after: Duration,
    +    ) -> BoxFuture<'a, Result> {
    +        Box::pin(async move {
    +            let _writer = self.write_lock.lock().await;
    +            let affected = sqlx::query(
    +                "DELETE FROM backup_index WHERE account_id = ? AND backup_id = ? \
    +                 AND status = 'CREATING' \
    +                 AND julianday(heartbeat_at) < julianday('now') - ? / 86400.0",
    +            )
    +            .bind(&key.account_id)
    +            .bind(&key.backup_id)
    +            .bind(stale_after.as_secs_f64())
    +            .execute(&self.pool)
    +            .await
    +            .map_err(db_err)?
    +            .rows_affected();
    +            Ok(affected == 1)
    +        })
    +    }
    +
         fn backup_in_use<'a>(
             &'a self,
             key: &'a BackupKey,
    @@ -846,12 +889,121 @@ impl BackupCatalog for SqliteEngine {
             })
         }
     
    -    fn check_maintenance_claim(&self) -> BoxFuture<'_, Result> {
    +    fn read_store_ids<'a>(
    +        &'a self,
    +        store_kind: StoreKind,
    +        store_root: &'a str,
    +    ) -> BoxFuture<'a, Result, StorageError>> {
    +        Box::pin(async move {
    +            sqlx::query_scalar(
    +                "SELECT DISTINCT store_id FROM backup_index \
    +                 WHERE store_kind = ? AND store_root = ? ORDER BY store_id",
    +            )
    +            .bind(store_kind.as_str())
    +            .bind(store_root)
    +            .fetch_all(&self.pool)
    +            .await
    +            .map_err(db_err)
    +        })
    +    }
    +
    +    fn maintenance_claim<'a>(
    +        &'a self,
    +        holder: &'a str,
    +        stale_after: Duration,
    +    ) -> BoxFuture<'a, Result, StorageError>> {
             Box::pin(async move {
    -            sqlx::query_scalar("SELECT EXISTS(SELECT 1 FROM backup_maintenance)")
    -                .fetch_one(&self.pool)
    +            // One process serves a file (the serve lock), so the row is the
    +            // claim: taken under the engine write lock, refused while a row
    +            // with a live heartbeat exists, taken over from a dead one.
    +            let _writer = self.write_lock.lock().await;
    +            let live: Option = sqlx::query_scalar(
    +                "SELECT holder FROM backup_maintenance \
    +                 WHERE julianday(heartbeat_at) >= julianday('now') - ? / 86400.0",
    +            )
    +            .bind(stale_after.as_secs_f64())
    +            .fetch_optional(&self.pool)
    +            .await
    +            .map_err(db_err)?;
    +            if let Some(current) = live {
    +                return Err(StorageError::AlreadyExists(format!(
    +                    "the backup maintenance claim is held by {current}"
    +                )));
    +            }
    +            let generation = new_generation();
    +            sqlx::query(&format!(
    +                "INSERT OR REPLACE INTO backup_maintenance \
    +                 (singleton, holder, owner_generation, claimed_at, heartbeat_at) \
    +                 VALUES (1, ?, ?, {NOW}, {NOW})"
    +            ))
    +            .bind(holder)
    +            .bind(generation_column(generation))
    +            .execute(&self.pool)
    +            .await
    +            .map_err(db_err)?;
    +            Ok(Box::new(SqliteMaintenanceClaim {
    +                engine: self.clone(),
    +                generation,
    +            }) as Box)
    +        })
    +    }
    +
    +    fn check_maintenance_claim(
    +        &self,
    +        stale_after: Duration,
    +    ) -> BoxFuture<'_, Result> {
    +        Box::pin(async move {
    +            sqlx::query_scalar(
    +                "SELECT EXISTS(SELECT 1 FROM backup_maintenance \
    +                 WHERE julianday(heartbeat_at) >= julianday('now') - ? / 86400.0)",
    +            )
    +            .bind(stale_after.as_secs_f64())
    +            .fetch_one(&self.pool)
    +            .await
    +            .map_err(db_err)
    +        })
    +    }
    +}
    +
    +/// The holder's handle on the `backup_maintenance` row.
    +struct SqliteMaintenanceClaim {
    +    engine: SqliteEngine,
    +    generation: u64,
    +}
    +
    +impl MaintenanceClaim for SqliteMaintenanceClaim {
    +    fn generation(&self) -> u64 {
    +        self.generation
    +    }
    +
    +    fn heartbeat(&self) -> BoxFuture<'_, Result> {
    +        Box::pin(async move {
    +            let _writer = self.engine.write_lock.lock().await;
    +            let affected = sqlx::query(&format!(
    +                "UPDATE backup_maintenance SET heartbeat_at = {NOW} WHERE owner_generation = ?"
    +            ))
    +            .bind(generation_column(self.generation))
    +            .execute(&self.engine.pool)
    +            .await
    +            .map_err(db_err)?
    +            .rows_affected();
    +            Ok(if affected == 1 {
    +                Heartbeat::Confirmed
    +            } else {
    +                Heartbeat::Lost
    +            })
    +        })
    +    }
    +
    +    fn release(&self) -> BoxFuture<'_, Result<(), StorageError>> {
    +        Box::pin(async move {
    +            let _writer = self.engine.write_lock.lock().await;
    +            sqlx::query("DELETE FROM backup_maintenance WHERE owner_generation = ?")
    +                .bind(generation_column(self.generation))
    +                .execute(&self.engine.pool)
                     .await
    -                .map_err(db_err)
    +                .map_err(db_err)?;
    +            Ok(())
             })
         }
     }
    diff --git a/crates/storage-sqlite/src/config.rs b/crates/storage-sqlite/src/config.rs
    index 6ba092de0..b0d947d56 100644
    --- a/crates/storage-sqlite/src/config.rs
    +++ b/crates/storage-sqlite/src/config.rs
    @@ -69,6 +69,10 @@ impl StorageConfig for SqliteConfig {
             &self.path
         }
     
    +    fn database_file(&self) -> Option {
    +        crate::serve_lock::database_file(&self.path).ok().flatten()
    +    }
    +
         fn max_connections(&self) -> u32 {
             self.pool_size
         }
    diff --git a/crates/storage-sqlite/src/lib.rs b/crates/storage-sqlite/src/lib.rs
    index 1e3c9a94e..3f2f7cbab 100644
    --- a/crates/storage-sqlite/src/lib.rs
    +++ b/crates/storage-sqlite/src/lib.rs
    @@ -214,18 +214,39 @@ fn sqlite_server_components_factory(
             // One server per database file, taken before the database is opened:
             // startup recovery below assumes no other server is using it, and a
             // migration holds the same lock. A file database only; an in-memory
    -        // one belongs to this process alone.
    -        let serve_lock = serve_lock::ServeLock::try_acquire_for_location(&db_path)
    -            .map_err(|failure| {
    -                BackendError::Catalog(match failure {
    -                    serve_lock::LockFailure::Held(m) => MigrationError::Locked(m),
    -                    serve_lock::LockFailure::Failed(m) => MigrationError::Store(m),
    +        // one belongs to this process alone. A command reading beside a
    +        // server (`options.beside_server`) takes no lock and runs none of
    +        // the recovery: the file is a served database and the server owns
    +        // both.
    +        let serve_lock = if options.beside_server {
    +            // The catalog's status is still read, on a read-only
    +            // connection: a catalog this binary cannot serve (behind, or
    +            // ahead with a migration that is not additive) is reported as
    +            // the server would report it, not as a missing table.
    +            match migrations::catalog_status(&db_path)
    +                .await
    +                .map_err(BackendError::Catalog)?
    +            {
    +                extenddb_storage::migrations::CatalogStatus::Current { .. }
    +                | extenddb_storage::migrations::CatalogStatus::AheadCompatible { .. } => {}
    +                status => {
    +                    return Err(BackendError::Catalog(MigrationError::Status(status)));
    +                }
    +            }
    +            None
    +        } else {
    +            serve_lock::ServeLock::try_acquire_for_location(&db_path)
    +                .map_err(|failure| {
    +                    BackendError::Catalog(match failure {
    +                        serve_lock::LockFailure::Held(m) => MigrationError::Locked(m),
    +                        serve_lock::LockFailure::Failed(m) => MigrationError::Store(m),
    +                    })
    +                })?
    +                .map(|lock| {
    +                    tracing::debug!("holding serve lock {}", lock.path().display());
    +                    std::sync::Arc::new(lock)
                     })
    -            })?
    -            .map(|lock| {
    -                tracing::debug!("holding serve lock {}", lock.path().display());
    -                std::sync::Arc::new(lock)
    -            });
    +        };
             // The catalog's status is read before the engine opens the file, so
             // a server that refuses has written nothing. A dev build migrates
             // instead, as `migrate --yes` would, copy included.
    @@ -241,6 +262,9 @@ fn sqlite_server_components_factory(
                     backend: "sqlite".to_owned(),
                     details: e.to_string(),
                 })?;
    +        if options.beside_server {
    +            return components_of(engine, &db_path).await;
    +        }
     
             // An in-memory database exists only on the engine's single
             // connection, so its catalog is created there, by the runner, every
    @@ -307,6 +331,19 @@ fn sqlite_server_components_factory(
                 Err(e) => tracing::error!("Failed to mark interrupted restores: {e}"),
             }
     
    +        // A maintenance claim (`backup purge`) the previous holder left: one
    +        // process has this file, so the holder is dead whatever the row's
    +        // heartbeat says, and the row would otherwise make the jobs refuse
    +        // for the whole stale window.
    +        match engine.clear_maintenance_claim().await {
    +            Ok(Some(holder)) => tracing::warn!(
    +                "the maintenance claim held by {holder} was left behind; cleared (one process \
    +                 serves this file, so its holder is gone)"
    +            ),
    +            Ok(None) => {}
    +            Err(e) => tracing::error!("Failed to clear a left-behind maintenance claim: {e}"),
    +        }
    +
             // Rebuild any GSI left mid-backfill (status CREATING) by a prior crash.
             match engine.reconcile_incomplete_gsis().await {
                 Ok(n) if n > 0 => tracing::info!("Reconciled {n} incomplete GSI(s) at startup"),
    @@ -326,59 +363,72 @@ fn sqlite_server_components_factory(
             }
     
             engine.serve_lock = serve_lock;
    -        let control_plane_notify = engine.control_plane_notify();
    -        let engine = Arc::new(engine);
    -
    -        // Catalog + auth queries: a file-backed database opens a dedicated
    -        // catalog pool over the same file; an in-memory database must reuse the
    -        // engine's single shared connection (its data lives only there).
    -        let catalog_pool = if in_memory {
    -            engine.pool.clone()
    -        } else {
    -            catalog_pool(&db_path)
    -                .await
    -                .map_err(|e| BackendError::ConnectionFailed {
    -                    backend: "sqlite".to_owned(),
    -                    details: format!("catalog pool: {e}"),
    -                })?
    -        };
    +        components_of(engine, &db_path).await
    +    })
    +}
     
    -        let enc_key: Option =
    -            sqlx::query_scalar("SELECT value FROM settings WHERE key = 'encryption_key'")
    -                .fetch_optional(&catalog_pool)
    -                .await
    -                .map_err(|e| {
    -                    BackendError::InitializationFailed(format!("fetch encryption key: {e}"))
    -                })?;
    -        let enc_key = enc_key.ok_or(BackendError::MissingEncryptionKey)?;
    -
    -        // The catalog store writes through the second pool over the same
    -        // file, so its writers must hold the SAME lock instance as the
    -        // engine's (design decision D1); an unshared lock would leave the two
    -        // pools contending at the SQLite level, where a slow commit exhausts
    -        // a concurrent writer's busy_timeout ("database is locked" -> 500).
    -        // For an in-memory database the pools are one and the sharing is
    -        // harmless. The credential store only reads and needs no lock.
    -        let catalog_store: Arc =
    -            Arc::new(SqliteCatalogStore::with_encryption_key(
    -                catalog_pool.clone(),
    -                enc_key.clone(),
    -                Arc::clone(&engine.write_lock),
    -            ));
    -        let credential_store: Arc =
    -            Arc::new(SqliteCredentialStore::new(catalog_pool, enc_key));
    -
    -        let runtime_hooks = Box::new(SqliteRuntimeHooks {
    -            engine: engine.clone(),
    -            control_plane_notify,
    -        });
    -
    -        Ok(ServerComponents {
    -            engine,
    -            catalog_store,
    -            credential_store,
    -            runtime_hooks: Some(runtime_hooks),
    -        })
    +/// The server components around an opened engine: the catalog and
    +/// credential stores over a second pool on the same file (the engine's own
    +/// connection for an in-memory database, whose data lives only there) and
    +/// the runtime hooks.
    +async fn components_of(
    +    engine: SqliteEngine,
    +    db_path: &str,
    +) -> Result {
    +    let in_memory = engine.in_memory;
    +    let control_plane_notify = engine.control_plane_notify();
    +    let engine = Arc::new(engine);
    +
    +    // Catalog + auth queries: a file-backed database opens a dedicated
    +    // catalog pool over the same file; an in-memory database must reuse the
    +    // engine's single shared connection (its data lives only there).
    +
    +    let catalog_pool = if in_memory {
    +        engine.pool.clone()
    +    } else {
    +        catalog_pool(db_path)
    +            .await
    +            .map_err(|e| BackendError::ConnectionFailed {
    +                backend: "sqlite".to_owned(),
    +                details: format!("catalog pool: {e}"),
    +            })?
    +    };
    +
    +    let enc_key: Option =
    +        sqlx::query_scalar("SELECT value FROM settings WHERE key = 'encryption_key'")
    +            .fetch_optional(&catalog_pool)
    +            .await
    +            .map_err(|e| {
    +                BackendError::InitializationFailed(format!("fetch encryption key: {e}"))
    +            })?;
    +    let enc_key = enc_key.ok_or(BackendError::MissingEncryptionKey)?;
    +
    +    // The catalog store writes through the second pool over the same
    +    // file, so its writers must hold the SAME lock instance as the
    +    // engine's (design decision D1); an unshared lock would leave the two
    +    // pools contending at the SQLite level, where a slow commit exhausts
    +    // a concurrent writer's busy_timeout ("database is locked" -> 500).
    +    // For an in-memory database the pools are one and the sharing is
    +    // harmless. The credential store only reads and needs no lock.
    +    let catalog_store: Arc =
    +        Arc::new(SqliteCatalogStore::with_encryption_key(
    +            catalog_pool.clone(),
    +            enc_key.clone(),
    +            Arc::clone(&engine.write_lock),
    +        ));
    +    let credential_store: Arc =
    +        Arc::new(SqliteCredentialStore::new(catalog_pool, enc_key));
    +
    +    let runtime_hooks = Box::new(SqliteRuntimeHooks {
    +        engine: engine.clone(),
    +        control_plane_notify,
    +    });
    +
    +    Ok(ServerComponents {
    +        engine,
    +        catalog_store,
    +        credential_store,
    +        runtime_hooks: Some(runtime_hooks),
         })
     }
     
    @@ -417,4 +467,63 @@ mod serve_lock_tests {
             drop(again);
             let _ = std::fs::remove_dir_all(&dir);
         }
    +
    +    /// A maintenance claim a killed `backup purge` left (its row's heartbeat
    +    /// young) is cleared when a server starts on the file: under the serve
    +    /// lock no holder can be alive, so the jobs' check answers free at once
    +    /// rather than after the stale window.
    +    #[tokio::test]
    +    async fn a_server_start_clears_a_left_behind_maintenance_claim() {
    +        let dir = std::env::temp_dir().join(format!(
    +            "extenddb-serve-clears-claim-{}",
    +            uuid::Uuid::new_v4().simple()
    +        ));
    +        std::fs::create_dir_all(&dir).expect("dir");
    +        let config = crate::config::SqliteConfig {
    +            path: dir.join("db.sqlite").to_string_lossy().into_owned(),
    +            pool_size: 2,
    +        };
    +        let mut options = extenddb_storage::server_components::ServerComponentsOptions::default();
    +        options.bootstrap_if_uninitialized = true;
    +        let stale = std::time::Duration::from_secs(300);
    +        let first = super::sqlite_server_components_factory(&config, "us-east-1", options)
    +            .await
    +            .expect("first server");
    +        let claim = first
    +            .engine
    +            .maintenance_claim("purge on this host (pid 1)", stale)
    +            .await
    +            .expect("claim");
    +        assert!(
    +            first
    +                .engine
    +                .check_maintenance_claim(stale)
    +                .await
    +                .expect("check")
    +        );
    +        // The holder dies without releasing: the row stays, heartbeat young.
    +        drop(claim);
    +        drop(first);
    +
    +        let second = super::sqlite_server_components_factory(&config, "us-east-1", options)
    +            .await
    +            .expect("starts once the first is gone");
    +        assert!(
    +            !second
    +                .engine
    +                .check_maintenance_claim(stale)
    +                .await
    +                .expect("check"),
    +            "the claim a dead holder left is cleared at startup, not honoured for the window"
    +        );
    +        // The claim is free to a new holder at once.
    +        let taken = second
    +            .engine
    +            .maintenance_claim("purge on this host (pid 2)", stale)
    +            .await
    +            .expect("a new holder takes the cleared claim");
    +        taken.release().await.expect("release");
    +        drop(second);
    +        let _ = std::fs::remove_dir_all(&dir);
    +    }
     }
    diff --git a/crates/storage/src/backup_catalog.rs b/crates/storage/src/backup_catalog.rs
    index d2168aced..8079298f5 100644
    --- a/crates/storage/src/backup_catalog.rs
    +++ b/crates/storage/src/backup_catalog.rs
    @@ -748,6 +748,18 @@ pub trait BackupCatalog: Send + Sync {
             key: &'a BackupKey,
         ) -> BoxFuture<'a, Result>;
     
    +    /// Remove a CREATING row whose owner is dead: `heartbeat_at` older than
    +    /// `stale_after` on the catalog's clock. `false` when the row is not
    +    /// CREATING, is gone, or has a live owner. The maintenance command's
    +    /// write for a CREATING row its wait saw no live owner for, after it has
    +    /// deleted the row's prefix; the row's own job, were it alive, would
    +    /// find its claim lost at its next heartbeat.
    +    fn remove_dead_creating_backup_row<'a>(
    +        &'a self,
    +        key: &'a BackupKey,
    +        stale_after: Duration,
    +    ) -> BoxFuture<'a, Result>;
    +
         /// The name of a CREATING restore target registered from the backup, if
         /// any: the rule under which DeleteBackup and a second restore of the
         /// backup are refused with `BackupInUseException`. Reads the target's
    @@ -798,10 +810,64 @@ pub trait BackupCatalog: Send + Sync {
             reason: &'a str,
         ) -> BoxFuture<'a, Result>;
     
    -    /// Whether the catalog-level maintenance claim is held. Jobs check it
    -    /// after their own claim and row exist and refuse while it is held; the
    -    /// reconciler runs only the duties the holder waits on.
    -    fn check_maintenance_claim(&self) -> BoxFuture<'_, Result>;
    +    /// The distinct `store_id`s the catalog's rows record for a root, of
    +    /// every status: the store identity check reads them before it knows an
    +    /// id. Empty when no row names the root.
    +    fn read_store_ids<'a>(
    +        &'a self,
    +        store_kind: StoreKind,
    +        store_root: &'a str,
    +    ) -> BoxFuture<'a, Result, StorageError>>;
    +
    +    /// Take the catalog-level maintenance claim for `holder` (a label naming
    +    /// the command and the host it runs on, for the operator who finds it
    +    /// held). The claim is one record, `backup_maintenance`, born owned
    +    /// with a fresh generation and heartbeated through the handle; a holder
    +    /// whose heartbeat is older than `stale_after` on the catalog's clock is
    +    /// dead and its claim is taken over. PostgreSQL is the exception to the
    +    /// liveness rule on the holder's side: there the holder's session holds
    +    /// an advisory lock, a holder whose session ended is dead whatever its
    +    /// heartbeat says, and `stale_after` is read by [`check_maintenance_claim`]
    +    /// alone; on SQLite a server's startup under the serve lock removes the
    +    /// record outright, since no holder can be alive there.
    +    ///
    +    /// [`check_maintenance_claim`]: BackupCatalog::check_maintenance_claim
    +    ///
    +    /// # Errors
    +    ///
    +    /// `AlreadyExists` naming the holder when a live one holds the claim: a
    +    /// second maintenance command fails at once.
    +    fn maintenance_claim<'a>(
    +        &'a self,
    +        holder: &'a str,
    +        stale_after: Duration,
    +    ) -> BoxFuture<'a, Result, StorageError>>;
    +
    +    /// Whether the catalog-level maintenance claim is held by a live holder:
    +    /// a record whose heartbeat is younger than `stale_after` on the
    +    /// catalog's clock. Jobs check it after their own claim and row exist and
    +    /// refuse while it is held; the reconciler runs only the duties the
    +    /// holder waits on.
    +    fn check_maintenance_claim(
    +        &self,
    +        stale_after: Duration,
    +    ) -> BoxFuture<'_, Result>;
    +}
    +
    +/// The holder's handle on the catalog-level maintenance claim: the
    +/// heartbeat that keeps it, and the release that ends it. Dropping the handle
    +/// without `release` leaves the record for the stale rule to expire.
    +pub trait MaintenanceClaim: Send + Sync {
    +    /// The fencing token the claim was taken with.
    +    fn generation(&self) -> u64;
    +
    +    /// Refresh `heartbeat_at` on the catalog's clock and confirm the claim is
    +    /// still this holder's. The caller treats an `Err` as [`Heartbeat::Lost`].
    +    fn heartbeat(&self) -> BoxFuture<'_, Result>;
    +
    +    /// Remove the record, conditioned on the generation; a record another
    +    /// holder took over is left to it. Idempotent.
    +    fn release(&self) -> BoxFuture<'_, Result<(), StorageError>>;
     }
     
     /// A job's handle on its row: the claim that excludes a second owner and
    diff --git a/crates/storage/src/config.rs b/crates/storage/src/config.rs
    index 64ffba9f5..21ff4914a 100644
    --- a/crates/storage/src/config.rs
    +++ b/crates/storage/src/config.rs
    @@ -59,6 +59,16 @@ pub trait StorageConfig: Send + Sync + std::fmt::Debug {
             Vec::new()
         }
     
    +    /// The local file the database lives in, for a backend whose database is
    +    /// one file on this host (SQLite with a file path); `None` for a server
    +    /// database and for an in-memory database. The backup configuration
    +    /// derives its per-backend defaults from it: a store root and a restore
    +    /// spool beside the file, and the rule that neither may sit in the
    +    /// file's directory. The default is `None`.
    +    fn database_file(&self) -> Option {
    +        None
    +    }
    +
         /// Clone this config into a boxed trait object.
         fn clone_box(&self) -> Box;
     
    diff --git a/crates/storage/src/lib.rs b/crates/storage/src/lib.rs
    index beca72727..f7fbd6cf6 100755
    --- a/crates/storage/src/lib.rs
    +++ b/crates/storage/src/lib.rs
    @@ -36,7 +36,9 @@ pub mod vector_catalog;
     pub mod vector_lifecycle;
     
     pub use backend::{Backend, BackendAlreadySet, backend_name, set_backend, try_backend};
    -pub use backup_catalog::{BackupCatalog, JobClaim, JobPermit, LegacyBackupReader};
    +pub use backup_catalog::{
    +    BackupCatalog, JobClaim, JobPermit, LegacyBackupReader, MaintenanceClaim,
    +};
     pub use backup_definition::{BackupRestoreSource, RestoreRequest, RestoreTableOverrides};
     pub use snapshot::{SnapshotEngine, SnapshotInfo, TableSnapshot};
     
    diff --git a/crates/storage/src/server_components.rs b/crates/storage/src/server_components.rs
    index bb07b8924..3fcbf57f3 100644
    --- a/crates/storage/src/server_components.rs
    +++ b/crates/storage/src/server_components.rs
    @@ -101,6 +101,15 @@ pub struct ServerComponentsOptions {
         /// sizes it from this ([`crate::backup_catalog::pool_size_for_jobs`]);
         /// `0` (the default outside a server) asks for no job capacity.
         pub max_concurrent_jobs: usize,
    +    /// The caller is a command that reads the catalog beside a running
    +    /// server (`extenddb backup list`), not a server: no serve lock is taken
    +    /// (on SQLite the file is opened as another reader of a served
    +    /// database) and none of the startup recovery a server runs (interrupted
    +    /// control-plane transitions, incomplete indexes, interrupted restores)
    +    /// is run, since the server that owns the catalog does that. A command
    +    /// that writes (`backup purge`) leaves this `false` and opens the catalog
    +    /// as a server does.
    +    pub beside_server: bool,
     }
     
     /// Factory function type for creating server components.
    diff --git a/devtools/run-tests b/devtools/run-tests
    index 421a97b3a..0c46e71be 100755
    --- a/devtools/run-tests
    +++ b/devtools/run-tests
    @@ -467,7 +467,7 @@ if $RUN_PYTEST; then
         fi
         echo "  AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID:-}"
         # CLI lifecycle tests manage their own server; exclude from main suite
    -    PYTEST_ARGS=(python3 -m pytest tests/ -v --ignore=tests/python --ignore=tests/test_cli_lifecycle.py --ignore=tests/test_cli_container_readiness.py --ignore=tests/test_cli_migrate_concurrency.py --ignore=tests/test_cli_vector_catalog_migration.py --ignore=tests/test_gsi_async_queue.py)
    +    PYTEST_ARGS=(python3 -m pytest tests/ -v --ignore=tests/python --ignore=tests/test_cli_lifecycle.py --ignore=tests/test_cli_container_readiness.py --ignore=tests/test_cli_migrate_concurrency.py --ignore=tests/test_cli_vector_catalog_migration.py --ignore=tests/test_gsi_async_queue.py --ignore=tests/test_backup_store_config.py)
         # Parallel execution (--parallel): distribute by file so module/class-scoped
         # fixtures stay within a single worker.
         if [[ -n "$PARALLEL" ]] && python3 -c "import xdist" 2>/dev/null; then
    diff --git a/docs/design/15-backup-restore.md b/docs/design/15-backup-restore.md
    index 477e344c1..5e9759fc0 100644
    --- a/docs/design/15-backup-restore.md
    +++ b/docs/design/15-backup-restore.md
    @@ -100,7 +100,7 @@ The invariants and state machines every later PR honours, by lettered part; the
     
     (e) The claim. There is no single-instance lease on PostgreSQL or MongoDB, so each job holds a claim that both excludes a second owner and lets a successor detect a dead one. Every backend sets `owner_generation` (a fresh random value) at claim time, and every catalog write by the job is `WHERE owner_generation = $gen`, so a job that lost its claim cannot publish over a successor's work. No row ever exists unowned: on every backend the claim precedes the row's insert or is the insert itself (5.1 step 2); what the claim is per backend is PR 6's section. Who is dead: on SQLite every CREATING row at startup, and at runtime a row whose in-process slot no task holds; on PostgreSQL, MongoDB and Cassandra a row whose `heartbeat_at` is older than `stale_job_after_secs` (default 300), the one liveness rule the three share. On PostgreSQL the advisory lock is the claim's proof against a second owner, not the liveness test: a free lock shows that the owner's catalog session ended, not that its process stopped (a session a managed server dropped leaves a process whose data writes carry no generation), and the owner itself aborts at its next heartbeat inside the stale window, so the reconciler reads the heartbeat alone and never probes the lock. Staleness is always measured on the database's clock, never on an application host's. The reconciler takes the same claim as a worker, with a fresh `owner_generation` that fences the dead job's generation out, leaves `heartbeat_at` untouched (a refreshed heartbeat would restart the stale window the prefix deletion waits for), writes AVAILABLE or FAILED at once, both conditioned on `status = CREATING`, and never continues the upload. A job whose conditional write matches no row, or whose conditional write errors with an unknown outcome, performs no further store or table operation, deletes of a prefix or of a restore target included (5.1 step 5, 5.3 step 7). A FAILED `restore_jobs` row whose target table is still CREATING (the FAILED write landed and the target's removal did not, whether the job's own or the reconciler's) is the FAILED-target duty: the reconciler lists it once the row's `completed_at` (or, where none was stamped, its `heartbeat_at`) is older than `stale_job_after_secs`, so a writer still on its own failure path finishes it, and removes the target through `remove_restore_target` with no second FAILED write; PR 6's reconciler runs it for every kind of record, since on this rule the kind does not matter (PR 8 adds the `target_not_empty` exception). On Cassandra the removal is itself two writes, a lightweight transaction claiming the target CREATING to DELETING and then the drop, so the duty also lists a FAILED row whose target is DELETING with its registration still present, on the same staleness rule, and `remove_restore_target` resumes the removal from the drop; PostgreSQL and SQLite remove in one transaction and MongoDB deletes the document first, so neither leaves that state.
     
    -(f) The maintenance claim. `backup purge`, `backup remove-target` and `backup sync` with `--move-store`, `--forget-store`, `--forget-backup`, `--prune-missing` or `--adopt-store` change rows or bytes that jobs on any instance may be using, and a job permit (5.1 step 1) is per instance, so these commands take a catalog-level maintenance claim first (PR 10's section: the claim per backend and the SQLite serve lock). Every backup and restore job checks the claim after its own claim and row exist (5.1 step 2, 5.3 step 4) and refuses while it is held; while a claim is held the reconciler runs only the duties that resolve the states the command waits on (DELETING rows, rows whose owner is dead, the FAILED-target duty) and skips every other duty; the command waits until no CREATING backup row with a live owner, no DELETING row of the configured store and no RUNNING `restore_jobs` row remains, and refuses and releases the claim if they have not gone within `2 * stale_job_after_secs`, which it prints; a second maintenance command while one holds the claim fails at once. Because the job checks after its row exists and the command checks rows after its claim exists, one of the two always sees the other.
    +(f) The maintenance claim. `backup purge`, `backup remove-target` and `backup sync` with `--move-store`, `--forget-store`, `--forget-backup`, `--prune-missing` or `--adopt-store` change rows or bytes that jobs on any instance may be using, and a job permit (5.1 step 1) is per instance, so these commands take a catalog-level maintenance claim first (PR 10's section: the claim per backend and the SQLite serve lock). Every backup and restore job checks the claim after its own claim and row exist (5.1 step 2, 5.3 step 4; this release's in-catalog create and restore check before their row, since the backends commit it and detach, revision item 28) and refuses while it is held; while a claim is held the reconciler runs only the duties that resolve the states the command waits on (DELETING rows, rows whose owner is dead, the FAILED-target duty) and skips every other duty; the command waits until no CREATING or FAILED backup row of the configured store has a live owner (a heartbeat within `stale_job_after_secs`; a FAILED row's heartbeat may still be refreshed by a job that has not seen its claim lost) and no RUNNING `restore_jobs` row remains, a DELETING row not being waited on (nothing heartbeats one; the command finishes it as it finishes every row), and refuses and releases the claim if they have not gone within `2 * stale_job_after_secs`, which it prints; a second maintenance command while one holds the claim fails at once. Because the job checks after its row exists and the command checks rows after its claim exists, one of the two always sees the other.
     
     (g) The store, the rows and sync. The store is the source of truth; the row is a cache. Sync considers only rows of the configured store (`store_kind`, `store_root`, `store_id`); rows for another store are reported, not touched, and `backup sync --forget-store ` is the explicit opt-in to drop them: it deletes that root's catalog rows outright, leaving no DELETED tombstone, so no deletion duty ever acts on that store's bytes. Sync reports a row whose manifest is gone and marks it DELETING only with `--prune-missing`; without a resolving option (PR 10) it never modifies CREATING, DELETING, DELETED or FAILED rows nor adopts their prefixes, and it never adopts a prefix without a manifest, one whose sidecar does not match, or one whose listing lacks a data file the manifest lists or holds it at another size. Neither Describe nor Restore reads the store for an ARN that has no row: a backup that exists only in the store is BackupNotFound until `backup sync` adopts it, so these rules are the only path from a prefix to an AVAILABLE row. ListBackups lists rows of another store like any other (the response shape has no marker); Describe of such a row returns it and logs a warning; DeleteBackup and Restore of such a row return a ValidationException saying the backup is in a store other than the configured one, without naming either root. No API error names a store root.
     
    @@ -117,7 +117,7 @@ Per operation, the contract: the request checks in their order, the error each r
     #### 5.1 CreateBackup
     
     1. Validate the request as #384 does and in its order, every check before the permit: `TableName` resolved (a table ARN is accepted), the table exists and is ACTIVE for the account (`TableNotFoundException`, "Table not found: ", otherwise, whether or not a permit is free), the server's region and the table name fit a backup ARN (`check_backup_arn_fits`, each refusal naming its cause: a `server.region` outside `[a-z0-9-]` is a ValidationException naming the setting, since no backup ARN can be formed for any table under it; a name over 255 bytes or made only of dots, the store-prefix rules, is one naming the rule, applied before the ARN is formed so the limit is named whatever the name's length). #384's `refuse_backup_of_vector_table` is not among the off-host path's checks (the manifest records vector indexes, section 1); the engine keeps the refusal in front of `LegacyBackupReader::create`, the path of every backend without `[backup.store]` and of Cassandra always (section 7), and the tests that change are PR 8's and PR 9a's sections. A second CreateBackup on a table that already has a CREATING backup is allowed and takes its own snapshot; the `BackupInUseException` the service models on CreateBackup is not raised. Then reserve a job slot: `max_concurrent_jobs` is a per-instance semaphore the request `try_acquire`s (the maintenance claim of section 4 is the cross-instance exclusion), and the permit lives with the job; if none is free return `LimitExceededException` before opening anything, because a queued snapshot would hold a PostgreSQL transaction or a MongoDB snapshot session while it waited; RestoreTableFromBackup and the legacy create take a permit from the same semaphore and return the same error (5.3 step 1, section 3). The default is 4 (section 7), so one running restore does not turn every other backup operation on the instance into `LimitExceededException`.
    -2. Open the table snapshot (`SnapshotEngine::open_table_snapshot`) in the request. The pin happens during the request, so the backup holds every write acknowledged before the request was received, which is stricter than the service's one-minute window. Mint the backup id from the millisecond the open returned, just after the pin (`format_backup_id`), and the owner generation, then, through `BackupCatalog::claim_backup` (section 3), insert the catalog row with status CREATING, `created_at` = the id's instant, `owner_generation`, `heartbeat_at` = the database's `now()`, and `source_table_details_json` taken from the snapshot's description; then check the maintenance claim (section 4): if it is held, set the row FAILED with reason `maintenance`, release the snapshot, the lock and the permit, and return `LimitExceededException`; otherwise return `BackupDetails` with BackupStatus CREATING. Everything from the open to the hand-off in step 3 runs in a detached task whose result the request awaits, so a client that disconnects mid-request cannot leave a row whose owner was never spawned; if the row insert fails the snapshot is dropped, the lock released, the permit returned, and nothing was written.
    +2. Open the table snapshot (`SnapshotEngine::open_table_snapshot`) in the request. The pin happens during the request, so the backup holds every write acknowledged before the request was received, which is stricter than the service's one-minute window. Mint the backup id from the millisecond the open returned, just after the pin (`format_backup_id`), and the owner generation, then, through `BackupCatalog::claim_backup` (section 3), insert the catalog row with status CREATING, `created_at` = the id's instant, `owner_generation`, `heartbeat_at` = the database's `now()`, and `source_table_details_json` taken from the snapshot's description; then check the maintenance claim (section 4; this release's in-catalog create checks it before its row, revision item 28): if it is held, set the row FAILED with reason `maintenance`, release the snapshot, the lock and the permit, and return `LimitExceededException`; otherwise return `BackupDetails` with BackupStatus CREATING. Everything from the open to the hand-off in step 3 runs in a detached task whose result the request awaits, so a client that disconnects mid-request cannot leave a row whose owner was never spawned; if the row insert fails the snapshot is dropped, the lock released, the permit returned, and nothing was written.
     3. The claim: section 4(e) states the rules; what the claim is on each backend, the claim connection's settings and the hand-off to the worker are PR 6's and PR 8's sections.
     4. The job streams items from the snapshot into gzip data files through `DataFileEncoder`; a file closes once its `uncompressed_bytes` reaches `data_file_target_bytes` (default 64 MiB), so a file exceeds the knob by less than one line, at most `MAX_LINE_BYTES` (32 MiB, section 1), which is also the knob's minimum. Every item is checked, before it is encoded, with the item validator PR 8 adds to `extenddb-core` (`extenddb_core::validation::validate_restorable_item`, with this instance's `max_item_size_bytes`; the restore binds the same function, 5.3 step 5), so no backup that no restore can load is reported AVAILABLE; that is a regression from main for a table that already holds a value #618 let in (Drawbacks), and the reason reaches the operator only (`backup list --failed` and the log); the client sees CREATING and then BackupNotFound. The heartbeat is written on a timer (`heartbeat_interval_secs`, default 15) for as long as the job task is alive, independent of upload progress; before each put the job checks that its latest heartbeat confirmed the claim within `heartbeat_interval_secs` and starts no put otherwise; a heartbeat that does not confirm the claim, or that has not returned within `heartbeat_interval_secs`, aborts the job without further store writes, deletes included. `stale_job_after_secs` must exceed several heartbeat periods; startup refuses a value under three times the period the jobs run at, which until the knob reaches the jobs is the larger of `heartbeat_interval_secs` and the 15 s constant `extenddb_storage::backup_catalog::JOB_HEARTBEAT_INTERVAL` every backend's heartbeat loop reads (so 45 s at the least), which is what makes that detection precede a reconciler's deletion. After the last file: write `manifest-files.json`, `manifest-summary.json`, `extenddb-manifest.sha256` (the order #394's `BackupDocuments::build` doc gives); re-check the claim; then write `extenddb-manifest.json` and update the row to AVAILABLE with size and count (`JobClaim::set_status(CREATING, AVAILABLE)`, conditional on the generation and on `status = CREATING`). An update whose outcome is unknown is followed by a re-read of the row, not treated as a failure; one that matches zero rows ends the job with no store access. Store writes are not fenced by the generation; the re-check narrows the window in which a fenced-out job can still write a manifest, and the FAILED tombstone (section 4) closes what remains: sync never adopts a prefix whose row is FAILED, and the reconciler never deletes a prefix while the row's `heartbeat_at` is younger than `stale_job_after_secs` on the catalog database's clock, so a late manifest from a fenced-out job lands under a tombstone that sync will not adopt.
     5. On any failure, including `SnapshotExpired`: set the row FAILED with the reason, conditional on the generation and on `status = CREATING`, so a job whose AVAILABLE update committed with its response lost cannot turn a backup a client has seen AVAILABLE into a deleted one. Only if that write reports exactly one row matched does the job stop its uploads, await its in-flight puts and `delete_prefix` the partial backup (which on S3 also aborts any upload still open under it); a write that matches zero rows, or any error on the claim connection, ends the job with no further store access of any kind, because the holder of the claim owns the prefix. If the delete fails the reconciler retries it (section 4). A backup never resumes from a cursor position, so a lost snapshot means a new CreateBackup. DescribeBackup returns BackupNotFound; ListBackups omits it.
    @@ -135,7 +135,7 @@ The request runs steps 1, 2 and 4 and returns CREATING in milliseconds plus the
     1. Check the request in the order main uses and the differences doc records, every refusal that needs no store access before the permit: the overrides' shape (below); the ARN with `BackupArn` (the account must equal the caller's before any store access); the target name (`TableAlreadyExistsException` when a table of that name exists, `TableInUseException` when it is a CREATING restore target); the backup row, read by the ARN's account and backup id and accepted only when its `account_id`, `table_name` and `backup_arn` (after the region re-spelling of section 4) equal the request's, or, if there is no such row, resolved through the `LegacyBackupReader` (section 3): an ARN with neither is `BackupNotFoundException`, so an ARN of account B that carries the id of account A's backup is BackupNotFound and never A's row, a CREATING row returns `BackupInUseException`, a backup in use (the rule of 5.2) returns `BackupInUseException` naming that table, a DELETING, DELETED or FAILED row returns `BackupNotFoundException`, a row of another store returns the ValidationException of section 4 and a row of an unhealthy configured store the one of section 7, none of them reading the store; then, last, the job permit (`LimitExceededException` when none is free). Only then read `extenddb-manifest.sha256` and `extenddb-manifest.json`, verify the sidecar against the manifest bytes, then `format_version`, the ARN and its agreement with `source_table`, and the data-file list rules (`ExtenddbManifest::from_json_bytes`). A manifest missing behind an AVAILABLE row returns `BackupNotFoundException` and logs the row for `backup sync --prune-missing`.
     2. Validate that the target backend supports every feature in `source_table` that a restore reproduces (a non-empty `vector_indexes` into a backend without vector support is the one case, section 1); refuse before creating anything otherwise. Unlike #384 (`ensure_single_part_base_key`), this path restores multi-part-key tables (as `unordered` backups, section 1): `bulk_load` keys each row and derives its index rows with the functions the backend's item write path uses (section 3), so a restored multi-part-key table is laid out as a natively created one.
     3. (Job.) Write the verified manifest and sidecar to the spool, then download every data file to the spool (PR 7's section), hashing each as it arrives and comparing with the manifest's `sha256`, `md5` and `bytes`; on a filesystem store the files are verified in place and no copy is made. A transport error during a download is retried with backoff for up to 15 minutes (a constant, not a knob in v1), the row kept RUNNING and heartbeated, before the restore fails (step 7); a checksum or size mismatch is not retried and fails the restore, naming the file, before any item has been written. Spooling everything before loading is what makes "no item is written before every checksum verifies" true for the second file as well as the first. The spool's `.verified` markers, `spool_host` and the resume rules are PR 9a's section.
    -4. Create the target table and the `restore_jobs` row (RUNNING, with the job's claim, `kind = manifest` and the store the backup row names; section 4) in one catalog transaction (PostgreSQL and SQLite), one session transaction (MongoDB) or, on Cassandra, one `INSERT ... IF NOT EXISTS` of the `tables` row, whose columns are the job row, so no crash leaves a CREATING table without a job row. If another CreateTable or restore took the name between steps 1 and 4, the request returns `TableAlreadyExistsException` (or `TableInUseException` when the winner is itself a CREATING restore target) and the job row insert rolls back with it; CreateTable keeps `ResourceInUseException`. The registration (`BackupCatalog::claim_restore`) re-checks inside its transaction, under the backup row's lock that DeleteBackup's `start_backup_delete` takes (5.2), that the backup row is still AVAILABLE and that no restore target registered from it is CREATING: a row that is no longer AVAILABLE returns `BackupNotFoundException`, a backup in use returns `BackupInUseException`, and no target is left. After the transaction commits the job checks the maintenance claim (section 4); if it is held, the job runs step 7 with reason `maintenance` and the request returns `LimitExceededException`. Otherwise return `TableDescription` with TableStatus CREATING. Overrides are applied here (below). A restore the reconciler resumes is a job and takes a `max_concurrent_jobs` permit like a new one; when none is free the row is skipped until the next cycle.
    +4. Create the target table and the `restore_jobs` row (RUNNING, with the job's claim, `kind = manifest` and the store the backup row names; section 4) in one catalog transaction (PostgreSQL and SQLite), one session transaction (MongoDB) or, on Cassandra, one `INSERT ... IF NOT EXISTS` of the `tables` row, whose columns are the job row, so no crash leaves a CREATING table without a job row. If another CreateTable or restore took the name between steps 1 and 4, the request returns `TableAlreadyExistsException` (or `TableInUseException` when the winner is itself a CREATING restore target) and the job row insert rolls back with it; CreateTable keeps `ResourceInUseException`. The registration (`BackupCatalog::claim_restore`) re-checks inside its transaction, under the backup row's lock that DeleteBackup's `start_backup_delete` takes (5.2), that the backup row is still AVAILABLE and that no restore target registered from it is CREATING: a row that is no longer AVAILABLE returns `BackupNotFoundException`, a backup in use returns `BackupInUseException`, and no target is left. After the transaction commits the job checks the maintenance claim (section 4; this release's in-catalog restore checks it before the registration, revision item 28); if it is held, the job runs step 7 with reason `maintenance` and the request returns `LimitExceededException`. Otherwise return `TableDescription` with TableStatus CREATING. Overrides are applied here (below). A restore the reconciler resumes is a job and takes a `max_concurrent_jobs` permit like a new one; when none is free the row is skipped until the next cycle.
     5. (Job, after step 3.) Per data file: read the verified file through #394's `ExtenddbManifest::reader_prehashed`, apply the per-file key-range check of section 1 when the backup is key-ordered, validate every item with PR 8's item validator with the target's limits and fail the restore on the first invalid item, then write the file through `bulk_load(target_table_id, claim, file_index, start_at, items)` (section 3), with `start_at` the recorded `items_applied` of a file a resume stopped in and 0 otherwise. Each backend writes base rows and index rows together and records progress so that a crash cannot leave an item applied twice: in one transaction with the data on PostgreSQL and SQLite, and by idempotent re-application of the one unrecorded file or batch on MongoDB and Cassandra (PR 9a's section: the batch sizes and progress records). The restored table has no streams and no TTL, so there is nothing to bypass.
     6. (Job.) After all files, and for an unordered backup after the count of section 1 matched, the job flips the target to ACTIVE conditional on the table still being CREATING as a restore target and on the job's `owner_generation`; on PostgreSQL and SQLite the table and every index of the target flip in one catalog transaction, which also marks the `restore_jobs` row DONE; on Cassandra the index rows are created ACTIVE with the table and the lightweight transaction on the `tables` row flips the table and marks the job row DONE. A Query on each GSI, and SearchVectors where the target supports it, therefore sees every restored item as soon as the target reports ACTIVE. MongoDB's sequence differs, because the existing backfill worker, not the job, promotes the indexes: the job marks the `restore_jobs` row DONE (conditioned on its generation and on RUNNING) and sets `restore_backfill_pending` in one session transaction, and the job task ends there; the worker promotes each index and then flips the table to ACTIVE, conditioned on `restore_backfill_pending` and on CREATING, not on the job's generation; and the reconciler never treats a row with `restore_backfill_pending` as a dead RUNNING job (the row is DONE). DeleteTable on a restore target is refused with `ResourceInUseException`, as #384 does and as the service does, which is why the job and the reconciler remove a target through `remove_restore_target` (step 7); there is no user-facing way to cancel a running restore. UpdateTable on a CREATING table is refused as today.
     7. Failure (verification or load): the job marks the `restore_jobs` row FAILED with the reason, conditional on its generation. Only if that write matched exactly one row does the job remove the partially created table, through `BackupCatalog::remove_restore_target` (section 3), keyed by the target's account, name and table id and conditional on the table still being a CREATING restore target, which also deletes its progress rows, never through the DeleteTable API; a write that matches zero rows ends the job with only its spool removed, because the claim holder owns the table. DescribeTable then returns ResourceNotFound. Crash mid-restore: the reconciler claims RUNNING `restore_jobs` rows whose owner is dead (section 4(e)) and resumes from the recorded progress, reading the store the job row records if it is the configured store, failing the restore and naming `backup sync --move-store` if it is not, and resuming nothing while the configured store is unhealthy; on PostgreSQL, as on MongoDB and Cassandra, once the row's `heartbeat_at` is older than `stale_job_after_secs`, the delay the prefix deletion already waits (4(e): the advisory lock is the claim's exclusion, not the liveness rule; the owner aborts at its next heartbeat, inside that window). A legacy RUNNING restore whose owner is dead is marked FAILED and its target removed, never resumed (section 3). Shutdown: a restore job stops at its next batch without writing a status, so step 7 does not run; its row stays RUNNING with its spool and its progress, for a successor once the owner is dead by its backend's rule.
    @@ -169,11 +169,11 @@ spool_path = ""            # restore download area for an S3 store; d
     
     `[backup.store]` and its `filesystem` and `s3` tables are the crate's `BackupStoreConfig` (`crates/backup-store/src/config.rs` module doc, #393: the TOML block with every key, its default and `open`'s refusals); `kind = "disabled"`, the per-backend default root, the spool and `backup purge` are PR 7's section.
     
    -On SQLite (file-backed) a missing `[backup]` section is equivalent to `kind = "filesystem"` with `path` = `backups/` beside the database file; a `validate` failure of that defaulted root is a logged warning that leaves the store unconfigured, not a startup error, so an upgrade never stops a server that never configured backups, while an explicitly configured root that fails `validate` is still a startup error. On PostgreSQL and MongoDB a missing `[backup.store]` leaves the off-host path unconfigured and startup logs that; `kind = "disabled"` leaves it unconfigured on purpose on any backend. Without a configured store, CreateBackup on every backend uses the legacy in-catalog create (`LegacyBackupReader::create`, section 3) and logs a deprecation warning naming `[backup.store]` on each call, until PR 14 removes that path; after it CreateBackup without a configured store returns a ValidationException naming `[backup.store]`. Legacy backups stay readable, listable, restorable and deletable either way. `sqlite-memory` (any `:memory:` database path) has no database file and so no default root: CreateBackup there takes the off-host path only with `[backup.store]` set and the legacy create without it. Cassandra has no `SnapshotEngine`, so it uses the legacy create even with a store configured (section 3); a configured store serves restores of off-host backups into Cassandra and nothing else there.
    +On SQLite (file-backed) a missing `[backup]` section is equivalent to `kind = "filesystem"` with `path` = `backups/` beside the database file; a `validate` failure of that defaulted root is a logged warning that leaves the store unconfigured, not a startup error, so an upgrade never stops a server that never configured backups, while an explicitly configured root that fails `validate` is still a startup error. The warning holds only while the catalog has no backup rows for the defaulted root (`read_store_ids`); once it has, the root has been used and is handled as an explicitly configured one (revision item 28). On PostgreSQL and MongoDB a missing `[backup.store]` leaves the off-host path unconfigured and startup logs that; `kind = "disabled"` leaves it unconfigured on purpose on any backend. Without a configured store, CreateBackup on every backend uses the legacy in-catalog create (`LegacyBackupReader::create`, section 3) and logs a deprecation warning naming `[backup.store]` on each call, until PR 14 removes that path; after it CreateBackup without a configured store returns a ValidationException naming `[backup.store]`. Legacy backups stay readable, listable, restorable and deletable either way. `sqlite-memory` (any `:memory:` database path) has no database file and so no default root: CreateBackup there takes the off-host path only with `[backup.store]` set and the legacy create without it. Cassandra has no `SnapshotEngine`, so it uses the legacy create even with a store configured (section 3); a configured store serves restores of off-host backups into Cassandra and nothing else there.
     
     Memory per running job, which operators size hosts from: a backup holds the upload window (`upload_part_size_bytes * max_concurrent_uploads` on an S3 store, one write buffer on a filesystem store), the read path, the gzip window and the manifest's data-file entries until the manifest is written; with the byte budget PR 8 adds to the PostgreSQL and SQLite streams (`SNAPSHOT_FETCH_BYTES`, a constant beside `DEFAULT_FETCH_BATCH`, not a `[backup]` key in v1) the read path holds at most one page, the budget plus one row of text, plus its decoded form, and neither term grows with `data_file_target_bytes` or with the table, except on `sqlite-memory`, whose open copies the whole table (section 3). A restore holds one line's decode plus one insert batch plus the parsed manifest for the whole job; neither grows with the table. Each running backup also holds one connection from the PostgreSQL data pool for its duration, and on the separate layout one catalog-pool connection while its snapshot opens (section 3); the backend admits at most half the pool (after the raise below) as open snapshots and refuses more with `LimitExceeded`; a running restore holds one data connection and has no slot of its own after PR 6; and each PostgreSQL job holds one dedicated claim connection outside the pool. `pool_size` sizes both pools, and the rule is `pool_size >= max(MIN_POOL_SIZE, 2 * max_concurrent_jobs + 10)`: twice the jobs, so the half-pool bound admits every job `max_concurrent_jobs` allows, plus ten for requests; 18 at the defaults, under the default `pool_size` of 20. It applies whether or not `[backup.store]` is configured, because legacy restores are jobs too. Startup raises a smaller `pool_size` to that value with a warning naming both numbers, as it already raises a `pool_size` under `MIN_POOL_SIZE` (10) today.
     
    -A configuration error (a missing or denied bucket, a part size, prefix or key-length problem, an insecure endpoint, a root that fails the owner-only rule) is a startup error naming the path or bucket, except for the defaulted SQLite root above, where it is a warning. A transport failure (no region, no credentials, an unreachable endpoint), and a failure of section 4's identity check or manifest probe, do not stop startup, because a store problem must not take the data plane out of rotation (section 9): the failure is logged with its cause, the store component is reported unhealthy, the operations that touch the store refuse with a ValidationException that names the store's kind and not its root (CreateBackup on the off-host path, and RestoreTableFromBackup and DeleteBackup of an off-host backup; Describe and List, legacy backups and Cassandra's legacy create are unaffected), the reconciler acts on no row of the store (its legacy duty, which touches no store, still runs) and resumes no restore, and the check runs again each reconciler cycle until it passes.
    +A configuration error that needs no request to the store (a part size, prefix or key-length problem, an insecure endpoint, a root that fails the owner-only rule or whose ancestors others can write to, a filesystem root that is or contains the database directory, a spool rule) is a startup error naming the path or bucket, except for the defaulted SQLite root above while the catalog has no rows for it, where it and a failure of the root's identity check are a warning that leaves the store unconfigured. A filesystem root's `validate` also writes a probe file, a request to the store: an I/O failure there or in resolving the root is a store condition that degrades the store (`failed validation`), as the same condition one reconciler cycle later does, not a startup error. A filesystem store's `validate` and identity check run before the listener, bounded by a timeout, so a mount that stops answering after the root is opened leaves the server starting with the store `checking` (a root that does not answer at startup holds the server before the listener, as the database directory would); an S3 store's are requests to the endpoint and run after the listener is up, in the reconciler's first cycle, with the store `checking` until the first result reports, so a stalled endpoint never holds the data plane (revision item 28). Every later check runs under the same bound (5 s for a filesystem root; 30 s for a bucket, under the SDK's per-attempt timeout so no retry outlives the check) and against the shutdown token: a store that has stopped answering costs a reconciler cycle its bound and nothing more, the cycle's other duties run, the check is recorded as a transport failure, and neither the drain at shutdown nor the process's exit waits on it (a filesystem root's abandoned call stays parked on a thread until the mount answers, no later check starts while it is, and the binary ends its runtime with a 5 s bound). A transport failure (no region, no credentials, an unreachable endpoint, a check that did not finish within its bound), a missing or denied bucket found after the listener, and a failure of section 4's identity check or manifest probe, do not stop startup, because a store problem must not take the data plane out of rotation (section 9): the failure is logged with its cause, the store component is reported unhealthy with the problem's category as its reason, the operations that touch the store refuse with a ValidationException that names the store's kind and the category and not its root (RestoreTableFromBackup and DeleteBackup of an off-host backup, and CreateBackup once it writes to the store in PR 8; Describe and List, legacy backups and this release's in-catalog CreateBackup are unaffected), the reconciler acts on no row of the store (its legacy duty, which touches no store, still runs) and resumes no restore, and the check runs again each reconciler cycle until it passes.
     
     Authority: the `[backup]` struct in `crates/config` (PR 6 adds it with the job keys; PR 7 adds `store` and its validation) and `BackupStoreConfig` in `crates/backup-store`; defaults above are the ones they encode.
     
    @@ -259,7 +259,7 @@ backup_id         14-digit zero-padded epoch millis of the snapshot instant, `-`
     account_id, table_name, backup_arn, backup_name,
     status            CREATING | AVAILABLE | DELETING | DELETED | FAILED
     store_kind        filesystem | s3
    -store_root        the configured root at creation time: the canonical directory for a filesystem store, `s3:///` for S3, with `?endpoint=` appended when an endpoint is configured (the URL without userinfo, query or fragment)
    +store_root        the configured root at creation time: the canonical directory for a filesystem store, `s3:///` for S3, with `?endpoint=` appended when the client uses an endpoint other than the SDK's default (configured, or from the SDK's environment and shared config), the URL in one spelling: lowercase scheme and host, no default port, no `/` for an empty path, no userinfo, query or fragment
     store_id          the id in the store's `extenddb-store.json` (store identity, below)
     size_bytes, item_count, created_at (= backup_creation_date_time, the id's instant), completed_at, failure_reason,
     source_table_details_json   the manifest's source_table object, verbatim
    @@ -308,7 +308,7 @@ pub trait BackupCatalog: Send + Sync {
             // the backend-internal removal of a CREATING restore target (5.3 step 7), not DeleteTable, which refuses one;
             // conditional on the table still being a CREATING restore target with this table id, so it and the ACTIVE flip
             // exclude each other; it also deletes the target's `restore_progress` rows; `false` when the table is no longer one.
    -    fn check_maintenance_claim(&self) -> BoxFuture>;   // true while one is held; jobs call it
    +    fn check_maintenance_claim(&self, stale_after: Duration) -> BoxFuture>;   // true while a live holder has it; jobs call it with stale_job_after_secs
     }
     
     pub trait JobClaim: Send + Sync {
    @@ -328,7 +328,7 @@ Documentation rows it corrects: the differences doc's `ListBackups` paging row's
     ### PR 7: configuration, the store identity file, `backup purge`
     
     Goal: the `[backup.store]` configuration with its validation, the store identity file and the health component, and `backup purge`. Its line names the interfaces and carries the acceptance:
    -- PR 7: configuration: `store` (`[backup.store]`) added to the `[backup]` struct PR 6 adds, with its validation (section 7, including the absent-section default and its warning-only validation, `kind = "disabled"`, the legacy create as the fallback without a store, the spool path rule and the spool `.lock`, the pool-size check), init template, both banners, store `validate()` at startup with the store-identity check around it (`extenddb-store.json` created with `put_if_absent`, which this PR adds to the `BackupStore` trait and both stores, then read and compared with the rows through `read_store_ids` and with the catalog's `catalog_id`; the manifest probe; both run again each reconciler cycle, and a failure marks the store unhealthy rather than stopping startup), the configuration-error versus transport-failure split, `/health` component, destroy semantics, `backup purge` deleting each configured-store row's prefix and the row. Acceptance: a bad bucket or an explicitly configured unwritable root fails startup with the name in the message; a config without `[backup]` starts and creates the default root `0700`; a config without `[backup]` on SQLite under a group-writable parent starts, logs the warning and leaves the store unconfigured, so CreateBackup there uses the legacy create with its warning, while the same root configured explicitly fails startup; an S3 store whose endpoint is unreachable at startup starts with the store component unhealthy and CreateBackup refusing, and recovers when the endpoint answers; a spool path inside or containing the root is refused, a spool path another live instance holds is refused, and on PostgreSQL an S3 store without `spool_path` fails startup; a backup created on `sqlite-memory` with `[backup.store.filesystem] path` set restores, and without `[backup.store]` CreateBackup there creates a legacy backup with the warning; a root whose `extenddb-store.json` names another catalog, carries another id than the rows, or is missing while rows exist starts with the store unhealthy (logged, `/health` unhealthy, CreateBackup, an off-host restore and DeleteBackup of an off-host backup refused naming the store's kind and not its root, the reconciler acting on no row of the store) and recovers once the file is restored; two instances of one catalog started against one empty root end with one id in the file and on the rows; two instances of one catalog with host-local filesystem roots at one path: the second, with no file, is unhealthy, and with the first's file copied to it the manifest probe marks it unhealthy; on either store a `put_if_absent` of a key that exists returns `None` and leaves the first object; purge takes the maintenance claim, waits for a CreateBackup running on a second instance, no job starts while it holds the claim, and afterwards no row of the configured store remains while `extenddb-store.json` and a prefix with no row do; PostgreSQL without `[backup.store]` creates an in-catalog backup, logs the deprecation warning naming `[backup.store]` and keeps serving legacy backups. Also `extenddb backup list` and `--failed`, printing `store_root`. Depends on PR 3.
    +- PR 7: configuration: `store` (`[backup.store]`) added to the `[backup]` struct PR 6 adds, with its validation (section 7, including the absent-section default and its warning-only validation, `kind = "disabled"`, the legacy create as the fallback without a store, the spool path rule and the spool `.lock`, the pool-size check), init template, both banners, store `validate()` with the store-identity check around it, before the listener for a filesystem store and in the reconciler's first cycle for an S3 store, the store `checking` until then (`extenddb-store.json` created with `put_if_absent`, which this PR adds to the `BackupStore` trait and both stores, then read and compared with the rows through `read_store_ids` and with the catalog's `catalog_id`; the manifest probe; both run again each reconciler cycle, and a failure marks the store unhealthy rather than stopping startup), the configuration-error versus transport-failure split, `/health` component, destroy semantics, `backup purge` deleting each configured-store row's prefix and the row. Acceptance: a bucket name `open` refuses or an explicitly configured unwritable root fails startup with the name in the message, and a missing or denied bucket makes the store unhealthy after the listener; a config without `[backup]` starts and creates the default root `0700`; a config without `[backup]` on SQLite under a group-writable parent starts, logs the warning and leaves the store unconfigured, so CreateBackup there uses the legacy create with its warning, while the same root configured explicitly fails startup; a row seeded under the defaulted SQLite root with its manifest missing starts the server with the store configured and unhealthy (`/health` degraded, `manifest probe failed`), the row's restore and delete refused naming the kind and the category, `backup purge --confirm` and `backup list` reaching the store, and the store healthy again once the manifest is back; a relative `[backup.store.filesystem] path` under daemon-mode `serve` resolves against the shell's directory; an S3 store whose endpoint is unreachable at startup serves with the store component `checking` and then unhealthy, the in-catalog CreateBackup still answering, a stalled endpoint holding the listener for no longer than its own startup and each reconciler cycle for no longer than the check's 30 s bound (the restore of a seeded row refused naming the pending check, then `unreachable`; the next cycle's check reaching the endpoint within seconds of the abandoned one), and recovers when the endpoint answers; a spool path inside or containing the root is refused, a spool path another live instance holds is refused, and on PostgreSQL an S3 store without `spool_path` fails startup; a backup created on `sqlite-memory` with `[backup.store.filesystem] path` set restores, and without `[backup.store]` CreateBackup there creates a legacy backup with the warning; a root whose `extenddb-store.json` names another catalog, carries another id than the rows, or is missing while rows exist starts with the store unhealthy (logged, `/health` unhealthy with the category as its reason, an off-host restore and DeleteBackup of an off-host backup refused naming the store's kind and the category and not its root, the in-catalog CreateBackup unaffected, the reconciler acting on no row of the store) and recovers once the file is restored, on an explicitly configured root and on the defaulted SQLite root once the catalog has rows for it; two instances of one catalog started against one empty root end with one id in the file and on the rows; two instances of one catalog with host-local filesystem roots at one path: the second, with no file, is unhealthy, and with the first's file copied to it the manifest probe marks it unhealthy; on either store a `put_if_absent` of a key that exists returns `None` and leaves the first object; purge takes the maintenance claim and heartbeats it for its whole run, waits for a CreateBackup running on a second instance (a CREATING or FAILED row with a live heartbeat), no job starts while it holds the claim (the in-catalog create and restore answer LimitExceededException), a CREATING row with a dead owner is purged with the rest, and afterwards no row of the configured store remains while `extenddb-store.json` and a prefix with no row do; PostgreSQL without `[backup.store]` creates an in-catalog backup, logs the deprecation warning naming `[backup.store]` and keeps serving legacy backups. Also `extenddb backup list` and `--failed`, printing `store_root` and `store_id`, a read beside a server (no SQLite serve lock, nothing created, an unhealthy store listed with its problem), as the purge's dry run is. Depends on PR 3.
     
     Invariants honoured: 4(c) (store identity and health, the shared-root rule), 4(f) (purge takes the maintenance claim), section 7's rules (an upgrade never stops a server that never configured backups; a configuration error stops startup, a transport or identity failure degrades the store).
     Interfaces it adds. `put_if_absent`, which PR 7 adds to the trait and both stores for the store-identity file (section 4), creates the object only if no object is under the key and returns `None`, leaving the existing object untouched, if one is: S3 sends `PutObject` with `If-None-Match: *` (a 412 is the key present; a 409 `ConditionalRequestConflict` from a concurrent conditional write is retried), and the filesystem store writes and syncs its temporary as `put` does, the temporary created with `O_CREAT | O_EXCL`, then `link`s it to the key, which fails if the key exists, so the object appears complete or not at all. `deny_unknown_fields` applies to every section; the server's `[backup]` struct holds the crate's `BackupStoreConfig` under its own key, `store`, which keeps one definition of the store shape, in the crate.
    @@ -336,11 +336,13 @@ Interfaces it adds. `put_if_absent`, which PR 7 adds to the trait and both store
         fn read_store_ids(&self, store_kind: StoreKind, store_root: &str) -> BoxFuture, StorageError>>;
             // the distinct `store_id`s recorded for a root: the identity check (section 4), before it knows an id; PR 7
         // Maintenance claim (section 4). PR 7 for purge, PR 10 for sync.
    -    fn maintenance_claim(&self) -> BoxFuture, StorageError>>;
    +    fn maintenance_claim(&self, holder: &str, stale_after: Duration) -> BoxFuture, StorageError>>;
    +        // `holder` labels the command and host for the operator who finds the claim held; `MaintenanceClaim` is
    +        // the handle's generation, heartbeat and release (the claim has no status machine, so it is not a `JobClaim`)
     ```
     The identity file (4(c) states what makes a store unhealthy): On a root with no `extenddb-store.json` and a catalog with no rows for it, the server creates the file with `BackupStore::put_if_absent` (section 2: `If-None-Match: *` on S3, a temporary created `O_CREAT | O_EXCL` and `link`ed to the key on the filesystem store), then reads the file and uses the id it holds. Of two instances of one catalog starting against one empty root, one creates the file and the other's create finds it present, and both use the winner's id. An S3-compatible server that ignores `If-None-Match` on `PutObject` makes the create a plain put, last writer wins: startup's re-read then catches a race lost before it and marks the store unhealthy, naming both ids, and a race lost after it is caught by the next check (a sync, a startup or a reconciler cycle), which finds rows carrying an id the file does not. Startup runs the store's `validate` (its checks are #393's README and `validate` doc) and, on both store kinds, the identity check and the manifest probe of section 4, which run again each reconciler cycle; a root on a filesystem that reports a permissive mode (`0777` from a mount without Unix permission semantics) fails `validate` with the path, and one that reports `0700` while enforcing nothing passes, the case the store cannot observe and the deployment guide names.
     The spool: The spool: restores from an S3 store download to `spool_path`, created `0700` with files `0600`. The default is per backend: on file-backed SQLite it is `backup-spool/` beside the database file; on PostgreSQL, MongoDB, Cassandra and `sqlite-memory` with an S3 store `spool_path` is required and startup refuses without it (a filesystem store needs no spool). Startup refuses a `spool_path` that contains or is contained in the filesystem store root (the store would list and delete it, and sync would read its first component as an account id) and one that is or contains the database file's directory, and the reconciler's spool sweep removes only entries whose names match the job-entry pattern (`restore-
    `), never other files under `spool_path`. `spool_host` is `:`, and startup refuses a `spool_path` whose `.lock` another live instance holds (5.3 step 3). Disk cost of a restore: S3 store, the compressed backup size at `spool_path` while the restore runs; filesystem store, none for the restore, because files are verified in place (section 5.3 step 3). The filesystem store itself holds every backup's compressed size under its root, beside the database on the SQLite default, kept through `extenddb destroy` (below); that is the disk statement a SQLite operator sizes from. -Destroy and purge: `extenddb destroy` removes the catalog and data databases and leaves the backup store untouched: it is the copy that survives destroy. A separate `extenddb backup purge --confirm` deletes the prefix of each of the configured store's rows (`BackupArn::store_prefix`), never a whole account prefix, so a prefix with no row (another catalog's, or a copy) survives, and deletes those catalog rows outright, leaving no tombstones (a row over no bytes would be a backup Describe reports and Restore cannot read); it leaves `extenddb-store.json`, which is under no `/` prefix, so the root keeps its identity, and the `exports/` component (5.4). It takes the maintenance claim of section 4 and refuses if a backup row is still CREATING or DELETING or a `restore_jobs` row RUNNING when the claim's wait ends, so no job on any instance starts between the check and the deletion (the job permits are per instance and would exclude nothing elsewhere), prints what it will delete before deleting, and is audit-logged (section 8). It lands in PR 7. +Destroy and purge: `extenddb destroy` removes the catalog and data databases and leaves the backup store untouched: it is the copy that survives destroy. A separate `extenddb backup purge --confirm` deletes the prefix of each of the configured store's rows (`BackupArn::store_prefix`), never a whole account prefix, so a prefix with no row (another catalog's, or a copy) survives, and deletes those catalog rows outright, leaving no tombstones (a row over no bytes would be a backup Describe reports and Restore cannot read); it leaves `extenddb-store.json`, which is under no `/` prefix, so the root keeps its identity, and the `exports/` component (5.4). It takes the maintenance claim of section 4 and refuses if a CREATING or FAILED backup row still has a live owner or a `restore_jobs` row is still RUNNING when the claim's wait ends (4(f)), so no job on any instance starts between the check and the deletion (the job permits are per instance and would exclude nothing elsewhere); a CREATING row whose owner is dead is deleted with the rest, a DELETING row is finished, and a row a DeleteBackup moves or removes between the command's read and its turn is finished and counted as deleted; it prints what it will delete before deleting, and is audit-logged (section 8). It lands in PR 7. State transitions it owns: the store's health (healthy, unhealthy for each of 4(c)'s conditions, and back once the check passes); purge deletes rows outright, leaving no tombstone. Documentation rows it corrects: limits doc backup rows (the knobs); upgrade manual: CreateBackup without `[backup.store]` keeps creating in-catalog backups with a deprecation warning until PR 14, so a store is to be configured before that release, and `kind = "disabled"` is the opt-out (section 7); the rule the SQLite default root must meet and what the warning means when it does not (section 7); instances that share a catalog must share one store root (section 4). @@ -402,13 +404,14 @@ Interfaces it adds: fn insert_backup_row(&self, row: &BackupRow) -> BoxFuture>; // sync's unowned insert-if-absent fn list_backup_rows_by_root(&self, store_kind: StoreKind, store_root: &str, status: Option) -> BoxStream>; // a root's rows whatever their `store_id`: `--forget-store` and `--move-store` (section 4), which get a root and no id; PR 10 - // on Cassandra this and `read_store_ids` read `backup_index_by_status` one partition per status for the root, - // then each row's `backups_by_arn` row, since the index carries neither `store_kind` nor `store_id` + // on Cassandra this and `read_store_ids` read `backup_index_by_status` one partition per status for the root; + // V006's entry carries `store_kind` and `store_id`, so `read_store_ids` needs no second read, and this + // listing reads each row's `backups_by_arn` row for the rest of the row fn rewrite_store(&self, keys: &[BackupKey], store: &StoreIdentity) -> BoxFuture>; // `--move-store`, PR 10 ``` `backup sync` and its options, each with its contract: Sync lists the root (every object, since the trait has no delimiter listing; only the distinct `/
    //` prefixes are kept, the reserved `changelog` and `exports/` components skipped), checks each prefix against the catalog as the listing reaches it, reads the manifest and sidecar of the prefixes that have no row, and inserts an AVAILABLE row for each verified manifest whose data files the listing contains, every `data_files` key present with its `bytes`, with an insert-if-absent (`BackupCatalog::insert_backup_row`): a CreateBackup whose row landed between the check and the insert keeps its row and sync skips the prefix. Before adopting, sync parses the manifest's `backup_arn` with the one `BackupArn` type in `extenddb-core` and requires the parsed account, table and backup id to equal the listed path components and the `source_table` identity; a mismatch is logged and skipped. Foreign region or partition in an adopted ARN is accepted and logged; requests check only the account. `--move-store ` re-points the rows of that root to the configured store. It first resolves that root's CREATING and DELETING rows itself, against the configured store: a CREATING row becomes AVAILABLE if a verified manifest is there and FAILED otherwise, and a DELETING row whose prefix there holds no manifest has its deletion finished in the configured store. For each AVAILABLE row it compares the configured store's listing of the row's prefix with the manifest's `data_files` (every key present, with its `bytes`) as well as checking the verified manifest, and stops on a row whose copy is incomplete, naming it; `--forget-backup ` deletes one row's catalog entry so the move can go ahead for the rest; it runs under the maintenance claim like the other commands, refuses a CREATING row with a live owner (the job's next conditional write would match nothing and its manifest would be adopted by the next sync) and a backup in use, one from which a restore target is CREATING, and for a legacy ARN, which has no new-format row, says so and does nothing. One rule covers the three tombstone states: the move refuses to carry a DELETING, DELETED or FAILED row whose prefix in the configured store holds a verified manifest, and names it (that prefix is a backup the operator copied before or during its deletion, not a leftover; `--forget-backup` on the row is the way out). Then it rewrites `store_kind`, `store_root` and `store_id`; the reconciler then cleans the leftovers of the moved tombstones in the configured store. `--move-store` and `--forget-store` match `store_root` exactly as stored, the `?endpoint=` suffix of an S3 root included. A root whose file names another catalog stays unhealthy until an operator adopts it with `backup sync --adopt-store`, which takes the maintenance claim, rewrites the file's owner to this catalog and then syncs, recording the file's store id on the rows it adopts; the previous owner finds the store unhealthy at its next check and stops acting on it. On `sqlite-memory` the catalog, and with it the `catalog_id`, is new at every start, so the server runs a plain `backup sync` itself at startup when a store is configured, which adopts a root whose file names a `sqlite-memory` catalog as `--adopt-store` does, rewriting the file's owner to the new `catalog_id` before it syncs, so the root has one owner at a time there too; a root owned by any other catalog is unhealthy there as anywhere, and the backup CLI commands do not exist there (no file to open), so that startup sync is the only catalog-loss recovery there. `migrate-legacy`: PR 10 adds `extenddb backup migrate-legacy |--all`, which streams a legacy backup through the frozen reader's item stream, checking each item with the backup job's item validator (5.1 step 4), into `DataFileEncoder` and the configured store under the legacy backup id, so the ARN survives, with a manifest whose `snapshot.kind` says it is a copy of an in-catalog backup (`legacy_in_catalog`, no marker). Before it inserts the AVAILABLE row or deletes anything it re-reads the manifest it wrote through `ExtenddbManifest::from_json_bytes` and checks that the manifest's item count equals the legacy backup's; only then does it insert the row and delete the legacy rows through the reader's `delete`. It refuses a backup, keeping its legacy rows, deleting the prefix it wrote and naming the reason, when the id carries no instant (the 13-digit and bare shapes, which the manifest parser refuses, section 4), when the validator refuses an item (a value #618 let in, which main's legacy restore copies as stored), and when it cannot state the definition (a pre-0.0.4 SQLite backup recorded no vector state, so `vector_indexes` would be a guess); the way out for such a backup is to restore it through the legacy reader and take a new backup of the restored table, which has a new ARN (section 10). -The maintenance claim per backend and the SQLite serve lock (4(f) states the rules): On PostgreSQL the claim is an advisory lock keyed `hashtextextended('backup-maintenance', RESTORE_LOCK_SEED)`, held exclusively on the command's dedicated connection; a job tests it with `pg_try_advisory_lock_shared` and releases the shared lock at once. On MongoDB it is a document in a `backup_maintenance` collection, claimed and heartbeated like a job row (5.1 step 3) and held while its heartbeat is younger than `stale_job_after_secs`; on Cassandra a `backup_maintenance` row claimed by lightweight transaction with a coordinator-clock heartbeat, read by jobs at `SERIAL`. On SQLite every backup CLI command that writes the catalog takes the database's serve lock, as `destroy` and `migrate` do, so the server must be stopped first and no other claim is needed; with the serve lock held every job owner is dead, so before the wait such a command runs the reconciler's dead-owner resolutions itself (a CREATING backup row to AVAILABLE or FAILED, a RUNNING restore to FAILED with its target removed, a DELETING row finished), and a job the stop interrupted never makes it wait or refuse; `backup list` is a read and runs beside a server. +The maintenance claim per backend and the SQLite serve lock (4(f) states the rules): On PostgreSQL the claim is an advisory lock keyed `hashtextextended('backup_maintenance', RESTORE_LOCK_SEED)` (`MAINTENANCE_LOCK_KEY`), held exclusively on the command's dedicated connection, with the `backup_maintenance` row carrying the holder, the generation and the heartbeat; a job tests the lock with `pg_try_advisory_xact_lock_shared` in one autocommit statement, so the shared lock ends with the statement, and then reads the row's heartbeat age, so a holder whose session ended is still seen held until its heartbeat is older than `stale_job_after_secs`. On MongoDB it is a document in a `backup_maintenance` collection, claimed and heartbeated like a job row (5.1 step 3) and held while its heartbeat is younger than `stale_job_after_secs`; on Cassandra a `backup_maintenance` row claimed by lightweight transaction with a coordinator-clock heartbeat, read by jobs at `LOCAL_SERIAL`. On SQLite every backup CLI command that writes the catalog takes the database's serve lock, as `destroy` and `migrate` do, so the server must be stopped first and no other claim is needed; with the serve lock held every job owner is dead, so before the wait such a command runs the reconciler's dead-owner resolutions itself (a CREATING backup row to AVAILABLE or FAILED, a RUNNING restore to FAILED with its target removed, a DELETING row finished) and then waits and removes CREATING rows with a zero stale window, so a job the stop interrupted never makes it wait or refuse whatever its heartbeat says (a RUNNING `manifest` restore, which PR 9a introduces, is the exception revision item 28 states), and a server starting under the serve lock removes a maintenance claim a stopped command left; `backup list` is a read and runs beside a server. State transitions it owns: a prefix to an AVAILABLE row (adoption); CREATING or DELETING rows of a moved root resolved by `--move-store`; AVAILABLE to DELETING by `--prune-missing`; rows deleted outright by `--forget-store` and `--forget-backup`; a FAILED restore's CREATING target removed by `remove-target`; legacy rows replaced by an AVAILABLE row by `migrate-legacy`. Documentation rows it corrects: upgrade manual: on SQLite the backup CLI commands that write the catalog take the serve lock, so the server is stopped for `backup sync` and its options and for `purge` (section 4); legacy backups are carried forward with `backup migrate-legacy` (above), and one it refuses (an id with no instant, an item the validator refuses, a pre-0.0.4 SQLite definition) is carried forward by restoring it and taking a new backup, which has a new ARN; a store owned by another catalog is adopted only with `backup sync --adopt-store`. @@ -529,3 +532,4 @@ Changed after a review of the draft against main `9f43d03` and PRs #384, #335, # 25. (2026-10-29, the split.) The document is cut from 279,584 bytes to 187,722 bytes: sections 1, 2 and 3 now point at the companions' documentation and keep only the rules later PRs build on (the format: `crates/core/src/backup_format` module docs and `crates/core/testdata/extenddb-manifest/README.md`, #394; the store: `crates/backup-store/README.md` and its `config.rs`, `key.rs` and `lib.rs` docs, #393; the snapshot: `crates/storage/src/snapshot.rs`, the four `snapshot_engine.rs` module docs and `docs/troubleshooting.md`, #395); the companions gain the facts only this document stated (the filesystem `list`'s whole-root `Vec` and the empty-prefix IAM policy in #393's README and `lib.rs`; the pre-#361 sort cost and the reason the fetch size is a constant in #395's `snapshot_engine.rs` and `snapshot.rs`, whose module docs already carried the half-pool measurement; a direction sentence in each companion's module doc saying that this overview states the rules the code's callers honour and the crate docs are the record of what the code does); the dependency-change facts of section 2 (lock entries, licences, pins) and the vacuum and WAL growth figures are dropped, the first being the PR description's record and the second the backup manual's. Sections 4 and 5 keep the invariants, the state machines and the per-operation contracts, by lettered part and numbered step; the schema, the per-backend mechanics, the reconciler's duty list, the sync procedure, the claim per backend, the page query and the memory arithmetic moved into a section per later PR under Phasing (PR 1, 6, 7, 8, 9a, 9b, 10, 11, 13, 14), each with its goal, the invariants it honours by reference, the interfaces it adds with their row keys, the transitions it owns and its acceptance list kept whole; section 10 is a table of which document and row each PR changes, the rows themselves in the owning PR's section; the PR 3, 4 and 5 lines go, their PR descriptions and docs being the record of their deliverables. Heading numbers 1 to 10 and the step numbers of 5.1 and 5.3 are unchanged, so items 1 to 24 still resolve; the three `version-literal-ok` markers stay in section 4(i), PR 1's and PR 6's sections. The rewordings the split needed, each keeping its sentence's meaning, are listed in the split's report (`.review/reviews/r8/DONE-SPLIT.md`). 26. (2026-10-30, PR 6 as implemented on #666's registry.) PR 6's migration is id 5 on PostgreSQL and SQLite (one `MigrationId` list for both, ids 1 to 4 taken), and the Cassandra file is V006 and the MongoDB step `002_backup_catalog_index`, both writing `0.0.5` so that one number names one change on every backend; the literals in 4(i) and PR 6's section are updated, PR 1's line keeps its own numbers and renumbers if it lands after this. 27. (2026-10-30, PR 6.) The FAILED-target duty lands in PR 6: `list_dead_restore_jobs` also returns a FAILED `restore_jobs` row whose target is still CREATING once the row is stale, and the reconciler removes the target without a second FAILED write, which closes the state the job's or the reconciler's failure path leaves when the FAILED write lands and the removal does not (4(e), PR 6, PR 8). PostgreSQL's dead rule is the heartbeat alone, as on MongoDB and Cassandra: the advisory lock is the claim's exclusion, not the liveness test, and the `IS NULL` arm item 21 named does not exist, the column being NOT NULL (4(e), 5.3 step 7, PR 6). The stale floor is computed from the period the jobs run at, `max(heartbeat_interval_secs, JOB_HEARTBEAT_INTERVAL)`, the one constant every backend's heartbeat loop reads until the knob reaches the jobs, and `max_concurrent_jobs` has an upper bound (5.1 step 4, section 7, PR 6). SQLite's `backup_server_slot_count` is kept for snapshot opens until PR 8, the legacy create trying it without waiting and answering the one `LimitExceededException` text (PR 6). The engine's `BackupArnSpellings` carries an optional parsed ARN per spelling, so a legacy backup of a name the store-prefix rules refuse (dot-only, over 255 bytes) still describes, restores and deletes through the legacy reader (4(d), PR 6). DeleteBackup over a row runs the in-use check before the other-store check, as a restore does, builds the description before the DELETING write, and a row whose source table the build cannot read is left out of ListBackups rather than failing the listing (5.2, PR 6). The acceptance cell running the two orderings over a `backup_index` row is deferred to PR 9a, with the row lock observed by a two-transaction test of `start_backup_delete` in PR 6 (PR 6). On Cassandra an index row's list rows exist for as long as the row does, whatever its status, readers re-reading the authoritative row; the legacy create keeps its publish-time rule (PR 6). The reconciler records the worker metrics of section 9 (PR 6). On Cassandra the migration runner writes `catalog_version` after the last file's statements, its backfill step and its `schema_history` row, so a run that fails in the step leaves the previous version and `migrate --yes` runs the file again; a Cassandra job stops after three heartbeat errors in a row, as after a heartbeat that does not apply, and the fence the reconciler puts on a dead create's row takes a fresh `owner_generation` (4(e), PR 6). MongoDB's RestoreTableFromBackup waits a bounded time for the registration and answers `InternalServerError` past it while the registration goes on; what the request leaves is stated in 5.3 and the differences document (5.3, PR 6). CreateBackup checks `server.region` before forming a backup ARN and refuses, naming the setting, under a region outside `[a-z0-9-]`, where main created a backup no operation could then address; the store-prefix rules on the name are applied before the ARN is formed so a name over 255 bytes is refused naming the limit whatever its length (5.1 step 1, PR 6). The `LimitExceededException` text names the bound that was met beside its number (`max_concurrent_jobs`, SQLite's snapshot slots, or PostgreSQL's connection slots for the claim connection), one implementation (`limit_exceeded_message`) every refusal binds (5.1 step 1, section 7, PR 6). Section 7 states, where it bounds snapshots by half the pool, that the pool is the raised one (section 7, PR 6). Item 21's `IS NULL` arm is withdrawn by this item alone; item 21 keeps its wording, as item 22 requires. On Cassandra, where the removal is a claim and then a drop, the FAILED-target duty also lists a FAILED record whose target is DELETING with its registration, and `remove_restore_target` resumes the removal from the drop (4(e), PR 6). On MongoDB the in-use check of a restore's registration and of DeleteBackup reads at the transaction read concern, majority-committed data, while DescribeTable reads at `local`, so a request issued within the majority-commit lag after DescribeTable first shows the target ACTIVE can still answer `BackupInUseException` and succeeds when retried; the differences document's concurrent-restores row states it with the measured lag (5.3, PR 6). The rule that a job performs no further operation after a write it cannot confirm is scoped to the conditional write itself: a legacy restore whose claim connection (the heartbeat's) errors runs 5.3 step 7, the conditional FAILED write on a pooled connection and the removal only when that matched, so on PostgreSQL the target is removed at the next heartbeat rather than by the reconciler once the record is stale (4(e), 5.3 step 7, PR 6). On MongoDB `remove_restore_target` deletes the target's catalog document first and returns the target removed once that is gone: a collection drop that fails after it is logged naming the collections and leaves them for `backup sync` (PR 10), there being no record for a later reconciler cycle to retry from (4(e), PR 6). +28. (2026-10-30, PR 7 as implemented.) A store problem never gates the in-catalog create: with a configured store that is unhealthy, CreateBackup writes the in-catalog format as it does with no store, and only the store-backed create of PR 8 is refused by `require_healthy_store`; a defaulted SQLite root whose identity check fails (a catalog re-created by `destroy` and `init` under the old root's `extenddb-store.json`) is the same warning as a failed validation, leaving the store unconfigured, and `destroy` and the admin guide say the file must go with the catalog (section 7, PR 7). An S3 store's `validate` and identity check run after the listener is up, in the reconciler's first cycle, with the store `checking` until the first result reports; a filesystem store's run before the listener; the configuration errors that need no request still stop startup; a `validate` refusal of an S3 bucket after the listener makes the store unhealthy (`failed validation`) rather than stopping startup (section 7, 9, PR 7). `/health`'s `reason` is the problem's category alone; the ids and ARNs go to the operator's log line (section 9, PR 7). The identity file is read up to 64 KiB. The S3 root label is built from the endpoint the client uses, in one spelling. A check that cannot read the catalog leaves the store's health unchanged. An existing spool must be owner-only. The maintenance claim is heartbeated on a timer from the take to the release, and PostgreSQL's `check_maintenance_claim` tests the advisory lock before the heartbeat's age (4(f), PR 7). The jobs' side of the claim is implemented in the engine for the in-catalog create and restore, before the permit and, since those paths commit their row inside the backend, before the row exists; the ordering 5.1 step 2 states applies to the store-backed job PR 8 adds (5.1, 5.3, PR 7). `backup list` opens the SQLite catalog without the serve lock, reads the identity file without creating it and runs on an unhealthy store; on SQLite `backup purge` runs the reconciler's duties with a zero window before its wait; a CREATING row with a dead owner is purged as any row, through `remove_dead_creating_backup_row`, a `BackupCatalog` method of PR 7; a FAILED row with a young heartbeat is waited on; the dry run exits 0 (PR 7, PR 10). Cassandra's claim reads run at `LOCAL_SERIAL` and an insert whose answer was lost is settled by a re-read of the generation (PR 7). The PR 10 comment on `backup_index_by_status` is corrected: V006's entry carries `store_kind` and `store_id`. The body sentences that paraphrased the earlier rules are edited with this item (section 7's configuration-error paragraph, 4(f) and 5.1 step 2 and 5.3 step 4's claim check, PR 6's `store_root` row, PR 7's interfaces and acceptance), and the filesystem store's startup checks are bounded by a timeout that leaves the store `checking` when a mount stops answering after the root is opened (a root that does not answer at startup holds the server before the listener, as the database directory would). Also: the PostgreSQL check takes its shared lock at transaction scope so a cancelled request leaves no lock on the pool; a dropped `HeldClaim` stops its timer; the purge's dry run is a read like `list`; a CREATING row is removed before its prefix; an empty `filesystem.path` is left for `open` to refuse; the spool meets the root's whole rule, parents included; `backup list` reads the catalog's status without the serve lock. The defaulted SQLite root's warning holds only while the catalog has no backup rows for the root (`read_store_ids(Filesystem, root)`: the upgrade, fresh and re-init cases); with rows for it the root has been used and is handled as an explicitly configured one, so its backups never vanish behind a warning: a failure of its identity check or manifest probe is an unhealthy store (reported, its rows' restores and deletes refused, rechecked each cycle), and a refusal of its `open` or `validate` a startup error naming the root (section 7, PR 7's acceptance). The server's `configure` carries the reason for an absent store out (`NotConfigured`), and `backup list` and `backup purge` print it, so a defaulted root that failed its check is no longer reported as "[backup.store] is absent or disabled". The store's check runs under a per-kind bound in every reconciler cycle (`check_configured_store`: 5 s for a filesystem root, the startup bound; 30 s for a bucket, under the SDK's 60 s per-attempt timeout so a timed-out check is dropped before the SDK retries its request), recording a transport failure naming the bound, and the cycle selects the check against the shutdown token; an unbounded check let a stalled S3 endpoint hold each cycle for the SDK's whole retry budget and a stalled mount hold it for good, with shutdown hitting the drain timeout (section 7). A filesystem root's `validate` reports no transport class; an I/O failure of its probe write or of resolving the root at startup degrades the store (`failed validation`) as the same condition one cycle later does, while the owner-only and existence refusals stay configuration errors (section 7). A transport failure's operator line carries the store error's whole cause chain (the SDK's top link is a bare class). Every store error of a check that is neither a transport failure nor a denial is `failed validation`, the category `validate` gives it, so a bucket deleted under a running server reads as a bucket missing at startup. A filesystem root that is or contains the database directory is refused in `resolve_store`, beside the spool rule (section 7). A filesystem store's synchronous root checks run on the blocking pool so the startup bound fires whichever call stalls. The checking store's refusal says its first check is pending; `/health`'s reason list carries `catalog unreadable`; `record_check` reports whether the category changed. Purge's wait set is CREATING and FAILED rows with a live heartbeat and RUNNING restores, DELETING rows being finished rather than waited on, and its refusal rule is that set at the wait's end; 4(f), the PR 7 "Destroy and purge" paragraph and the PR 10 claim paragraph (`MAINTENANCE_LOCK_KEY`, `pg_try_advisory_xact_lock_shared` then the heartbeat age, `LOCAL_SERIAL`) and the PR 6 and PR 7 signatures (`check_maintenance_claim(stale_after)`, `maintenance_claim(holder, stale_after) -> Box`) are edited to the code with this item. A row a DeleteBackup moves to DELETING or DELETED, or removes, between purge's read and its turn is finished and counted as deleted, not left (the deletion is one idempotent sequence on both sides). On SQLite purge waits and removes CREATING rows with a zero stale window (under the serve lock every owner is dead, so a young heartbeat proves nothing), and the server's startup under the serve lock removes a maintenance claim a stopped command left, so a killed purge costs no stale window; on every backend purge releases the claim on `SIGINT` or `SIGTERM`, at once during the wait and before the next row during the deletion. The wait's three failures (jobs still running at the limit, the claim lost, the catalog unreadable) and the signal are reported each as itself. PostgreSQL's factory starts the GSI queue workers only for a server, not for a command reading beside one. Purge's restore part of the wait set is the RUNNING `restore_jobs` records alone: the dead-job listing it reads with a zero window also returns the FAILED records whose target is still CREATING (the FAILED-target duty's, 4(e)), which are the reconciler's to finish and no job runs under, so they are not waited on; the wait finishes at once on a stopped deployment that holds one. On SQLite the zero-window pass resolves the kinds this release writes; a RUNNING `restore_jobs` record of kind `manifest`, which PR 9a introduces, is left for the reconciler that resumes it and purge waits on it and refuses naming the table, the one state the PR 10 paragraph's "never makes it wait or refuse" does not cover. The reconciler finishes the configured store's DELETING rows while the maintenance claim is held, as 4(f) lists. The binary ends its tokio runtime with a bounded shutdown (5 s) after `serve` and after every command: a filesystem store's check abandoned at its bound leaves its call parked on a blocking thread until the mount answers, and a plain drop of the runtime joined that thread, so a server whose root had stopped answering drained its workers at SIGTERM and then stayed alive until SIGKILL. Each call of a store check runs as a task of its own that an abandoned check does not take with it, and the store records that a call is in flight; no check starts while one is (the store is recorded `unreachable` again, with the reason), so a mount that stays stalled parks one thread rather than one per cycle. A command's identity read (`backup list`, the dry run) runs under the store's check bound as the server's check does, where the unbounded read waited out the SDK's three attempts of 60 s on an endpoint that accepts and never answers. An empty `filesystem.path` is compared with nothing in `resolve_store` (compared as written it was the working directory, refused by name as the database directory when the server starts there) and left for `open` to refuse. A root that refuses `link(2)` with `EPERM` is reported `failed validation` (the README and the deployment guide said `unreachable`). `tests/test_backup_store_config.py` is excluded from `devtools/run-tests`'s pytest run, as its docstring says; a catalog whose rows cannot be read keeps a defaulted root configured, and that arm is tested. diff --git a/docs/dynamodb-limits.md b/docs/dynamodb-limits.md index 0044bd5de..eecff2545 100755 --- a/docs/dynamodb-limits.md +++ b/docs/dynamodb-limits.md @@ -128,10 +128,11 @@ Source: [AWS DynamoDB Service Quotas](https://docs.aws.amazon.com/amazondynamodb | Limit | DynamoDB Value | Status | Notes | |-------|---------------|--------|-------| -| Concurrent restores | 50 | Partial | Every backend: one server process runs at most `[backup] max_concurrent_jobs` (default 4, at most 1024) backup and restore jobs at once, `CreateBackup` and `RestoreTableFromBackup` together and not per account; a request over the bound is refused at once with `LimitExceededException` ("Too many backup and restore jobs are running on this server (at most N at once, the [backup] max_concurrent_jobs bound); retry later") and creates nothing, with no wait (storage-level and engine tests; measured on PostgreSQL with `max_concurrent_jobs = 1`: a running restore made a CreateBackup and a second restore refuse in under 60 ms). On PostgreSQL `pool_size` and `catalog_pool_size` are raised to at least `2 * max_concurrent_jobs + 10` so every admitted job has a connection. SQLite also bounds the snapshot slots `CreateBackup` and table snapshots (#395) share, a count derived from `[storage.sqlite] pool_size` (the formula is stated once, on that setting's documentation in `SqliteConfig`; 4 for the default pool) so half the pool stays free for ordinary requests; a `CreateBackup` that finds no slot free is refused at once with the same `LimitExceededException`, its parenthesis naming that bound and its count in place of `max_concurrent_jobs` ("at most N at once, the [storage.sqlite] pool_size snapshot-slot bound"); a PostgreSQL job whose dedicated claim connection the database server refuses is refused with the same exception naming the server's connection slots ("the database server has no connection slot left for the job"), with no count. A MongoDB restore request that is not registered within 60 seconds answers an internal error while the restore continues | +| Concurrent restores | 50 | Partial | Every backend: one server process runs at most `[backup] max_concurrent_jobs` (default 4, at most 1024) backup and restore jobs at once, `CreateBackup` and `RestoreTableFromBackup` together and not per account; a request over the bound is refused at once with `LimitExceededException` ("Too many backup and restore jobs are running on this server (at most N at once, the [backup] max_concurrent_jobs bound); retry later") and creates nothing, with no wait (storage-level and engine tests; measured on PostgreSQL with `max_concurrent_jobs = 1`: a running restore made a CreateBackup and a second restore refuse in under 60 ms). On PostgreSQL `pool_size` and `catalog_pool_size` are raised to at least `2 * max_concurrent_jobs + 10` so every admitted job has a connection. SQLite also bounds the snapshot slots `CreateBackup` and table snapshots (#395) share, a count derived from `[storage.sqlite] pool_size` (the formula is stated once, on that setting's documentation in `SqliteConfig`; 4 for the default pool) so half the pool stays free for ordinary requests; a `CreateBackup` that finds no slot free is refused at once with the same `LimitExceededException`, its parenthesis naming that bound and its count in place of `max_concurrent_jobs` ("at most N at once, the [storage.sqlite] pool_size snapshot-slot bound"); a PostgreSQL job whose dedicated claim connection the database server refuses is refused with the same exception naming the server's connection slots ("the database server has no connection slot left for the job"), with no count; the same error while `extenddb backup purge` holds the catalog's maintenance claim names the command, not a bound. A MongoDB restore request that is not registered within 60 seconds answers an internal error while the restore continues | | `BackupName` and `TableName` on `CreateBackup`, `TableName` on `ListBackups` | Length 3 to 255, `[a-zA-Z0-9_.-]+` | Enforced | The service model's constraints, with its texts ("1 validation error detected: Value '' at 'backupName' failed to satisfy constraint: ...", "2 validation errors detected: ...; ..." when length and pattern both fail). Length is counted in characters. `BackupName` takes the model's 3 to 255; a `TableName` member takes the deployment's `[limits]` `min_table_name_length` and `max_table_name_length` (defaults 3 and 255), as `CreateTable` does, so a table the deployment can create can be listed, and can be backed up unless no backup ARN can carry its name: a name over 255 bytes (with `max_table_name_length` raised) or made only of dots can be created but `CreateBackup` refuses it with `ValidationException` "TableName cannot be backed up: a backup ARN cannot be formed for it: invalid backup ARN: table name of N bytes is longer than 255" or "... invalid backup ARN: table name \"...\" is only dots" (the service backs up any table it can create; see the differences document's backup creation row). Both members may also be the table's ARN. The value is echoed in the message as sent | | `BackupArn` | 37 to 1024 characters | Maximum enforced; minimum implied | Surrounding ASCII whitespace is trimmed first, then over 1024 bytes, not of the backup ARN shape, a control or non-ASCII character, or a region outside `[a-z0-9-]`, an account that is not digits, or a table name outside `[a-zA-Z0-9_.-]` is `ValidationException` "Invalid Request: BackupArn is not valid"; the right shape with an id that is not a backup id is "Invalid Backup ARN"; a digit-only account that is not the caller's is `AccessDeniedException`. No minimum is checked: the shortest string of the right shape is longer than 37 characters. Another region of the caller's account resolves (the server's region is tried first, then the ARN as written). `ListBackups`'s `ExclusiveStartBackupArn` must also match the account, the region, and `TableName` when given | | `ListBackups` page size | 100 | Enforced | `Limit` 1 to 100, default 100; more rows come back through `LastEvaluatedBackupArn` and `ExclusiveStartBackupArn`. Each request reads every backup of the account (or of the table, with `TableName`) and pages in memory, so walking N backups reads about N²/100 rows. `TimeRangeLowerBound`, `TimeRangeUpperBound`, and `BackupType` filter the list before paging; a bound before the epoch, or a lower bound above the upper bound ("Time range lower bound must be less than or equal to upper bound"), is `ValidationException` | +| Backup store | A backup lives in the service's own storage | N/A | `[backup.store]` selects where ExtendDB's backups will live from a later release's CreateBackup on (this release's writes the in-catalog format and the store serves restores and deletes of the backups it holds): a filesystem root (`backups/` beside a SQLite database file when the section is absent), an S3 bucket and prefix, or `disabled`. The knobs are `[backup]`: `data_file_target_bytes` 64 MiB (minimum 32 MiB), `heartbeat_interval_secs` 15, `stale_job_after_secs` 300 (at least three heartbeats), `failed_row_retention_secs` 7 days, `reconcile_interval_secs` 60, `max_concurrent_jobs` 4 per instance, `spool_path` (required with an S3 store except beside a SQLite database file). A filesystem store holds every backup's compressed size under its root, beside the database on the SQLite default, and `extenddb destroy` leaves it; a restore from an S3 store holds the compressed backup size at `spool_path` while it runs. Instances that share a catalog must share one store root; an unhealthy store refuses the restore and delete of store-backed backups (CreateBackup, which writes the in-catalog format in this release, is not refused) with `ValidationException` naming the store's kind, and the server keeps serving (`/health` reports `degraded`). See the admin guide's `[backup.store]` | | Backup contents | Key schema, attribute definitions, items, indexes, billing mode, provisioned throughput, on-demand throughput (table and GSI level), SSE. Table class is not carried (a restore is `STANDARD`) | Partial | ExtendDB restores the same set, except a GSI's on-demand throughput (it records none per index); table class is not restored, as on the service. Streams, TTL, tags, and deletion protection are not restored, as on the service. Backups are stored inside the database the server uses; take database-level backups for off-host copies | | Point-in-time recovery | 35-day window | N/A | `RestoreTableToPointInTime` is refused. `UpdateContinuousBackups` accepts enabling point-in-time recovery and reports a 35-day restorable window, but nothing is recorded (see differences-from-dynamodb.md) | | `RestoreTableFromBackup` overrides | `BillingModeOverride`, `ProvisionedThroughputOverride`, `GlobalSecondaryIndexOverride`, `LocalSecondaryIndexOverride`, `SSESpecificationOverride`, `OnDemandThroughputOverride`, `VectorIndexOverride` | Partial | All but `VectorIndexOverride` are applied as the service applies them: an index override list replaces the backup's indexes of that kind (empty restores none, a subset restores those, a GSI entry's throughput is the restored index's), `OnDemandThroughputOverride` is the restored table's on-demand throughput on an on-demand restore. Refused with `ValidationException` before the target is created: an index the backup does not have or a changed index definition, a throughput member contradicting the billing mode, `{0, 0}`, a `PROVISIONED` restore of a GSI without throughput ("Invalid Request: Must override ProvisionedThroughput if BillingMode is overridden to PROVISIONED for index "), a GSI entry's `OnDemandThroughput` on a `PROVISIONED` restore (on an on-demand restore it is accepted and not recorded), and `VectorIndexOverride`. See differences-from-dynamodb.md | diff --git a/docs/manuals/05-admin-guide.md b/docs/manuals/05-admin-guide.md index 134d05218..788058fff 100755 --- a/docs/manuals/05-admin-guide.md +++ b/docs/manuals/05-admin-guide.md @@ -53,9 +53,31 @@ SIGTERM directly. Alternatively start it with ```bash curl --cacert ~/.extenddb/tls/cert.pem https://127.0.0.1:18443/health -# {"status":"healthy"} +# {"status":"healthy","components":{}} +# with a backup store configured: +# {"status":"healthy","components":{"backup_store":{"status":"healthy","kind":"filesystem"}}} +# while it cannot be reached: +# {"status":"degraded","components":{"backup_store":{"status":"unhealthy","kind":"s3","reason":"unreachable"}}} +# right after startup, before an S3 store's first check has returned: +# {"status":"healthy","components":{"backup_store":{"status":"checking","kind":"s3"}}} ``` +`status` is `healthy`, or `degraded` when a component is unhealthy; the HTTP +status stays 200 either way, because a component problem never takes the data +plane out of rotation. The one component is `backup_store`, present when +`[backup.store]` configures a store; its `reason` is the problem's category in +one or two words (`unreachable`, `permission denied`, `failed validation`, +`identity file unreadable`, `identity file missing`, `identity mismatch`, +`owned by another catalog`, `manifest probe failed`, `catalog unreadable`) and +nothing else: the endpoint is unauthenticated, so the ids, ARNs and table +names a problem involves are in the server's log line only. An S3 store's +first check runs after the listener is up (the next section); until it +returns the component is `checking`, with the server `healthy`. The +reconciler re-runs the store's checks every `reconcile_interval_secs`, each +run bounded (5 s for a filesystem root, 30 s for a bucket, after which the +check is abandoned and the store is `unreachable` until a later check +answers), and the component recovers on its own. + Or without `curl` — useful inside a minimal container image, and what a Docker `HEALTHCHECK` should call: @@ -67,9 +89,8 @@ echo $? # 0 = healthy, 1 = not `healthcheck` reads the port from the config file, or takes an explicit `--endpoint https://host:port`. It accepts the server's self-signed certificate. -This checks liveness, not readiness. `/health` is a static handler that does not -query PostgreSQL, so it reports healthy even when the database has become -unreachable since startup. That is intentional for a liveness probe: one that +This checks liveness, not readiness. `/health` does not query the database, so +it reports healthy even when the database has become unreachable since startup. That is intentional for a liveness probe: one that failed on a database outage would restart every replica at once and prolong the outage. A database that is unreachable at startup stops the server from listening at all, so that case is still caught. @@ -129,7 +150,25 @@ The knobs backup and restore jobs and the reconciler read; every key has a defau | `failed_row_retention_secs` | `604800` | How long FAILED backup rows are kept for the operator (7 days) | | `reconcile_interval_secs` | `60` | The reconciler's cycle; it also runs once at startup | | `max_concurrent_jobs` | `4` | Backup and restore jobs this instance runs at once (at most 1024); a request over it is `LimitExceededException`. On PostgreSQL the pool sizes are raised to `2 * max_concurrent_jobs + 10` when smaller | -| `spool_path` | (unset) | Restore download area for an S3 store, used by a later release | +| `spool_path` | (unset) | Restore download area for an S3 store, created `0700` with files `0600`, holding a `.lock` another instance on the host refuses. Required with an S3 store, except on file-backed SQLite where it defaults to `backup-spool/` beside the database file. Refused inside or containing a filesystem store root, and when it is or contains the database file's directory | + +#### [backup.store] + +The store CreateBackup's backups will live in, and the one restores and deletes of store-backed backups read. The section's keys, defaults and the refusals of a bad value are the backup store crate's (`crates/backup-store/src/config.rs`); the table below is what the server adds. One definition of the shape exists, so an unknown key anywhere under `[backup]` is refused at load. + +| Key | Default | Description | +|-----|---------|-------------| +| `kind` | see below | `filesystem`, `s3`, or `disabled` | +| `filesystem.path` | `backups/` beside a SQLite database file | The root directory of a filesystem store. It must exist, be owned by the extenddb user with mode `0700`, and sit under directories owned by root or that user which others cannot write to; `validate` refuses anything else at startup. A relative path, like a relative `spool_path`, is resolved against the working directory the server was started from (the daemon's own working directory is `/`) | +| `s3.bucket`, `s3.prefix`, `s3.region`, `s3.endpoint`, `s3.force_path_style`, `s3.allow_insecure_endpoint`, `s3.upload_part_size_bytes`, `s3.max_concurrent_uploads` | | The S3 store; see the crate's `config` module doc. Credentials come from the AWS SDK's chain and are never in the file | + +When the section is absent, the backend decides: on file-backed SQLite it is a filesystem store at `backups/` beside the database file, created `0700` if missing, whose validation or identity-check failure is a logged warning that leaves the store unconfigured rather than a startup error or an unhealthy store while the catalog has no backup rows for that root (so an upgrade never stops a server that never configured backups, and CreateBackup keeps working on one); once the catalog has rows for the default root, the root has been used and is handled as a configured one: a failure of its identity check or manifest probe is an unhealthy store (reported on `/health`, its rows' restores and deletes refused, `backup purge` refusing on it rather than reporting no store), and a refusal of its `open` or validation a startup error naming the root; on PostgreSQL, MongoDB, Cassandra and an in-memory SQLite database there is no default and the store is unconfigured, logged at startup. `kind = "disabled"` leaves it unconfigured on purpose on any backend. Without a configured store, CreateBackup writes the in-catalog backup format and logs a deprecation warning naming `[backup.store]` on each call; with one configured, this release's CreateBackup still writes the in-catalog format (the store-backed writer is a later release's) and the store serves restores and deletes of the backups it holds. Backups of either format stay listable, describable, restorable and deletable. + +Startup opens the configured store and, for a filesystem store, validates it and checks its identity before the listener is up (bounded by 5 s: a mount that stops answering after the root is opened leaves the server starting with the store `checking`; a root that does not answer at startup holds the server before the listener, as the database directory would, with the port bound and silent); an S3 store's validation and identity check are requests to the endpoint, and run after the listener is up, in the reconciler's first cycle, so an endpoint that stalls never holds the data plane down (the component is `checking` until the first result, and the operations that touch the store refuse meanwhile as they do while it is unhealthy). A configuration error, every one decided without a request to the store, stops startup with the key, path or bucket in the message: a root that does not exist or fails the owner-only rule, a filesystem root another user can reach, a filesystem root that is or contains the database directory, a bucket name the SDK routes to S3 Express, a bad part size or prefix, an insecure endpoint, an S3 store without `spool_path` on a backend with no default, a `spool_path` that breaks its placement rule, that another live instance holds or that exists with a mode other than `0700`. A transport failure (no region, no credentials, an endpoint that refuses connections or does not answer within the check's bound), a missing or denied bucket, an I/O failure of a filesystem root's probe write or resolution (a full or read-only disk, a stale network handle), or a failed identity check leaves the server running with the store unhealthy: the problem is logged once at error, `/health` reports the `backup_store` component unhealthy (status `degraded`), the restore and delete of a store-backed backup are refused with a `ValidationException` that names the store's kind and the problem's category and not its root, the reconciler acts on no row of the store, and the check runs again each `reconcile_interval_secs` until it passes, at which point the server logs `is healthy again` and resumes without a restart. Each check is bounded (5 s for a filesystem root, 30 s for a bucket): a store that has stopped answering costs a reconciler cycle that bound and nothing more, the cycle's other duties run as before, and the abandoned check is logged; on a filesystem root the abandoned check's call stays parked on one thread until the mount answers, no later cycle starts a check while it is (logged each cycle, the store staying `unreachable`), and `stop` or SIGTERM ends the server within 5 s with that thread still parked. CreateBackup is not refused by an unhealthy store in this release: it writes the in-catalog format, which touches no store; the store-backed create of a later release is. + +The store identity. The root carries `extenddb-store.json`: a random store id and the id of the catalog that owns the root. Startup creates it when the root has no file and the catalog has no rows for the root, and otherwise reads it and compares: a file naming another catalog, a store id that differs from the one the catalog's rows for the root carry, a root with no file while rows exist, or rows carrying more than one id each mark the store unhealthy. Instances that share a catalog must share one store root (one S3 bucket and prefix, or one shared mount): host-local roots configured at the same path give each instance different bytes under one root label, so each check also `head`s the manifest of the newest AVAILABLE backup and marks the store unhealthy when it is absent, which is what a second instance with a local copy of the directory sees. The file is left by `backup purge` and by every deletion; remove it only when retiring the root, or after `extenddb destroy`: a catalog re-created with `init` has a new id, and the old root's file names the old one, so the re-initialised server reports the store owned by another catalog until the file is removed or a new root configured (on the default SQLite root, that is the logged warning above and CreateBackup keeps the in-catalog path). While the catalog has no rows for a root, two host-local directories at one path cannot be told apart (each instance creates its own file and id); point every instance at one shared root before the first backup. The identity file is read up to 64 KiB; a larger object under its key is reported unreadable. + +`extenddb destroy` removes the databases and leaves the store untouched: it is the copy that survives destroy. `extenddb backup list [--failed]` prints the catalog's rows for the configured store's root, every store id they carry, with each row's `store_root` and `store_id`; it is a read that runs beside a server (on SQLite it takes no serve lock), reads the identity file without creating one, and runs on an unhealthy store, naming the problem in its header. `extenddb backup purge --confirm` deletes every backup of the configured store, each backup's objects (its own prefix, never a whole account prefix, so a prefix the catalog has no row for survives) and its catalog row outright, leaving no tombstone, and leaves `extenddb-store.json` and the `exports/` component; it needs the store healthy. It takes the catalog-level maintenance claim first and heartbeats it for the command's whole run, waits up to `2 * stale_job_after_secs` until no store-backed CreateBackup is running on any instance (a CREATING or FAILED row with a heartbeat within `stale_job_after_secs`) and no restore is RUNNING, refuses and releases the claim if they have not finished, deletes a CREATING row the wait saw no live owner for as it deletes any other row, finishes any DeleteBackup left in progress as part of the purge (a backup a DeleteBackup finishes while the purge runs is counted as deleted, not left), and logs each deletion under the `extenddb::audit` target with the account and the ARN. While the claim is held, CreateBackup and RestoreTableFromBackup on every instance answer `LimitExceededException` naming the command, and a second maintenance command fails at once naming the holder. Without `--confirm` it prints the plan, changes nothing and exits 0. `--config` is accepted before or after the subcommand. On SQLite the command opens the database as `serve` does and so takes the serve lock: stop the server first; under that lock it first resolves what the stopped server left (a RUNNING restore is failed and its target removed, the CREATING target of a FAILED restore removed, an in-catalog CREATING backup removed, a DELETING row finished) and deletes a CREATING or FAILED row whatever its heartbeat says (the server that wrote it is stopped, and its clock is this host's), so such a job never makes it wait or refuse; the one exception is a RUNNING restore of kind `manifest`, which no release before PR 9a writes: the command leaves it for the reconciler that resumes it, waits on it as on any RUNNING restore and refuses naming the table; and a server starting on the file removes a claim a killed purge left, so no job is refused for the stale window. On the other backends it runs beside live servers, and a claim whose holder died expires after `stale_job_after_secs`. On every backend Ctrl-C or `SIGTERM` makes the command release the claim and exit: at once while it waits, and before its next backup while it deletes, so an interrupted purge leaves no half-deleted backup and no claim to wait out; run it again for the rest. #### [auth] @@ -887,9 +926,12 @@ JSON metrics endpoint with DynamoDB CloudWatch-style metric names and dimensions ```bash curl --cacert ~/.extenddb/tls/cert.pem https://127.0.0.1:18443/health -# {"status":"healthy"} +# {"status":"healthy","components":{}} ``` +The `components` object carries the `backup_store` component when a backup +store is configured (see [Health Check](#health-check) under Server Lifecycle). + ## Troubleshooting ### Vector Index Support (PostgreSQL) diff --git a/docs/manuals/07-upgrade-manual.md b/docs/manuals/07-upgrade-manual.md index 142302fbd..9bcb91627 100755 --- a/docs/manuals/07-upgrade-manual.md +++ b/docs/manuals/07-upgrade-manual.md @@ -6,6 +6,8 @@ This release versions the PostgreSQL and SQLite catalogs by a migration ledger and carries catalog migrations 1 to 5, which `extenddb migrate --list` prints. **Every PostgreSQL and SQLite deployment created by a 0.1.x release runs `extenddb migrate --yes` once** before a server of this release starts on it: the server refuses a catalog with pending migrations. Migration 1 brings the catalog of every 0.1.x release to the v1.0 schema. Cassandra and MongoDB keep their version strings and both move to catalog 0.0.5 (Cassandra `0.0.4 -> 0.0.5` through `V006__backup_catalog_index.cql`, MongoDB `0.0.2 -> 0.0.5` through its step `002_backup_catalog_index`), the same change as migration 5: run `extenddb migrate --yes` once on each with every pre-upgrade server stopped, as [the section below](#cassandra-and-mongodb-catalog-005) says. +Configure `[backup.store]` before the release that stops creating in-catalog backups. In this release CreateBackup writes the in-catalog format whether or not a store is configured, and logs a deprecation warning naming `[backup.store]` on each call when none is; a later release's CreateBackup writes to the store and refuses without one. `kind = "disabled"` under `[backup.store]` is the opt-out that keeps a deployment on the in-catalog format on purpose and silences the warning. On file-backed SQLite an absent section is a filesystem store at `backups/` beside the database file, created `0700` at startup; that root must be owned by the extenddb user with mode `0700`, under directories owned by root or that user which others cannot write to. A default root that breaks the rule (a database file in a group-writable directory, say), or whose `extenddb-store.json` names another catalog (a catalog re-created by `destroy` and `init` under the old root), is logged at startup as a warning (`backup store: the default root ... failed its check`) and leaves the store unconfigured, not unhealthy, so the upgrade never stops a server that never configured backups and CreateBackup keeps working on it; the same root configured explicitly is a startup error or an unhealthy store. The warning applies only while the catalog has no backup rows for the default root (an upgrade, a fresh deployment, a catalog re-created by `destroy` and `init`): once a server has written rows for it, the default root is handled as a configured one, so its backups never vanish behind a warning (a failed identity check is an unhealthy store, reported on `/health`; a root that can no longer be opened is a startup error naming it). Instances that share a catalog must share one store root, an S3 bucket and prefix or a shared mount: a second instance pointed at a host-local directory under the same configured path starts with the store unhealthy (`/health` reports `backup_store` unhealthy; the admin guide's `[backup.store]` section says what the identity file and the manifest probe check). `extenddb destroy` leaves the store; `extenddb backup purge --confirm` is what deletes its backups, and a catalog re-created with `init` needs the old root's `extenddb-store.json` removed or a new root, since the file names the old catalog. + A backup that a 0.1.x release took can restore without its global and local secondary indexes: a SQLite backup taken by 0.1.14 records only the table's key schema and attribute definitions, and its restore creates the table with its items and no index. Take a new backup of each table after upgrading to v1.0, and keep the 0.1.x backups for their items only. The [general upgrade procedure](#general-upgrade-procedure) gives the sequence. The [version history](#version-history) records the 0.1.x catalog versions and what the 0.1.x binaries print. @@ -155,7 +157,7 @@ Adds the table definition a backup records and the registration of a running res A backup taken before this upgrade has no recorded definition. It restores with its keys, items, and billing mode but no secondary indexes; a provisioned one restores with 5/5 throughput on PostgreSQL and SQLite and with its recorded throughput on Cassandra. On PostgreSQL a pre-upgrade backup of a table with vector indexes is refused at restore; on SQLite it restores without the vector indexes. Tables restored before this upgrade have no registration and report no `RestoreSummary`. -The upgrade is not reversible in place: a 0.0.3 PostgreSQL or SQLite binary refuses to start against a 0.0.4 catalog (`Catalog version mismatch: expected 0.0.3, found 0.0.4`). Roll back by restoring what you saved before the upgrade. Backups live in the catalog, so that also removes every backup created after the upgrade. +The upgrade is not reversible in place: a 0.0.3 PostgreSQL or SQLite binary refuses to start against a 0.0.4 catalog (`Catalog version mismatch: expected 0.0.3, found 0.0.4`). Roll back by restoring what you saved before the upgrade. Backups live in the catalog, so that also removes every backup created after the upgrade (this is the in-catalog format; a backup in a `[backup.store]` store of a later catalog is outside the database, and a catalog restored from a pre-upgrade dump loses its `backup_index` rows while the store keeps the bytes). #### PostgreSQL diff --git a/docs/manuals/11-deployment-guide.md b/docs/manuals/11-deployment-guide.md index b27176654..ff20018b8 100755 --- a/docs/manuals/11-deployment-guide.md +++ b/docs/manuals/11-deployment-guide.md @@ -267,7 +267,8 @@ Run one extenddb instance per catalog. Multiple instances behind a load balancer - Credentials, policies, and table metadata are cached in each process (see the cache TTLs in the admin guide). Nothing invalidates a cache on another instance, so a key revoked or a policy changed through one instance stays in force on the others until the TTL expires. - Every instance runs every background worker (TTL expiry, GSI backfill, stream retention, control-plane transitions). The workers are safe to run twice, but the work is duplicated. -- `/health` is a liveness check that does not query the database, so a load balancer cannot use it to take an instance with a broken database connection out of rotation. +- `/health` is a liveness check that does not query the database, so a load balancer cannot use it to take an instance with a broken database connection out of rotation. It does report the backup store: `components.backup_store` is `unhealthy` and `status` is `degraded` while the configured store cannot be reached or fails its identity check, with HTTP 200 kept, so a store outage never drains a replica. +- Instances that share a catalog must share one backup store root (`[backup.store]`: one S3 bucket and prefix, or one shared mount that supports hard links, which the store's conditional create needs; some CIFS and FUSE mounts refuse `link(2)` with `EPERM`, and the server then reports the store as having failed validation, with the `link` error in the log line). A host-local directory configured at the same path on each host is not one root: once the catalog has rows for the root, the second instance finds the root's identity file missing or the newest backup's manifest absent and starts with the store unhealthy. While the catalog has no rows for the root (a fresh deployment, or one whose backups were purged), nothing tells the two directories apart: each instance creates its own identity file and store id and both start healthy, and the first backups then split the rows between two ids, which the next check reports as an identity mismatch that only `backup purge` resolves. Point every instance at one root before the first backup. Each instance on a host needs its own `spool_path`; the spool's `.lock` refuses a second instance. An existing spool directory must be owned by the server's user with mode `0700`, as the root must; a spool created `0755` stops startup naming it. On SQLite this limit is enforced: `extenddb serve` holds an exclusive lock on `.lock` and a second server on the same file refuses to start. On PostgreSQL, MongoDB, and Cassandra it is not enforced. On PostgreSQL, if you start a second instance anyway, size `pool_size × instance_count + 3 × instance_count ≤ PostgreSQL max_connections`. diff --git a/extenddb.sample.toml b/extenddb.sample.toml index 2443ccd3d..34d07a562 100755 --- a/extenddb.sample.toml +++ b/extenddb.sample.toml @@ -188,7 +188,30 @@ # failed_row_retention_secs = 604800 # FAILED tombstones kept for the operator (7 days) # reconcile_interval_secs = 60 # Reconciler cycle # max_concurrent_jobs = 4 # Backup and restore jobs running at once on this instance -# spool_path = "/var/lib/extenddb/spool" # Restore download area for an S3 store +# spool_path = "/var/lib/extenddb/spool" # Restore download area for an S3 store; required +# # with an S3 store except on file-backed SQLite, where +# # it defaults to backup-spool/ beside the database file + +# [backup.store] +# Where CreateBackup's backups will live, from a later release (this release's +# CreateBackup writes in-catalog backups with any store, and the store serves +# restores and deletes of the backups it holds). Absent on file-backed SQLite: a filesystem +# store at backups/ beside the database file. Absent on PostgreSQL, MongoDB, +# Cassandra and an in-memory SQLite database: no store, and CreateBackup +# writes in-catalog backups with a deprecation warning. kind = "disabled" +# opts out on any backend. Instances that share a catalog must share one +# store root (an S3 bucket and prefix, or a shared mount). +# kind = "filesystem" # "filesystem" | "s3" | "disabled" +# [backup.store.filesystem] +# path = "/var/lib/extenddb/backups" # Must exist, owned by the extenddb user, mode 0700 +# [backup.store.s3] +# bucket = "my-extenddb-backups" +# prefix = "extenddb/" # Optional key prefix +# region = "us-east-1" # Optional; the AWS SDK's chain otherwise +# endpoint = "http://127.0.0.1:9000" # Optional, for an S3-compatible server +# force_path_style = true # Required with an http:// endpoint +# upload_part_size_bytes = 8388608 # 5 MiB to 5 GiB +# max_concurrent_uploads = 4 # Import/Export file operations are disabled by default (secure default). # Configure allowed directories separately for imports and exports. diff --git a/tests/test_backup_store_config.py b/tests/test_backup_store_config.py new file mode 100644 index 000000000..bf7926444 --- /dev/null +++ b/tests/test_backup_store_config.py @@ -0,0 +1,1331 @@ +# Copyright 2026 ExtendDB contributors +# SPDX-License-Identifier: Apache-2.0 + +"""The `[backup.store]` configuration at the server's boundary. + +Each test starts its own `extenddb serve --foreground` on an isolated +deployment and reads `/health`, the server's log and the API's answers: + +- the component `backup_store` on `/health`: absent without a store, healthy + with a filesystem store, `checking` while an S3 store's first check is in + flight, unhealthy (status `degraded`) while an S3 endpoint refuses + connections or stalls, and healthy again once it answers, without a restart; +- the restore and delete of a store-backed backup refused with a + ValidationException naming the store's kind and the problem's category and + not its root while the store is checking or unhealthy; CreateBackup served + throughout (this release's create writes the in-catalog format, which + touches no store); +- a stalled S3 endpoint holding the listener for no longer than the server's + own startup, and each reconciler cycle for no longer than the check's bound; +- the configuration errors that stop startup with the key, path or bucket in + the message: an S3 store without `spool_path` on PostgreSQL, a `spool_path` + that is the database directory, a filesystem root that does not exist, and + a defaulted SQLite root under a group-writable parent configured explicitly + (the same root defaulted is a warning that leaves the store unconfigured + while the catalog has no rows for it; with rows for it the defaulted root + is handled as a configured one: unhealthy, reported, its rows refused); +- the default root beside a SQLite database file created `0700` with + `extenddb-store.json` in it, and a relative root resolved against the + shell's directory in daemon mode; +- `extenddb backup list` and `backup purge --confirm` on an empty store, and + `backup purge` refused while a server holds the SQLite serve lock. + +The binary at `EXTENDDB_BINARY` decides which cells run: a SQLite build runs +the SQLite cells, a PostgreSQL build (with +`EXTENDDB_TEST_PG_CONNECTION_STRING`) the PostgreSQL ones. The S3 cells need +moto at `EXTENDDB_TEST_S3_ENDPOINT` (an `http://127.0.0.1:` endpoint) +and skip with a reason otherwise. Excluded from the backend-agnostic suite. +""" + +from __future__ import annotations + +import json +import os +import signal +import socket +import ssl +import subprocess +import threading +import time +import urllib.request +import uuid + +import boto3 +import pytest +from botocore.exceptions import ClientError + +import lifecycle_helpers as lh + +EXTENDDB_BINARY = lh.EXTENDDB_BINARY +S3_ENDPOINT = os.environ.get("EXTENDDB_TEST_S3_ENDPOINT", "").strip() +ADMIN_PASSWORD = "TestPass1!" +_DEVTOOLS = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "devtools") + + +def _health(port: int) -> dict: + ctx = ssl.create_default_context() + ctx.check_hostname = False + ctx.verify_mode = ssl.CERT_NONE + req = urllib.request.Request(f"https://127.0.0.1:{port}/health") + with urllib.request.urlopen(req, context=ctx, timeout=5) as resp: + assert resp.status == 200 + return json.loads(resp.read()) + + +def _wait_health(port: int, predicate, timeout: float = 20.0, server: "_Server | None" = None) -> dict: + deadline = time.monotonic() + timeout + last = None + while time.monotonic() < deadline: + if server is not None and server.proc.poll() is not None: + raise AssertionError(f"the server exited {server.proc.returncode}:\n{server.log_text()}") + try: + last = _health(port) + if predicate(last): + return last + except Exception: # noqa: BLE001 - the server may not listen yet + pass + time.sleep(0.25) + log = server.log_text() if server is not None else "" + raise AssertionError(f"/health did not reach the expected state; last: {last!r}\n{log}") + + +def _append(config_path: str, toml: str) -> None: + with open(config_path, "a") as f: + f.write("\n" + toml + "\n") + + +def _free_port() -> int: + with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s: + s.bind(("127.0.0.1", 0)) + return s.getsockname()[1] + + +class _Server: + """A `serve --foreground` process with its log in a file.""" + + def __init__(self, config_path: str, log_path: str, env: dict | None = None): + self.log_path = log_path + self.log = open(log_path, "wb") # noqa: SIM115 - closed in stop() + # A SQLite deployment's paths are relative to the config's directory. + self.proc = subprocess.Popen( + [EXTENDDB_BINARY, "serve", "--foreground", "--config", config_path], + stdout=self.log, + stderr=subprocess.STDOUT, + stdin=subprocess.DEVNULL, + env=env, + cwd=os.path.dirname(os.path.abspath(config_path)), + ) + + def log_text(self) -> str: + self.log.flush() + with open(self.log_path, errors="replace") as f: + return f.read() + + def wait_exit(self, timeout: float = 20.0) -> int: + return self.proc.wait(timeout=timeout) + + def stop(self) -> None: + if self.proc.poll() is None: + self.proc.terminate() + try: + self.proc.wait(timeout=20) + except subprocess.TimeoutExpired: + self.proc.kill() + self.proc.wait(timeout=10) + self.log.close() + + +def _provision(port: int, cert_path: str, admin_password: str) -> dict: + env = os.environ.copy() + env.update( + { + "EXTENDDB_TEST_ENDPOINT": f"https://127.0.0.1:{port}", + "EXTENDDB_ADMIN_USER": "admin", + "EXTENDDB_ADMIN_PASSWORD": admin_password, + "EXTENDDB_CA_CERT": cert_path, + } + ) + res = subprocess.run( + ["python3", os.path.join(_DEVTOOLS, "provision-test-credentials")], + capture_output=True, + text=True, + env=env, + check=True, + timeout=60, + ) + creds = {} + for line in res.stdout.splitlines(): + line = line.strip() + if line.startswith("export "): + k, _, v = line[len("export "):].partition("=") + creds[k] = v.strip().strip('"').strip("'") + assert "AWS_ACCESS_KEY_ID" in creds, f"provisioning failed: {res.stdout}\n{res.stderr}" + return creds + + +def _client(port: int, creds: dict): + return boto3.client( + "dynamodb", + endpoint_url=f"https://127.0.0.1:{port}", + region_name="us-east-1", + aws_access_key_id=creds["AWS_ACCESS_KEY_ID"], + aws_secret_access_key=creds["AWS_SECRET_ACCESS_KEY"], + verify=False, + ) + + +def _create_table(ddb, name: str) -> None: + ddb.create_table( + TableName=name, + KeySchema=[{"AttributeName": "pk", "KeyType": "HASH"}], + AttributeDefinitions=[{"AttributeName": "pk", "AttributeType": "S"}], + BillingMode="PAY_PER_REQUEST", + ) + deadline = time.monotonic() + 60 + while time.monotonic() < deadline: + if ddb.describe_table(TableName=name)["Table"]["TableStatus"] == "ACTIVE": + return + time.sleep(0.05) + raise AssertionError("table did not become ACTIVE") + + +class _Forwarder: + """A TCP forwarder from a local port to moto, started when the test wants + the S3 endpoint to start answering.""" + + def __init__(self, listen_port: int, upstream: tuple[str, int]): + self.listen_port = listen_port + self.upstream = upstream + self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + self.sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) + self.sock.bind(("127.0.0.1", listen_port)) + self.sock.listen(16) + self.stopping = threading.Event() + self.thread = threading.Thread(target=self._accept, daemon=True) + self.thread.start() + + def _pump(self, a: socket.socket, b: socket.socket) -> None: + try: + while not self.stopping.is_set(): + data = a.recv(65536) + if not data: + break + b.sendall(data) + except OSError: + pass + finally: + for s in (a, b): + try: + s.shutdown(socket.SHUT_RDWR) + except OSError: + pass + + def _accept(self) -> None: + self.sock.settimeout(0.5) + while not self.stopping.is_set(): + try: + client, _ = self.sock.accept() + except socket.timeout: + continue + except OSError: + break + try: + upstream = socket.create_connection(self.upstream, timeout=5) + except OSError: + client.close() + continue + threading.Thread(target=self._pump, args=(client, upstream), daemon=True).start() + threading.Thread(target=self._pump, args=(upstream, client), daemon=True).start() + + def stop(self) -> None: + self.stopping.set() + self.sock.close() + + +def _moto_or_skip() -> tuple[str, int]: + if not S3_ENDPOINT.startswith("http://"): + pytest.skip("EXTENDDB_TEST_S3_ENDPOINT is not an http:// moto endpoint") + host_port = S3_ENDPOINT[len("http://"):].rstrip("/") + host, _, port = host_port.partition(":") + return host, int(port) + + +def _s3_env() -> dict: + env = os.environ.copy() + env.update( + { + "AWS_ACCESS_KEY_ID": "testing", + "AWS_SECRET_ACCESS_KEY": "testing", + "AWS_REGION": "us-east-1", + "AWS_EC2_METADATA_DISABLED": "true", + } + ) + env.pop("AWS_SESSION_TOKEN", None) + env.pop("AWS_PROFILE", None) + return env + + +def _s3_client(): + return boto3.client( + "s3", + endpoint_url=S3_ENDPOINT, + region_name="us-east-1", + aws_access_key_id="testing", + aws_secret_access_key="testing", + ) + + +@pytest.fixture() +def moto_bucket(): + """A bucket of this test's own on the shared moto, emptied and deleted at + teardown so the endpoint does not accumulate one per run.""" + _moto_or_skip() + bucket = f"extenddb-wire-{uuid.uuid4().hex[:12]}" + s3 = _s3_client() + s3.create_bucket(Bucket=bucket) + yield bucket + try: + listed = s3.list_objects_v2(Bucket=bucket).get("Contents", []) + if listed: + s3.delete_objects(Bucket=bucket, Delete={"Objects": [{"Key": o["Key"]} for o in listed]}) + for upload in s3.list_multipart_uploads(Bucket=bucket).get("Uploads", []): + s3.abort_multipart_upload(Bucket=bucket, Key=upload["Key"], UploadId=upload["UploadId"]) + s3.delete_bucket(Bucket=bucket) + except ClientError: + pass + + +def _assert_store_refusal(exc: ClientError, kind: str, root_fragment: str, category: str = "unreachable") -> None: + """The refusal of a store-backed backup's restore or delete: a + ValidationException naming the store's kind and the problem's category + (`unreachable`, `manifest probe failed`, `first check is pending`) and + never the root.""" + err = exc.response["Error"] + assert err["Code"] == "ValidationException", err + assert f"{kind} backup store" in err["Message"], err + assert category in err["Message"], err + assert root_fragment not in err["Message"], err + + +_SOURCE_TABLE_JSON = json.dumps({ + "table_name": "Music", "table_id": "t-1", "table_arn": "arn:aws:dynamodb:us-east-1:123456789012:table/Music", + "key_schema": [{"AttributeName": "pk", "KeyType": "HASH"}], + "attribute_definitions": [{"AttributeName": "pk", "AttributeType": "S"}], + "billing_mode": "PAY_PER_REQUEST", "provisioned_throughput": None, "on_demand_throughput": None, + "global_secondary_indexes": [], "local_secondary_indexes": [], "vector_indexes": None, + "stream_specification": None, "time_to_live": None, "sse_description": None, + "item_count": 0, "table_size_bytes": 0, "table_creation_date_time": 0.0, +}) + + +def _sqlite_module(): + try: + # The catalog's schema needs a SQLite newer than some hosts' stdlib + # module links; the wheel carries a current library. + import pysqlite3 as sqlite3 + except ImportError: + import sqlite3 + return sqlite3 + + +def _seed_backup_row(db_path: str, backup_id: str, status: str, store_kind: str, store_root: str, store_id: str) -> str: + """One `backup_index` row of account 123456789012's `Music` table under + the given store, as a server that used the store leaves it; returns the + backup's ARN.""" + account = "123456789012" + arn = f"arn:aws:dynamodb:us-east-1:{account}:table/Music/backup/{backup_id}" + conn = _sqlite_module().connect(db_path, timeout=10) + try: + conn.execute( + "INSERT INTO backup_index (account_id, backup_id, table_name, backup_arn, backup_name, status, " + "store_kind, store_root, store_id, size_bytes, item_count, created_at, source_table_details_json, " + "owner_generation, heartbeat_at) VALUES (?, ?, 'Music', ?, 'nightly', ?, ?, ?, ?, 1, 1, " + "'2026-10-01T00:00:00.000Z', ?, 7, '2026-10-01T00:00:00.000Z')", + (account, backup_id, arn, status, store_kind, store_root, store_id, _SOURCE_TABLE_JSON), + ) + conn.commit() + finally: + conn.close() + return arn + + +# --- SQLite --------------------------------------------------------------------------- + + +def _sqlite_build() -> bool: + if not os.path.isfile(EXTENDDB_BINARY): + return False + res = subprocess.run( + [EXTENDDB_BINARY, "version"], capture_output=True, text=True, timeout=30 + ) + return "(sqlite)" in res.stdout + + +class _SqliteDeployment: + def __init__(self, tmp_dir: str): + self.dir = tmp_dir + self.config = os.path.join(tmp_dir, "extenddb.toml") + self.db_path = os.path.join(tmp_dir, "extenddb.sqlite") + self.port = _free_port() + env = os.environ.copy() + env["EXTENDDB_ADMIN_PASSWORD"] = ADMIN_PASSWORD + subprocess.run( + [EXTENDDB_BINARY, "init", "--backend", "sqlite", "--config", self.config], + cwd=tmp_dir, + capture_output=True, + text=True, + stdin=subprocess.DEVNULL, + timeout=60, + check=True, + env=env, + ) + lh._patch_config_port(self.config, self.port) + self.cert = os.path.expanduser("~/.extenddb/tls/cert.pem") + + def run(self, *args, env=None): + return subprocess.run( + [EXTENDDB_BINARY, args[0], "--config", self.config, *args[1:]], + cwd=self.dir, + capture_output=True, + text=True, + stdin=subprocess.DEVNULL, + timeout=180, + env=env, + ) + + +@pytest.fixture() +def sqlite_deployment(tmp_path): + if not _sqlite_build(): + pytest.skip("EXTENDDB_BINARY is not a SQLite build") + return _SqliteDeployment(str(tmp_path)) + + +class TestSqlite: + def test_default_root_is_created_private_and_reported_healthy(self, sqlite_deployment, tmp_path): + d = sqlite_deployment + server = _Server(d.config, str(tmp_path / "serve.log")) + try: + health = _wait_health(d.port, lambda h: "components" in h, server=server) + assert health["status"] == "healthy", health + component = health["components"]["backup_store"] + assert component == {"status": "healthy", "kind": "filesystem"}, component + root = os.path.join(d.dir, "backups") + assert os.path.isdir(root) + assert (os.stat(root).st_mode & 0o777) == 0o700 + identity = os.path.join(root, "extenddb-store.json") + with open(identity) as f: + file = json.load(f) + assert file["format_version"] == 1 + assert file["backend"] == "sqlite" + assert file["store_id"] and file["catalog_id"] + assert (os.stat(identity).st_mode & 0o777) == 0o600 + log = server.log_text() + assert "backup store: filesystem" in log and root in log, log + assert f"store id {file['store_id']}" in log, log + # The console banner names the store beside the storage line. + assert " storage: sqlite" in log, log + assert f" backup store: filesystem path {root} (the default beside the database file)" in log, log + # CreateBackup takes the in-catalog path and says the store is configured. + creds = _provision(d.port, d.cert, ADMIN_PASSWORD) + ddb = _client(d.port, creds) + _create_table(ddb, "Music") + details = ddb.create_backup(TableName="Music", BackupName="nightly")["BackupDetails"] + assert details["BackupStatus"] in ("CREATING", "AVAILABLE") + assert "writing the in-catalog backup format; the configured filesystem backup store" in server.log_text() + assert "configure [backup.store]" not in server.log_text() + finally: + server.stop() + + def test_disabled_leaves_no_component_and_the_legacy_create_warns(self, sqlite_deployment, tmp_path): + d = sqlite_deployment + _append(d.config, '[backup.store]\nkind = "disabled"\n') + server = _Server(d.config, str(tmp_path / "serve.log")) + try: + health = _wait_health(d.port, lambda h: "components" in h, server=server) + assert health["status"] == "healthy" + assert "backup_store" not in health["components"], health + assert not os.path.exists(os.path.join(d.dir, "backups")) + assert 'backup store: disabled' in server.log_text() + creds = _provision(d.port, d.cert, ADMIN_PASSWORD) + ddb = _client(d.port, creds) + _create_table(ddb, "Music") + ddb.create_backup(TableName="Music", BackupName="nightly") + assert "configure [backup.store] to write backups to a backup store" in server.log_text() + finally: + server.stop() + + def test_default_root_under_a_group_writable_parent_warns_and_the_same_root_explicit_fails( + self, sqlite_deployment, tmp_path + ): + d = sqlite_deployment + os.chmod(d.dir, 0o775) + server = _Server(d.config, str(tmp_path / "serve.log")) + try: + health = _wait_health(d.port, lambda h: "components" in h, server=server) + assert "backup_store" not in health["components"], health + log = server.log_text() + # `open` applies the ancestor rule; `validate` would report the + # same root the same way. Either is the warning path. + assert ("is not usable" in log or "failed its validation" in log) and "unconfigured" in log, log + assert "writable by group or others" in log, log + assert "kind = \"disabled\" to silence this" in log, log + creds = _provision(d.port, d.cert, ADMIN_PASSWORD) + ddb = _client(d.port, creds) + _create_table(ddb, "Music") + ddb.create_backup(TableName="Music", BackupName="nightly") + assert "configure [backup.store] to write backups to a backup store" in server.log_text() + finally: + server.stop() + # The same root configured explicitly is a startup error naming it. + root = os.path.join(d.dir, "backups") + _append(d.config, f'[backup.store]\nkind = "filesystem"\n[backup.store.filesystem]\npath = "{root}"\n') + server = _Server(d.config, str(tmp_path / "serve2.log")) + try: + assert server.wait_exit() != 0 + log = server.log_text() + assert "cannot be opened" in log or "failed its startup validation" in log, log + assert "writable by group or others" in log and d.dir in log, log + finally: + server.stop() + + def test_a_missing_explicit_root_and_a_spool_in_the_database_directory_stop_startup( + self, sqlite_deployment, tmp_path + ): + d = sqlite_deployment + missing = os.path.join(d.dir, "nowhere") + _append(d.config, f'[backup.store]\nkind = "filesystem"\n[backup.store.filesystem]\npath = "{missing}"\n') + server = _Server(d.config, str(tmp_path / "serve.log")) + try: + assert server.wait_exit() != 0 + log = server.log_text() + assert "cannot be opened" in log and missing in log, log + finally: + server.stop() + # A spool that is the database directory. + (tmp_path / "second").mkdir() + d2 = _SqliteDeployment(str(tmp_path / "second")) + _append( + d2.config, + f'[backup]\nspool_path = "{d2.dir}"\n[backup.store]\nkind = "s3"\n[backup.store.s3]\nbucket = "b"\n', + ) + server = _Server(d2.config, str(tmp_path / "serve2.log")) + try: + assert server.wait_exit() != 0 + log = server.log_text() + assert "backup.spool_path" in log and "database directory" in log, log + finally: + server.stop() + + def test_an_unreachable_s3_endpoint_starts_the_server_unhealthy_and_recovers( + self, sqlite_deployment, tmp_path, moto_bucket + ): + d = sqlite_deployment + moto_host, moto_port = _moto_or_skip() + bucket = moto_bucket + dead_port = _free_port() + _append( + d.config, + "[backup]\nreconcile_interval_secs = 1\n" + '[backup.store]\nkind = "s3"\n[backup.store.s3]\n' + f'bucket = "{bucket}"\nendpoint = "http://127.0.0.1:{dead_port}"\n' + 'force_path_style = true\nregion = "us-east-1"\n', + ) + server = _Server(d.config, str(tmp_path / "serve.log"), env=_s3_env()) + forwarder = None + try: + # The first check runs after the listener is up: the component is + # `checking` first, then `unhealthy` when the connection is refused. + health = _wait_health( + d.port, lambda h: h.get("components", {}).get("backup_store", {}).get("status") == "unhealthy", + server=server, + ) + assert health["status"] == "degraded", health + component = health["components"]["backup_store"] + assert component["status"] == "unhealthy" and component["kind"] == "s3", component + assert component["reason"] == "unreachable", component + assert bucket not in json.dumps(component) + spool = os.path.join(d.dir, "backup-spool") + assert (os.stat(spool).st_mode & 0o777) == 0o700 + assert (os.stat(os.path.join(spool, ".lock")).st_mode & 0o777) == 0o600 + creds = _provision(d.port, d.cert, ADMIN_PASSWORD) + ddb = _client(d.port, creds) + _create_table(ddb, "Music") + # The in-catalog create touches no store: it runs on an unhealthy + # one, as it does on a server with none. + details = ddb.create_backup(TableName="Music", BackupName="nightly")["BackupDetails"] + assert details["BackupStatus"] in ("CREATING", "AVAILABLE") + log = server.log_text() + assert "is unhealthy: the store could not be reached" in log, log + assert bucket in log, "the operator's line names the bucket" + + # The endpoint starts answering: the next reconciler cycle recovers. + forwarder = _Forwarder(dead_port, (moto_host, moto_port)) + health = _wait_health( + d.port, lambda h: h["components"]["backup_store"]["status"] == "healthy", timeout=30, server=server + ) + assert health["status"] == "healthy" + assert "is healthy again" in server.log_text() + details = ddb.create_backup(TableName="Music", BackupName="nightly2")["BackupDetails"] + assert details["BackupStatus"] in ("CREATING", "AVAILABLE") + # The identity file was created on the first passing check. + body = _s3_client().get_object(Bucket=bucket, Key="extenddb-store.json")["Body"].read() + assert json.loads(body)["backend"] == "sqlite" + + # And degrades again once the endpoint is gone. + forwarder.stop() + forwarder = None + health = _wait_health( + d.port, lambda h: h["components"]["backup_store"]["status"] == "unhealthy", timeout=30, server=server + ) + assert health["status"] == "degraded" + details = ddb.create_backup(TableName="Music", BackupName="again")["BackupDetails"] + assert details["BackupStatus"] in ("CREATING", "AVAILABLE"), "the in-catalog create is not gated" + finally: + if forwarder is not None: + forwarder.stop() + server.stop() + + def test_a_stalled_s3_endpoint_holds_neither_the_listener_nor_the_reconciler(self, sqlite_deployment, tmp_path): + """An endpoint that accepts connections and never answers. Checked: + the server binds and serves before the store's first check, which runs + after the listener is up (`/health` answers well within the SDK's + attempt timeout with the component `checking`); the in-catalog + CreateBackup is served meanwhile; the restore of a seeded store-backed + row is refused naming the pending check; the check is abandoned at its + 30 s bound, the component turns `unhealthy` (`unreachable`) without a + restart and the refusal names that; and the reconciler keeps cycling + while the endpoint stalls (the next cycle's check opens a new + connection to the tarpit within seconds of the abandoned one).""" + d = sqlite_deployment + tarpit = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + tarpit.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) + tarpit.bind(("127.0.0.1", 0)) + tarpit.listen(16) + tarpit_port = tarpit.getsockname()[1] + held = [] + accepted_at = [] + + def accept_and_hold(): + while True: + try: + conn, _ = tarpit.accept() + except OSError: + return + held.append(conn) + accepted_at.append(time.monotonic()) + + threading.Thread(target=accept_and_hold, daemon=True).start() + # A row of the store, as a server that used it leaves one; the root + # label is the bucket and the endpoint in the spelling the store uses. + store_root = f"s3://tarpit-bucket/?endpoint=http://127.0.0.1:{tarpit_port}" + arn = _seed_backup_row(d.db_path, "01791976431438-00000001", "AVAILABLE", "s3", store_root, "store-seeded") + _append( + d.config, + "[backup]\nreconcile_interval_secs = 1\n" + '[backup.store]\nkind = "s3"\n[backup.store.s3]\n' + f'bucket = "tarpit-bucket"\nendpoint = "http://127.0.0.1:{tarpit_port}"\n' + 'force_path_style = true\nregion = "us-east-1"\n', + ) + started = time.monotonic() + server = _Server(d.config, str(tmp_path / "serve.log"), env=_s3_env()) + try: + health = _wait_health(d.port, lambda h: "components" in h, timeout=30, server=server) + answered_after = time.monotonic() - started + # The first answer is the server's own startup (TLS, the catalog + # open), about a second here; a check awaited before the listener + # would add the SDK's 60 s attempt timeout at the least. + assert answered_after < 30.0, f"/health first answered after {answered_after:.1f} s" + assert health["status"] == "healthy", health + assert health["components"]["backup_store"] == {"status": "checking", "kind": "s3"}, health + deadline = time.monotonic() + 20 + while not held and time.monotonic() < deadline: + time.sleep(0.1) + assert held, "the store's check reached the endpoint" + log = server.log_text() + assert "listening on" in log and "run after the listener is up" in log, log + assert "backup store: s3" in log and "tarpit-bucket" in log, log + # Requests are served while the check hangs: the in-catalog create + # runs, and the seeded row's restore is refused naming the pending + # check and the store's kind, not the bucket. + creds = _provision(d.port, d.cert, ADMIN_PASSWORD) + ddb = _client(d.port, creds) + _create_table(ddb, "Music") + details = ddb.create_backup(TableName="Music", BackupName="nightly")["BackupDetails"] + assert details["BackupStatus"] in ("CREATING", "AVAILABLE") + with pytest.raises(ClientError) as refused: + ddb.restore_table_from_backup(TargetTableName="Restored", BackupArn=arn) + _assert_store_refusal(refused.value, "s3", "tarpit-bucket", category="first check is pending") + assert "Restored" not in [t for t in ddb.list_tables()["TableNames"]] + + # The check is abandoned at its bound (30 s, under the SDK's 60 s + # attempt timeout): the component turns unhealthy without a + # restart, the log says the check was abandoned and the cycle's + # duties ran. + health = _wait_health( + d.port, lambda h: h["components"]["backup_store"]["status"] == "unhealthy", timeout=45, server=server + ) + assert health["status"] == "degraded", health + assert health["components"]["backup_store"]["reason"] == "unreachable", health + log = server.log_text() + assert "check did not finish within 30 s and was abandoned" in log, log + assert "is unhealthy: the store could not be reached: the check did not finish within 30 s" in log, log + with pytest.raises(ClientError) as refused: + ddb.restore_table_from_backup(TargetTableName="Restored", BackupArn=arn) + _assert_store_refusal(refused.value, "s3", "tarpit-bucket", category="unreachable") + # The reconciler keeps cycling: the next cycle's check opens a new + # connection to the tarpit within seconds of the abandoned one + # (the SDK's own retry would come 60 s after the first). + first = accepted_at[0] + deadline = time.monotonic() + 15 + while len(accepted_at) < 2 and time.monotonic() < deadline: + time.sleep(0.2) + assert len(accepted_at) >= 2, "the next cycle's check reached the endpoint" + assert accepted_at[1] - first < 45.0, f"the second connection came {accepted_at[1] - first:.1f} s after the first" + assert "is unhealthy" in log and log.count("is unhealthy: the store could not be reached") == 1, \ + "the transition is logged once" + details = ddb.create_backup(TableName="Music", BackupName="nightly2")["BackupDetails"] + assert details["BackupStatus"] in ("CREATING", "AVAILABLE") + finally: + server.stop() + tarpit.close() + for conn in held: + conn.close() + # Shutdown did not wait on the stalled check. + with open(server.log_path, errors="replace") as f: + final_log = f.read() + assert "did not drain" not in final_log, final_log + + def test_a_default_root_with_rows_is_unhealthy_rather_than_unconfigured(self, sqlite_deployment, tmp_path): + """The default root's leniency ends with the catalog's first row for + it: a row seeded under the default root with its manifest missing + starts the server with the store configured and unhealthy (`/health` + degraded, `manifest probe failed`), the row's restore refused naming + the kind and the category and not the root, recovery when the manifest + is put back, and `backup purge --confirm` and `backup list` reaching + the store (the purge refuses on the unhealthy store, not for want of + one).""" + d = sqlite_deployment + _append(d.config, "[backup]\nreconcile_interval_secs = 1\n") + server = _Server(d.config, str(tmp_path / "serve.log")) + try: + _wait_health(d.port, lambda h: "backup_store" in h.get("components", {}), server=server) + finally: + server.stop() + root = os.path.realpath(os.path.join(d.dir, "backups")) + with open(os.path.join(root, "extenddb-store.json")) as f: + store_id = json.load(f)["store_id"] + backup_id = "01791976431438-00000001" + arn = _seed_backup_row(d.db_path, backup_id, "AVAILABLE", "filesystem", root, store_id) + manifest_dir = os.path.join(root, "123456789012", "Music", backup_id) + manifest = os.path.join(manifest_dir, "extenddb-manifest.json") + + server = _Server(d.config, str(tmp_path / "serve2.log")) + try: + health = _wait_health(d.port, lambda h: "backup_store" in h.get("components", {}), server=server) + assert health["status"] == "degraded", health + assert health["components"]["backup_store"] == { + "status": "unhealthy", "kind": "filesystem", "reason": "manifest probe failed", + }, health + log = server.log_text() + assert "is unhealthy: the manifest of " + arn + " is not under the root" in log, log + assert "backup rows for it, so it is handled as a configured root" in log, log + assert "unconfigured" not in log, log + creds = _provision(d.port, d.cert, ADMIN_PASSWORD) + ddb = _client(d.port, creds) + with pytest.raises(ClientError) as refused: + ddb.restore_table_from_backup(TargetTableName="Restored", BackupArn=arn) + _assert_store_refusal(refused.value, "filesystem", root, category="manifest probe failed") + with pytest.raises(ClientError) as refused: + ddb.delete_backup(BackupArn=arn) + _assert_store_refusal(refused.value, "filesystem", root, category="manifest probe failed") + # The in-catalog create is served on the unhealthy store. + _create_table(ddb, "Music") + details = ddb.create_backup(TableName="Music", BackupName="nightly")["BackupDetails"] + assert details["BackupStatus"] in ("CREATING", "AVAILABLE") + # The manifest put back: the next cycle recovers the store. + for level in (os.path.dirname(os.path.dirname(manifest_dir)), os.path.dirname(manifest_dir), manifest_dir): + if not os.path.isdir(level): + os.mkdir(level, 0o700) + with open(manifest, "wb") as f: + f.write(b"{}") + health = _wait_health( + d.port, lambda h: h["components"]["backup_store"]["status"] == "healthy", timeout=30, server=server + ) + assert health["status"] == "healthy", health + assert "is healthy again" in server.log_text() + finally: + server.stop() + + # The commands reach the store: with the manifest gone again, the + # listing (a read of the file and the rows, no probe) shows the row + # under the store, and the purge, which runs the check, refuses on + # the unhealthy store and deletes nothing. + os.remove(manifest) + res = d.run("backup", "list") + assert res.returncode == 0, res.stdout + res.stderr + assert backup_id in res.stdout and f"backups in the filesystem store {root}" in res.stdout, res.stdout + res = d.run("backup", "purge", "--confirm") + out = res.stdout + res.stderr + assert res.returncode != 0, out + assert "is unhealthy" in out and "manifest" in out, out + assert "no backup store is configured" not in out, out + res = d.run("backup", "list") + assert backup_id in res.stdout, "nothing was deleted: " + res.stdout + + def test_a_relative_root_resolves_against_the_shell_in_daemon_mode(self, sqlite_deployment, tmp_path): + """A relative `[backup.store.filesystem] path` is made absolute at + config load, against the shell's directory, so the daemonized child + (whose working directory is `/`) opens the same root the operator + named: the root beside the config gets the identity file and nothing + appears under `/`.""" + d = sqlite_deployment + os.mkdir(os.path.join(d.dir, "rel-backups"), 0o700) + run_dir = os.path.join(d.dir, "run") + with open(d.config) as f: + text = f.read() + import re + text, n = re.subn(r'(?m)^[ \t]*#?[ \t]*run_dir[ \t]*=.*$', f'run_dir = "{run_dir}"', text, count=1) + assert n == 1, text + with open(d.config, "w") as f: + f.write(text) + _append(d.config, '[backup.store]\nkind = "filesystem"\n[backup.store.filesystem]\npath = "rel-backups"\n') + res = subprocess.run( + [EXTENDDB_BINARY, "serve", "--config", d.config], cwd=d.dir, capture_output=True, text=True, + stdin=subprocess.DEVNULL, timeout=60, + ) + try: + assert res.returncode == 0, res.stdout + res.stderr + assert f"backup store: filesystem path {os.path.join(d.dir, 'rel-backups')}" in res.stdout, res.stdout + _wait_health(d.port, lambda h: h.get("components", {}).get("backup_store", {}).get("status") == "healthy") + assert os.path.isfile(os.path.join(d.dir, "rel-backups", "extenddb-store.json")) + assert not os.path.exists("/rel-backups") + finally: + # Stop the daemon whatever the outcome above; a daemon that never + # started has no PID file and `stop` says so. + subprocess.run( + [EXTENDDB_BINARY, "stop", "--config", d.config], cwd=d.dir, capture_output=True, text=True, + stdin=subprocess.DEVNULL, timeout=60, + ) + deadline = time.monotonic() + 30 + while time.monotonic() < deadline: + try: + _health(d.port) + except Exception: # noqa: BLE001 - the listener is gone + break + time.sleep(0.25) + + def test_a_catalog_re_created_after_destroy_warns_about_the_old_default_root(self, sqlite_deployment, tmp_path): + """`destroy` then `init --overwrite` on the same directory: the default + root's identity file names the old catalog. The server starts with the + store unconfigured and a warning naming the file (not degraded, not a + startup error), CreateBackup keeps the in-catalog path, and `destroy`'s + output said to remove the file.""" + d = sqlite_deployment + server = _Server(d.config, str(tmp_path / "serve.log")) + try: + _wait_health(d.port, lambda h: "backup_store" in h.get("components", {}), server=server) + finally: + server.stop() + identity = os.path.join(d.dir, "backups", "extenddb-store.json") + with open(identity) as f: + old_catalog = json.load(f)["catalog_id"] + res = d.run("destroy", "--yes") + assert res.returncode == 0, res.stdout + res.stderr + assert "needs that file removed" in res.stdout, res.stdout + assert os.path.isfile(identity) + env = os.environ.copy() + env["EXTENDDB_ADMIN_PASSWORD"] = ADMIN_PASSWORD + subprocess.run( + [EXTENDDB_BINARY, "init", "--backend", "sqlite", "--config", d.config, "--overwrite"], + cwd=d.dir, capture_output=True, text=True, stdin=subprocess.DEVNULL, timeout=60, check=True, env=env, + ) + lh._patch_config_port(d.config, d.port) + server = _Server(d.config, str(tmp_path / "serve2.log")) + try: + health = _wait_health(d.port, lambda h: "components" in h, server=server) + assert health == {"status": "healthy", "components": {}}, health + log = server.log_text() + assert "failed its check" in log and "extenddb-store.json names catalog" in log, log + assert old_catalog in log and "destroy" in log and "unconfigured" in log, log + creds = _provision(d.port, d.cert, ADMIN_PASSWORD) + ddb = _client(d.port, creds) + _create_table(ddb, "Music") + details = ddb.create_backup(TableName="Music", BackupName="nightly")["BackupDetails"] + assert details["BackupStatus"] in ("CREATING", "AVAILABLE") + assert "configure [backup.store] to write backups to a backup store" in server.log_text() + finally: + server.stop() + + def test_backup_purge_deletes_each_rows_prefix_and_row_and_leaves_the_rest(self, sqlite_deployment, tmp_path): + """Rows of every status seeded in the catalog with objects under their + prefixes: purge deletes each row's objects and the row outright; a + prefix the catalog has no row for, the identity file and `exports/` + stay; `backup list` shows the rows before and none after. + """ + try: + # The catalog's schema needs a SQLite newer than some hosts' + # stdlib module links; the wheel carries a current library. + import pysqlite3 as sqlite3 + except ImportError: + import sqlite3 + + d = sqlite_deployment + # The shortest window the config accepts (three of the 15 s the jobs + # heartbeat at), so a purge that waited on a dead job would fail + # this test in 90 s rather than the default 600. + _append(d.config, "[backup]\nstale_job_after_secs = 45\n") + server = _Server(d.config, str(tmp_path / "serve.log")) + try: + _wait_health(d.port, lambda h: "backup_store" in h.get("components", {}), server=server) + finally: + server.stop() + root = os.path.join(d.dir, "backups") + with open(os.path.join(root, "extenddb-store.json")) as f: + store_id = json.load(f)["store_id"] + source_table = json.dumps({ + "table_name": "Music", "table_id": "t-1", "table_arn": "arn:aws:dynamodb:us-east-1:123456789012:table/Music", + "key_schema": [{"AttributeName": "pk", "KeyType": "HASH"}], + "attribute_definitions": [{"AttributeName": "pk", "AttributeType": "S"}], + "billing_mode": "PAY_PER_REQUEST", "provisioned_throughput": None, "on_demand_throughput": None, + "global_secondary_indexes": [], "local_secondary_indexes": [], "vector_indexes": None, + "stream_specification": None, "time_to_live": None, "sse_description": None, + "item_count": 0, "table_size_bytes": 0, "table_creation_date_time": 0.0, + }) + account = "123456789012" + rows = [ + ("01791976431438-00000001", "AVAILABLE"), + ("01791976431439-00000002", "FAILED"), + ("01791976431440-00000003", "DELETING"), + # A CREATING row whose owner is dead (its heartbeat is old). + ("01791976431441-00000004", "CREATING"), + # A FAILED and a CREATING row whose heartbeats are young (five + # seconds old): the stopped server's jobs wrote them, and under + # the serve lock those jobs are dead, so the command neither + # waits on them nor leaves them. + ("01791976431442-00000005", "FAILED"), + ("01791976431443-00000006", "CREATING"), + ] + young = {"01791976431442-00000005", "01791976431443-00000006"} + + def prefix(backup_id): + return os.path.join(root, account, "Music", backup_id) + + def private_dirs(path): + # Every level 0700: the store refuses a directory under its root + # that another user can reach (`makedirs` leaves the intermediate + # levels at the umask's mode). + parts = [] + while not os.path.isdir(path): + parts.append(path) + path = os.path.dirname(path) + for p in reversed(parts): + os.mkdir(p, 0o700) + + conn = sqlite3.connect(d.db_path, timeout=10) + try: + for backup_id, status in rows: + arn = f"arn:aws:dynamodb:us-east-1:{account}:table/Music/backup/{backup_id}" + heartbeat = ( + "strftime('%Y-%m-%dT%H:%M:%fZ', 'now', '-5 seconds')" if backup_id in young + else "'2026-10-01T00:00:00.000Z'" + ) + conn.execute( + "INSERT INTO backup_index (account_id, backup_id, table_name, backup_arn, backup_name, status, " + "store_kind, store_root, store_id, size_bytes, item_count, created_at, source_table_details_json, " + "owner_generation, heartbeat_at) VALUES (?, ?, 'Music', ?, 'nightly', ?, 'filesystem', ?, ?, 1, 1, " + f"'2026-10-01T00:00:00.000Z', ?, 7, {heartbeat})", + (account, backup_id, arn, status, root, store_id, source_table), + ) + # A restore the stopped server left RUNNING (its target CREATING): + # under the serve lock its owner is dead, and the command resolves + # it before the wait instead of waiting on it. + conn.execute( + "INSERT INTO accounts (account_id, account_name) SELECT ?, 'purge-test' WHERE NOT EXISTS " + "(SELECT 1 FROM accounts WHERE account_id = ?)", + (account, account), + ) + conn.execute( + "INSERT INTO tables (account_id, table_name, key_schema, attribute_definitions, table_status, " + "table_arn, table_id) VALUES (?, 'Restored', '[]', '[]', 'CREATING', " + "'arn:aws:dynamodb:us-east-1:123456789012:table/Restored', 'tid-restored')", + (account,), + ) + conn.execute( + "INSERT INTO table_restores (table_id, source_backup_arn, restore_date_time, status, kind, " + "owner_generation, heartbeat_at) VALUES ('tid-restored', " + "'arn:aws:dynamodb:us-east-1:123456789012:table/Music/backup/01791976431400-00000000', " + "'2026-10-01T00:00:00.000Z', 'RUNNING', 'legacy', 9, '2026-10-01T00:00:00.000Z')", + ) + conn.commit() + finally: + conn.close() + for backup_id, status in rows: + private_dirs(os.path.join(prefix(backup_id), "data")) + # A CREATING backup has its data files and no manifest yet (the + # writer puts the manifest last), so nothing adopts a prefix a + # failure between purge's two steps leaves. + names = ["data/00001.json.gz"] if status == "CREATING" else ["extenddb-manifest.json", "data/00001.json.gz"] + for name in names: + with open(os.path.join(prefix(backup_id), name), "wb") as f: + f.write(b"x") + # A prefix with no row (another catalog's, or a copy) and exports/. + orphan = prefix("01791976431499-0000beef") + private_dirs(orphan) + with open(os.path.join(orphan, "extenddb-manifest.json"), "wb") as f: + f.write(b"x") + private_dirs(os.path.join(root, "exports", "e-1")) + with open(os.path.join(root, "exports", "e-1", "manifest-summary.json"), "wb") as f: + f.write(b"x") + + res = d.run("backup", "list") + assert res.returncode == 0, res.stdout + res.stderr + for backup_id, status in rows: + assert backup_id in res.stdout and status in res.stdout, res.stdout + assert f"store_root={root}" in res.stdout and f"store_id={store_id}" in res.stdout, res.stdout + assert "6 row(s)" in res.stdout, res.stdout + res = d.run("backup", "list", "--failed") + assert res.returncode == 0 and "2 row(s)" in res.stdout and "FAILED" in res.stdout, res.stdout + + # The dry run prints the plan, changes nothing and exits 0. + res = d.run("backup", "purge") + assert res.returncode == 0, res.stdout + res.stderr + assert "Dry run: nothing was deleted" in res.stdout and "--confirm" in res.stdout, res.stdout + for backup_id, _ in rows: + assert os.path.isfile(os.path.join(prefix(backup_id), "data", "00001.json.gz")), "nothing deleted without --confirm" + # `--config` after the subcommand is accepted too. + res = subprocess.run( + [EXTENDDB_BINARY, "backup", "purge", "--config", d.config], cwd=d.dir, capture_output=True, text=True, + stdin=subprocess.DEVNULL, timeout=120, + ) + assert res.returncode == 0 and "Dry run" in res.stdout, res.stdout + res.stderr + + res = d.run("backup", "purge", "--confirm") + assert res.returncode == 0, res.stdout + res.stderr + assert "6 backup(s) deleted, 0 left" in res.stdout, res.stdout + # The interrupted restore was resolved before the wait, which then + # had nothing to wait for (the default window would be 600 s); the + # young FAILED and CREATING rows were neither waited on (the window + # is zero under the serve lock) nor left. + assert "removed the restore target Restored" in res.stdout, res.stdout + assert "Waiting for running jobs" not in res.stdout, "purge waited on a dead job: " + res.stdout + for backup_id, _ in rows: + assert not os.path.exists(prefix(backup_id)), f"{backup_id}'s prefix is gone" + assert f"deleted arn:aws:dynamodb:us-east-1:{account}:table/Music/backup/{backup_id}" in res.stdout + assert os.path.isfile(os.path.join(orphan, "extenddb-manifest.json")), "a prefix with no row stays" + assert os.path.isfile(os.path.join(root, "exports", "e-1", "manifest-summary.json")), "exports/ stays" + assert os.path.isfile(os.path.join(root, "extenddb-store.json")), "the identity file stays" + out = res.stdout + res.stderr + assert "extenddb::audit" in out and out.count("deleted backup objects and catalog row") == 6, out + conn = sqlite3.connect(d.db_path, timeout=10) + try: + assert conn.execute("SELECT count(*) FROM backup_index").fetchone()[0] == 0, "rows gone, no tombstones" + assert conn.execute("SELECT count(*) FROM backup_maintenance").fetchone()[0] == 0, "the claim is released" + assert conn.execute("SELECT count(*) FROM tables WHERE table_name = 'Restored'").fetchone()[0] == 0, \ + "the dead restore's target is gone" + finally: + conn.close() + res = d.run("backup", "list") + assert res.returncode == 0 and "(none)" in res.stdout, res.stdout + + # The wait's limit and its refusal: purge waits the window twice, + # refuses naming the row, releases the claim and deletes nothing. + # The lever is a FAILED row whose heartbeat is stamped 120 s ahead + # of the clock: under the zero window the serve lock gives (every + # owner is dead, as the first half of this test showed on a 5 s old + # heartbeat) only a heartbeat in the future still reads as live, and + # no stopped server on this host writes one. The rule the wait + # applies on the other backends, a FAILED row with a heartbeat + # within the window held while a job that has not yet seen its + # claim lost may still write under its prefix, is tested on + # PostgreSQL in `crates/storage-postgres/tests/backup_restore.rs` + # (`the_wait_passes_a_failed_record_and_holds_on_a_failed_row_with_a_live_heartbeat`). + live_failed = "01791976431450-00000005" + conn = sqlite3.connect(d.db_path, timeout=10) + try: + conn.execute( + "INSERT INTO backup_index (account_id, backup_id, table_name, backup_arn, backup_name, status, " + "store_kind, store_root, store_id, size_bytes, item_count, created_at, source_table_details_json, " + "owner_generation, heartbeat_at) VALUES (?, ?, 'Music', ?, 'nightly', 'FAILED', 'filesystem', ?, ?, 1, 1, " + "'2026-10-01T00:00:00.000Z', ?, 7, strftime('%Y-%m-%dT%H:%M:%fZ', 'now', '+120 seconds'))", + (account, live_failed, f"arn:aws:dynamodb:us-east-1:{account}:table/Music/backup/{live_failed}", root, + store_id, source_table), + ) + conn.commit() + finally: + conn.close() + private_dirs(prefix(live_failed)) + with open(os.path.join(prefix(live_failed), "extenddb-manifest.json"), "wb") as f: + f.write(b"x") + started = time.monotonic() + res = d.run("backup", "purge", "--confirm") + waited = time.monotonic() - started + assert res.returncode != 0, res.stdout + res.stderr + out = res.stdout + res.stderr + assert "Waiting for running jobs to finish: the FAILED backup" in out and live_failed in out, out + assert "jobs were still running after waiting" in out and "(2 * stale_job_after_secs)" in out, out + assert waited >= 90, f"the wait is twice the window: {waited:.1f} s" + assert os.path.isfile(os.path.join(prefix(live_failed), "extenddb-manifest.json")), "nothing deleted" + conn = sqlite3.connect(d.db_path, timeout=10) + try: + assert conn.execute("SELECT count(*) FROM backup_index").fetchone()[0] == 1 + assert conn.execute("SELECT count(*) FROM backup_maintenance").fetchone()[0] == 0, "the claim is released" + conn.execute("DELETE FROM backup_index") + conn.commit() + finally: + conn.close() + import shutil + shutil.rmtree(prefix(live_failed)) + + # A signal while the command waits releases the claim at once and + # deletes nothing; a SIGKILL leaves the claim's row, which the next + # server start clears, so CreateBackup is not refused for the + # window. The waited-on row's heartbeat is 300 s ahead (the zero + # window under the serve lock cannot pass it). + waited_on = "01791976431460-00000007" + conn = sqlite3.connect(d.db_path, timeout=10) + try: + conn.execute( + "INSERT INTO backup_index (account_id, backup_id, table_name, backup_arn, backup_name, status, " + "store_kind, store_root, store_id, size_bytes, item_count, created_at, source_table_details_json, " + "owner_generation, heartbeat_at) VALUES (?, ?, 'Music', ?, 'nightly', 'FAILED', 'filesystem', ?, ?, 1, 1, " + "'2026-10-01T00:00:00.000Z', ?, 7, strftime('%Y-%m-%dT%H:%M:%fZ', 'now', '+300 seconds'))", + (account, waited_on, f"arn:aws:dynamodb:us-east-1:{account}:table/Music/backup/{waited_on}", root, + store_id, source_table), + ) + conn.commit() + finally: + conn.close() + private_dirs(prefix(waited_on)) + with open(os.path.join(prefix(waited_on), "extenddb-manifest.json"), "wb") as f: + f.write(b"x") + + def waiting_purge(): + proc = subprocess.Popen( + [EXTENDDB_BINARY, "backup", "--config", d.config, "purge", "--confirm"], cwd=d.dir, + stdout=subprocess.PIPE, stderr=subprocess.STDOUT, stdin=subprocess.DEVNULL, text=True, + ) + lines = [] + deadline = time.monotonic() + 60 + while time.monotonic() < deadline: + line = proc.stdout.readline() + if not line: + break + lines.append(line) + if "Waiting for running jobs to finish" in line: + return proc, lines + proc.kill() + raise AssertionError("purge did not reach its wait: " + "".join(lines)) + + def claim_rows(): + conn = sqlite3.connect(d.db_path, timeout=10) + try: + return conn.execute("SELECT count(*) FROM backup_maintenance").fetchone()[0] + finally: + conn.close() + + proc, lines = waiting_purge() + assert claim_rows() == 1, "the claim is held during the wait" + signalled = time.monotonic() + proc.send_signal(signal.SIGTERM) + rest = proc.communicate(timeout=30)[0] + out = "".join(lines) + rest + released_after = time.monotonic() - signalled + assert proc.returncode != 0, out + assert "SIGTERM received" in out and "stopped before any deletion: SIGTERM received while waiting" in out, out + assert "The maintenance claim is released" in out, out + assert released_after < 5, f"the signal ended the wait at once, not at the limit: {released_after:.1f} s" + assert claim_rows() == 0, "the claim is released on SIGTERM" + assert os.path.isfile(os.path.join(prefix(waited_on), "extenddb-manifest.json")), "nothing deleted" + + proc, lines = waiting_purge() + proc.kill() + proc.communicate(timeout=30) + assert claim_rows() == 1, "a killed purge leaves its claim's row" + # The server starts on the file and clears the row; the first + # CreateBackup is accepted at once (it would be LimitExceeded for + # the 45 s window otherwise). + server = _Server(d.config, str(tmp_path / "serve-after-kill.log")) + try: + _wait_health(d.port, lambda h: "backup_store" in h.get("components", {}), server=server) + assert claim_rows() == 0, "the server start cleared the left-behind claim" + assert "maintenance claim held by extenddb backup purge" in server.log_text() \ + and "was left behind; cleared" in server.log_text(), server.log_text() + creds = _provision(d.port, d.cert, ADMIN_PASSWORD) + ddb = _client(d.port, creds) + _create_table(ddb, "Music") + details = ddb.create_backup(TableName="Music", BackupName="after-kill")["BackupDetails"] + assert details["BackupStatus"] in ("CREATING", "AVAILABLE"), details + assert "LimitExceededException" not in server.log_text(), server.log_text() + finally: + server.stop() + conn = sqlite3.connect(d.db_path, timeout=10) + try: + conn.execute("DELETE FROM backup_index") + conn.commit() + finally: + conn.close() + shutil.rmtree(prefix(waited_on)) + + # The server starts again against the purged root, healthy. + server = _Server(d.config, str(tmp_path / "serve2.log")) + try: + health = _wait_health(d.port, lambda h: "backup_store" in h.get("components", {}), server=server) + assert health["components"]["backup_store"]["status"] == "healthy", health + finally: + server.stop() + + def test_backup_list_and_purge_on_an_empty_store_and_the_serve_lock(self, sqlite_deployment, tmp_path): + d = sqlite_deployment + # Create the default root with a server once, then stop it. + # Before any server: `list` is a read that creates neither the default + # root nor its identity file, and says the root does not exist yet. + res = d.run("backup", "list") + assert res.returncode == 0, res.stdout + res.stderr + assert "does not exist yet" in res.stdout and "(none)" in res.stdout, res.stdout + assert not os.path.exists(os.path.join(d.dir, "backups")), "a read creates nothing" + # The dry run is the same read: it prints an empty plan naming the + # reason and exits 0. + res = d.run("backup", "purge") + assert res.returncode == 0, res.stdout + res.stderr + assert "Store: none (the default root" in res.stdout and "does not exist yet" in res.stdout, res.stdout + assert "(none)" in res.stdout and "Dry run: nothing was deleted" in res.stdout, res.stdout + assert not os.path.exists(os.path.join(d.dir, "backups")), "a read creates nothing" + # The root created by hand, empty: `list` and the dry run read it and + # leave it without an identity file. + os.mkdir(os.path.join(d.dir, "backups"), 0o700) + res = d.run("backup", "list") + assert res.returncode == 0, res.stdout + res.stderr + assert "has no extenddb-store.json yet" in res.stdout and "(none)" in res.stdout, res.stdout + res = d.run("backup", "purge") + assert res.returncode == 0, res.stdout + res.stderr + assert "Dry run: nothing was deleted" in res.stdout and "no extenddb-store.json yet" in res.stdout, res.stdout + assert not os.path.exists(os.path.join(d.dir, "backups", "extenddb-store.json")), "a read creates nothing" + server = _Server(d.config, str(tmp_path / "serve.log")) + try: + _wait_health(d.port, lambda h: "backup_store" in h.get("components", {}), server=server) + # While the server runs, `list` and the dry run are reads beside + # it (no serve lock); `purge --confirm` opens the catalog as a + # server does and refuses. + res = d.run("backup", "list") + assert res.returncode == 0, res.stdout + res.stderr + assert "backups in the filesystem store" in res.stdout and "(none)" in res.stdout, res.stdout + res = d.run("backup", "purge") + assert res.returncode == 0, res.stdout + res.stderr + assert "Dry run: nothing was deleted" in res.stdout, res.stdout + res = d.run("backup", "purge", "--confirm") + assert res.returncode != 0, res.stdout + res.stderr + assert "another extenddb process" in (res.stdout + res.stderr), res.stdout + res.stderr + assert server.proc.poll() is None, "the server kept running" + finally: + server.stop() + res = d.run("backup", "list") + assert res.returncode == 0, res.stdout + res.stderr + assert "backups in the filesystem store" in res.stdout and "(none)" in res.stdout, res.stdout + res = d.run("backup", "list", "--failed") + assert res.returncode == 0 and "(none)" in res.stdout, res.stdout + res.stderr + res = d.run("backup", "purge") + assert res.returncode == 0, res.stdout + res.stderr + assert "Dry run: nothing was deleted" in res.stdout, res.stdout + res = d.run("backup", "purge", "--confirm") + assert res.returncode == 0, res.stdout + res.stderr + assert "0 backup(s) deleted" in res.stdout, res.stdout + assert "Took the maintenance claim" in res.stdout, res.stdout + assert os.path.isfile(os.path.join(d.dir, "backups", "extenddb-store.json")) + # `destroy` names the store as left untouched. + res = d.run("destroy", "--yes") + assert res.returncode == 0, res.stdout + res.stderr + assert "backup store ([backup.store]) is left untouched" in res.stdout, res.stdout + assert os.path.isfile(os.path.join(d.dir, "backups", "extenddb-store.json")) + + +# --- PostgreSQL ----------------------------------------------------------------------- + + +def _pg_build() -> bool: + if not os.path.isfile(EXTENDDB_BINARY): + return False + res = subprocess.run([EXTENDDB_BINARY, "version"], capture_output=True, text=True, timeout=30) + return "(postgres)" in res.stdout + + +@pytest.fixture() +def pg_env(request): + if not _pg_build(): + pytest.skip("EXTENDDB_BINARY is not a PostgreSQL build") + # `cli_env` requires PostgreSQL and a binary; it is requested only once + # the build is known to be the PostgreSQL one, so the other build skips. + cli_env = request.getfixturevalue("cli_env") + result = lh._run_extenddb( + "init", *lh._init_args(cli_env), config=cli_env["config_path"], + env_override={"EXTENDDB_ADMIN_PASSWORD": ADMIN_PASSWORD}, + ) + assert result.returncode == 0, result.stdout + result.stderr + lh._patch_config_port(cli_env["config_path"], cli_env["port"]) + return cli_env + + +class TestPostgres: + def test_without_a_store_there_is_no_component_and_create_backup_warns(self, pg_env, tmp_path): + server = _Server(pg_env["config_path"], str(tmp_path / "serve.log")) + try: + health = _wait_health(pg_env["port"], lambda h: "components" in h, server=server) + assert health == {"status": "healthy", "components": {}}, health + log = server.log_text() + assert "backup store: none configured" in log, log + creds = _provision(pg_env["port"], os.path.join(pg_env["tls_dir"], "cert.pem"), ADMIN_PASSWORD) + ddb = _client(pg_env["port"], creds) + _create_table(ddb, "Music") + arn = ddb.create_backup(TableName="Music", BackupName="nightly")["BackupDetails"]["BackupArn"] + assert "configure [backup.store] to write backups to a backup store" in server.log_text() + # The in-catalog backup is served. + assert ddb.describe_backup(BackupArn=arn)["BackupDescription"]["BackupDetails"]["BackupArn"] == arn + listed = ddb.list_backups(TableName="Music")["BackupSummaries"] + assert [b["BackupArn"] for b in listed] == [arn] + finally: + server.stop() + + def test_an_s3_store_without_a_spool_stops_startup(self, pg_env, tmp_path): + _append(pg_env["config_path"], '[backup.store]\nkind = "s3"\n[backup.store.s3]\nbucket = "b"\n') + server = _Server(pg_env["config_path"], str(tmp_path / "serve.log"), env=_s3_env()) + try: + assert server.wait_exit() != 0 + log = server.log_text() + assert "backup.spool_path is required" in log, log + finally: + server.stop() + + def test_a_filesystem_store_is_healthy_and_a_bad_bucket_name_stops_startup(self, pg_env, tmp_path): + root = tmp_path / "backups" + root.mkdir(mode=0o700) + _append( + pg_env["config_path"], + f'[backup.store]\nkind = "filesystem"\n[backup.store.filesystem]\npath = "{root}"\n', + ) + server = _Server(pg_env["config_path"], str(tmp_path / "serve.log")) + try: + health = _wait_health(pg_env["port"], lambda h: "components" in h, server=server) + assert health["components"]["backup_store"] == {"status": "healthy", "kind": "filesystem"} + with open(root / "extenddb-store.json") as f: + assert json.load(f)["backend"] == "postgres" + finally: + server.stop() + # A bucket name the SDK routes to S3 Express is a configuration error + # naming the bucket. + with open(pg_env["config_path"]) as f: + text = f.read() + text = text[: text.index("[backup.store]")] + with open(pg_env["config_path"], "w") as f: + f.write(text) + _append( + pg_env["config_path"], + f'[backup]\nspool_path = "{tmp_path / "spool"}"\n' + '[backup.store]\nkind = "s3"\n[backup.store.s3]\nbucket = "my-bucket--usw2-az1--x-s3"\n', + ) + server = _Server(pg_env["config_path"], str(tmp_path / "serve2.log"), env=_s3_env()) + try: + assert server.wait_exit() != 0 + log = server.log_text() + assert "cannot be opened" in log and "my-bucket--usw2-az1--x-s3" in log, log + finally: + server.stop()