From d4a9eb798695010b4f6f4a7dd4ddcde3939c97c2 Mon Sep 17 00:00:00 2001 From: Jacob Stephens Date: Tue, 10 Nov 2026 22:47:17 -0500 Subject: [PATCH 1/8] Wait out GitHub quota exhaustion in running Commands (#1997) --- docs/guide.md | 8 + site/docs/index.html | 4 + site/how-it-works/index.html | 1 + src/github.rs | 30 +++- src/github/check_read.rs | 2 + src/github/execution_tests.rs | 6 +- src/github/quota.rs | 76 +++++---- src/github/quota/command.rs | 153 ++++++++++++++++++ src/github/quota/wait.rs | 24 ++- src/logs.rs | 20 +++ src/logs/command_record.rs | 3 + src/logs/reading.rs | 27 +++- src/logs/reading/health.rs | 6 +- src/logs/reading/tests.rs | 33 ++++ src/logs/recording.rs | 7 + src/logs/recording/tests.rs | 29 ++++ src/notification.rs | 7 + tests/console.rs | 24 +++ tests/quota_wait.rs | 292 ++++++++++++++++++++++++++++++++++ 19 files changed, 702 insertions(+), 50 deletions(-) create mode 100644 src/github/quota/command.rs create mode 100644 tests/quota_wait.rs diff --git a/docs/guide.md b/docs/guide.md index e3824b73d..76a3c5797 100644 --- a/docs/guide.md +++ b/docs/guide.md @@ -45,6 +45,8 @@ From a clone of the issue's repository, `thirdshift `: 7. In a **Ticket run** that Self-merges (`thirdshift merge `), does the **Self-merge**: once the pull request is open, ready for review, mergeable and green, thirdshift merges it into the Base branch with a merge commit, on exactly the head commit whose CI it watched (`gh pr merge --merge --match-head-commit `). Just before each attempt, it reads the PR's `reviewDecision`: `REVIEW_REQUIRED` or `CHANGES_REQUESTED` ends the Ticket run as **review required**, a policy refusal, without attempting the merge or taking another Repair round. An empty or `APPROVED` decision, or a failed read, keeps the existing merge behavior. It never uses GitHub's auto-merge ([ADR-0004](adr/0004-self-merge-by-thirdshift-not-github-auto-merge.md)). A merge that fails goes back round step 6, within the same budgets, and is tried again on the new green head. If that round finds nothing to fix, the refusal is a **policy refusal**, such as merge commits being disallowed or a review being required. After the merge, thirdshift deletes the Issue branch on `origin`, and closes the issue if it is still open, with the comment `Closed by #, merged into by a thirdshift Ticket run with Self-merge.` GitHub closes it on its own only for a merge into the repository's default branch, and then thirdshift leaves it alone. Once the issue is closed, either way, thirdshift takes `in-progress` off it, [removing the Claim](#when-the-claim-ends). 8. Cleans up: removes the worktree, the local Issue branch and the temporary directory the Factory skills were written out to, whether the Ticket run succeeded or not. The one exception is a **Failed ticket run** whose work could not be pushed: see below. +CI watch quota exhaustion follows the [Command's Quota wait policy](#delivery-and-pull-request-identity): reset plus one minute and 0–60 seconds of jitter, within a shared two-hour cap. Secondary limits retry after 60 seconds, at most five times. Neither is a CI failure or a reason to start a Repair while retries remain. + ### Worktree ownership and cleanup A factory-owned checkout, on an **Issue branch** for a **Ticket run** or detached for an **Architecture review**. @@ -63,6 +65,10 @@ See [ADR-0014](adr/0014-successful-review-acquisition-authorizes-stale-disposal. What a **Ticket run**, or a **Spec run** for its **Spec PR**, does from its worktree once its work begins. It runs the opening session (the implement session, or the **Spec review**), pushes, marks the pull request ready, keeps it mergeable and green through the **Repair loop**, and, in a **Ticket run** that Self-merges, does the **Self-merge**. +When thirdshift's own GitHub reads or writes hit primary quota exhaustion, the running **Command** records a **Quota wait**, waits until reset plus one minute and a random 0–60 seconds, then retries the same call. This includes CI watch and Delivery's pull request updates, labels, comments and Self-merge, across Ticket, Spec, Review, Fix, Security and Triage runs, Custom processes and work a Pass dispatched. The Command and its child Ticket runs share a two-hour total wait cap; a call that would exceed it fails as before. A secondary-limit failure instead waits 60 seconds and retries up to five times, without recording a Quota wait. + +While waiting, `status`, `show` and the Console show `Quota wait until