diff --git a/site/blog/choose-the-check.html b/site/blog/choose-the-check.html new file mode 100644 index 0000000..70c4ce1 --- /dev/null +++ b/site/blog/choose-the-check.html @@ -0,0 +1,158 @@ + + + + + +How to pick the test command that checks your AI agent's work + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+ Time Machine + + + Join the waitlist +
+ +
+ +
+
+
+

Blog4 min read

+

How to pick the test command that checks your AI agent's work

+

A save point is only as useful as the check behind it. Here is how to choose one that tells the truth.

+

The Time Machine team

+
+
+ +
+

Time Machine decides whether a save point works by running one command on it, in a separate copy. Exit 0 means it works. Anything else means it does not. That command is verify.command in timemachine.yml, and everything Time Machine says about "the last version that works" depends on it.

+ +

Start with what you already trust

+

The best check is the one you already run before you push. For most projects that is the test suite:

+
# timemachine.yml
+version: 1
+verify:
+  setup: "npm ci"
+  command: "npm test"
+  timeoutSeconds: 600
+  network: allow
+

tm init writes this file with a guessed check and install step, and keeps a file that already exists. Read what it guessed before you approve it. tm status shows whether a check is set and approved on this computer.

+

A few rules of thumb:

+
    +
  • It should fail when the app is broken. A check that only runs the linter will pass a version whose main function throws.
    • +
  • It should finish. The default time limit is 600 seconds, and a check that runs longer ends as ERROR. Raise verify.timeoutSeconds if your suite is slow, or pick a faster subset that still covers the core.
    • +
  • It should not need your secrets. Checks run with a clean environment: PATH, HOME, USER, LOGNAME, LANG, TMPDIR, SHELL, proxy and certificate settings, package manager settings, the keys you list in config.plain, and CI=1. A test that needs a real API key may fail here, because the key isn't passed in.
    • +
  • No tests yet? Use the build. npm run build or tsc --noEmit catches a lot of what an agent breaks. It is a weaker check, so treat its PASS as "it builds", not "it works".
    • +
+ +

Setup and the check are separate

+

verify.setup runs first, for example npm ci, and always has the network so it can install dependencies. It shares the time limit with the check. Keep installs in setup and keep the check itself to the test run.

+ +

When to turn the network off

+

With network: deny, the check runs with no network, so a test cannot pass just because a live service happened to answer. It needs a sandbox: sandbox-exec on macOS, bubblewrap on Linux. If there is no sandbox, Time Machine does not quietly run the check with the network on. It reports BLOCKED. tm init writes deny only when it finds a sandbox.

+ +

Reading the verdicts

+
    +
  • PASS: your check passed. Safe to recover to.
    • +
  • FAIL: your check failed. Run tm find-good and read the log path it shows.
    • +
  • INCOMPLETE: something saved is missing, so the check did not run. Often a file over the size limit (capture.maxFileMB, 5 MB by default).
    • +
  • BLOCKED: the check cannot run yet. For example, no verify.command is set, or the network is set to deny and no sandbox is available.
    • +
  • ERROR: the check crashed, timed out, or the save point failed its integrity check.
    • +
+

tm find-good keeps going past FAIL, INCOMPLETE and ERROR, and stops at the first PASS or at BLOCKED. A save point that was never checked is never shown as working.

+ +

If the check changes

+

The approval on your computer covers the exact setup, command and network setting. If any of them changes, for example after a git pull, the hooks stop until you run tm init again and approve the new check. A pulled change cannot run a command you never agreed to.

+ +
+

Sources

+
    +
  1. Time Machine guide: timemachine.yml reference, verdicts and BLOCKED reasons
    2. +
  2. Time Machine README: what happens when the check changes
    3. +
+
+
+
+ +
+ +
+
+
+

Build with
a way back.

+

We write to the list when the CLI is on npm. Nothing else.

+
+
+
+ +
+ + +
+
+ + +

+

We store your email, your consent, the page and time you joined, and referral codes. Never your IP address, only a one-way code kept about an hour to limit requests. See the privacy policy.

+
+
+
+
+ + + + + + + + + diff --git a/site/blog/index.html b/site/blog/index.html index 4ca968a..3828198 100644 --- a/site/blog/index.html +++ b/site/blog/index.html @@ -59,6 +59,7 @@

Notes fromNo. 04

Why a safety net makes people braver

Easy reversal, loss aversion, the peak end rule and the 66 day habit. The research behind the design, and why we refuse streaks.

6 min
  • No. 05

    Building Time Machine in the open: the plan, and why we will raise

    Where we are, how gates decide what comes next, and what raising money would pay for.

    5 min
  • No. 06

    git bisect or tm find-good: which one finds your last working version

    One finds the change that broke it. The other finds a version you can go back to. When to use which.

    4 min
    • +
  • No. 07

    How to pick the test command that checks your AI agent's work

    What makes a good check, when to turn the network off, and what each verdict means.

    4 min

    • All posts by the Time Machine team.

    diff --git a/site/sitemap.xml b/site/sitemap.xml index 793a5cd..fe1f846 100644 --- a/site/sitemap.xml +++ b/site/sitemap.xml @@ -12,4 +12,5 @@ https://time-machine.company/blog/why-a-safety-net-makes-people-braver https://time-machine.company/blog/building-in-the-open https://time-machine.company/blog/git-bisect-or-find-good + https://time-machine.company/blog/choose-the-check