From b62b296af764b7d1110fa0cbbd266fa0c8eec4b6 Mon Sep 17 00:00:00 2001 From: Lalit Maganti Date: Wed, 16 Sep 2026 14:02:43 +0100 Subject: [PATCH] docs: make the README a front page The README had grown to 409 lines, most of it reference material: GitHub Actions, remote machines, collection options, compiler details, the build systems the conformance suite covers, daemon and container troubleshooting, the trace format's compatibility rules, and self-hosting the UI. Anyone arriving to find out what Buildprof is had to scroll past all of it, and the features worth knowing about were buried among the parts that are only needed once. The front page now names each feature in a line of its own, with the flag that turns it on and a link to the detail: the process tree, file producers and consumers, compiler internals, recordings from CI, builds over SSH, lighter collection, and where the data goes. Installation folds each method away behind what it is called. The reference material moves into docs/ unchanged: usage.md, build-systems.md, troubleshooting.md and internals.md, with the README carrying an index. The /diagnose/ links the CLI prints are redirects, so they now point at the troubleshooting document, and the release metadata check reads the action's version example from its new home. --- .gitignore | 4 + README.md | 425 +++++++---------------- RELEASING.md | 8 +- docs/build-systems.md | 17 + docs/internals.md | 34 ++ docs/investigating-builds.md | 4 +- docs/troubleshooting.md | 78 +++++ docs/usage.md | 112 ++++++ infra/buildprof.lalitm.com/assemble-site | 12 +- infra/prepare-action-release | 6 +- justfile | 2 +- 11 files changed, 378 insertions(+), 324 deletions(-) create mode 100644 docs/build-systems.md create mode 100644 docs/internals.md create mode 100644 docs/troubleshooting.md create mode 100644 docs/usage.md diff --git a/.gitignore b/.gitignore index 5abb451..44a2af1 100644 --- a/.gitignore +++ b/.gitignore @@ -17,6 +17,10 @@ third_party/src/ /docs/* !/docs/investigating-builds.md !/docs/ripgrep-tutorial.md +!/docs/usage.md +!/docs/build-systems.md +!/docs/troubleshooting.md +!/docs/internals.md !/docs/assets/ /docs/assets/* !/docs/assets/ripgrep-release-clean.png diff --git a/README.md b/README.md index dccbdd0..e1cdd2e 100644 --- a/README.md +++ b/README.md @@ -13,25 +13,30 @@

Quick start · Try the demo · - Investigation guide · - Install + Features · + Install · + Documentation

## What is Buildprof? -Buildprof traces every process a Linux build launches and turns the recording -into an interactive timeline you can explore in the browser. Put `buildprof --` -in front of your build command to get started. - -- **See the whole build.** Find expensive commands, gaps in parallelism, and - work that starts unexpectedly late across Make, Ninja, CMake, Meson, Cargo, - Go, and shell scripts. -- **Follow the files.** Inspect commands, working directories, and exit statuses, - then follow inputs back to the processes that produced them. -- **Look inside compilers.** Optionally add internal timing data from Clang, - LLD, and nightly Rust alongside the process timeline. +Buildprof traces **every process** a Linux build launches and turns the +recording into an **interactive timeline** you can explore in the browser. Put +`buildprof --` in front of your build command to get started. + +- **See the whole build.** Every command the build ran, however it was + launched, with its command line, working directory, exit status, and place in + the process tree. +- **Follow the files.** File opens and renames are recorded alongside the + processes, so the recording links a file's producer to everything that + consumed it. That link is usually how you find the step waiting on work it + did not need. +- **Whatever your build system.** Buildprof follows processes rather than build + systems, so Make, Ninja, CMake, Meson, Cargo, Go, npm, Bazel and Buck2 all + work, as do the shell scripts, code generators and wrapper scripts they + launch. Recording requires Linux. You can explore recordings on any platform in the [web UI](https://buildprof.lalitm.com); trace data stays in your browser. @@ -67,9 +72,8 @@ buildprof -- make -j8 ``` Replace `make -j8` with your build command, such as `cargo build` or -`ninja -C out`. For Bazel, Gradle, Buck2, and other daemon build systems, see -[daemon builds](#bazel-gradle-and-other-daemon-build-systems); for Docker, see -[container builds](#builds-inside-docker-containers). +`ninja -C out`. Bazel, Gradle and Docker hand work to a daemon, which needs +[a little more care](docs/troubleshooting.md). ### 3. Explore the recording @@ -81,327 +85,130 @@ localhost. Nothing is ever uploaded. Start with the longest commands and gaps in parallelism. The [investigation guide](docs/investigating-builds.md) walks through finding bottlenecks, following file dependencies, and checking whether a change helped. -For builds over SSH, see [Builds on a remote machine](#builds-on-a-remote-machine). + +## Features + +- **Look inside compilers.** `--compiler-traces` records what a compiler did + internally, as per-thread phase tracks under the process that produced them, + so you can go from "this rustc took 40 seconds" to which part of it did. + ([details](docs/usage.md#compiler-details)) + - Clang `-ftime-trace`: parsing, template instantiation, optimisation. + - LLD `--time-trace`, when the build selects LLD explicitly. + - Nightly Rust self-profile data, per compiler query. +- **Record in CI.** The GitHub Action records a build and attaches the + recording to the job summary, including when the build fails, which is the + quickest way to find out why CI is slower than your laptop. + ([details](docs/usage.md#recording-in-github-actions)) +- **Record over SSH.** On a build host there is no browser to launch, so + Buildprof prints the port forward to run from your own machine and waits for + it. ([details](docs/usage.md#builds-on-a-remote-machine)) +- **Turn collection down.** `--no-file-events` keeps the process timeline and + skips filesystem interception entirely, for builds where that overhead + matters. ([details](docs/usage.md#collection-options)) +- **Keep your build to yourself.** buildprof.lalitm.com delivers the UI and + nothing else: your browser fetches the recording from localhost and processes + it in the page. Recordings do contain command lines and paths, so review one + before sending it to anyone. +- **Check the conformance suite.** Every change records a real build with each + supported build system and checks the resulting trace. + ([details](docs/build-systems.md)) ## Why use Buildprof? -Build tools generally explain only the work they manage themselves. Cargo -timings cannot break down an arbitrary `build.rs` script; Ninja cannot see -inside commands it launches; compiler traces describe one compiler invocation -rather than the build around it. +Build tools generally explain only the work they manage themselves: -Buildprof follows the complete process tree, so the same view includes the -build system, compilers, linkers, code generators, and arbitrary tools launched -along the way. You can see how their work fits together and where the build -spends its time. +- **Cargo timings** cannot break down an arbitrary `build.rs` script. +- **Ninja** cannot see inside the commands it launches. +- **Compiler traces** describe a single compiler invocation rather than the + build around it. + +Buildprof follows the complete process tree instead, so the build system, +compilers, linkers, code generators and whatever else the build launches all +land in one timeline, and you can see how their work fits together. ## Install Install on your Linux build machine using whichever method you prefer. -**Shell installer**: +
+ +Shell installer + +

```bash curl -fsSL https://buildprof.lalitm.com/install.sh | sh ``` +
-**Homebrew**: +
+ +Homebrew, mise, or Cargo + +

```bash brew install lalitmaganti/tap/buildprof -``` - -**mise**: - -```bash mise use -g github:LalitMaganti/buildprof +cargo install --locked buildprof # builds from source; needs Rust 1.91 or newer ``` +
-**Cargo** (builds from source; needs Rust 1.91 or newer): +
+ +Debian, Ubuntu, Fedora, and other .deb or .rpm distributions + +

-```bash -cargo install --locked buildprof -``` - -**Debian, Ubuntu, Fedora, and other `.deb` or `.rpm` distributions**: download -the package for your architecture from the -[latest release](https://github.com/LalitMaganti/buildprof/releases/latest) -and install it with `apt install ./buildprof_*.deb` or -`dnf install ./buildprof-*.rpm`. - -**Tarballs**: the same release page carries prebuilt binaries for x86_64 and -aarch64 Linux, both glibc and static musl. - -### Requirements - -**Recording builds is currently supported on Linux only.** The kernel or -container configuration must permit tracing child processes: Docker needs -`--cap-add SYS_PTRACE`, `kernel.yama.ptrace_scope` must be below 3, and -gVisor-style sandboxes cannot trace at all. Installing from source needs Rust -1.91 or newer. - -Existing recordings can be viewed on any platform in the -[web UI](https://buildprof.lalitm.com), without installing Buildprof. Viewing a -recording does not require the same operating system it was recorded on. - -## Usage - -### Recording a build - -Put `buildprof --` in front of the build command. Choose another output path -or disable automatic opening when needed: +Download the package for your architecture from the +[latest release](https://github.com/LalitMaganti/buildprof/releases/latest): ```bash -buildprof -o clean-build.buildprof --no-open -- ninja -C out -``` - -### Recording in GitHub Actions - -On a Linux runner, install your build tools, then record the build: - -```yaml -steps: - - uses: actions/checkout@v7 - - uses: LalitMaganti/buildprof@v0.2.6 - with: - command: | - cmake -S . -B build - cmake --build build -j2 +apt install ./buildprof_*.deb +dnf install ./buildprof-*.rpm ``` +
-The job summary links to the recording artifact, including when the build -fails. Download it and open it in [the web UI](https://buildprof.lalitm.com). -See [action inputs](action.yml) for version and retention settings; use a -unique `artifact-name` for each matrix entry. For upgrades, use release tags -or release SHAs with Dependabot (see [action releases](RELEASING.md#github-action-releases)). +
+ +Tarballs + +

-### Opening recordings +The [release page](https://github.com/LalitMaganti/buildprof/releases/latest) +carries prebuilt binaries for x86_64 and aarch64 Linux, both glibc and static +musl. The musl build is the one to mount into a container. +
-Open an existing recording later with: - -```bash -buildprof open clean-build.buildprof -``` - -buildprof.lalitm.com only delivers the UI itself. Your browser fetches the -recording from localhost and processes it entirely in the page; no trace data -leaves your machine. - -Because the page comes from buildprof.lalitm.com and the recording from -localhost, the browser asks once whether the site may access other apps and -services on this device. Allow it. If you block it, the trace never loads. To -recover, allow it again in the site settings next to the address bar, under -"Apps on device" in Chrome or "Access this device" in Firefox, then run -`buildprof open` again. - -Recordings contain command lines and filesystem paths. Review them before -sending them to anyone. - -### Builds on a remote machine - -Over SSH there is no browser to launch, so Buildprof prints the port forward -to run from your own machine instead and waits for the browser to fetch the -trace: - -```bash -ssh -L 9001:127.0.0.1:9001 user@buildhost -``` - -VS Code Remote and JetBrains Gateway forward the port automatically. The wait -gives up after ten minutes; adjust it with `--wait `, where `0` waits -forever. Alternatively, copy the recording to your own machine and open it -in the [web UI](https://buildprof.lalitm.com). No local installation is needed -for viewing. - -### Collection options - -Process creation, commands, and timing are always recorded. File opens and -renames are also recorded by default; to reduce overhead on builds with lots -of filesystem activity, disable that layer: - -```bash -buildprof --no-file-events -- make -j6 -``` - -The process timeline remains available, but file lists and producer/consumer -links are unavailable. This skips filesystem interception itself, rather than -collecting and discarding events. The UI identifies recordings made this way. - -### Compiler details - -Process timing is usually the right level for understanding a build. When a -particular compiler or linker invocation needs a closer look, enable compiler -tracing: - -```bash -buildprof --compiler-traces -- cargo build -``` - -Buildprof currently imports Clang `-ftime-trace`, explicitly selected LLD -`--time-trace`, and nightly Rust self-profile data. These events appear as a -summary of active compiler threads with expandable per-thread phase tracks. -Compiler tracing can be combined with process-only recording: - -```bash -buildprof --no-file-events --compiler-traces -- ninja -C build -``` - -A build which invokes Clang through an absolute path currently bypasses -compiler tracing. Buildprof will still record the compiler process, but its -Clang and LLD internal phases will be absent. - -Compiler tracing can also change compiler cache keys or turn cache hits into -misses. Existing Rust compiler wrappers remain in the invocation chain, but -cache preservation is not guaranteed in this mode. - -## Build systems - -Buildprof follows the process tree, so it does not need to understand the -build system. These are exercised by the conformance suite on every change: - -- **Make**: compiles, archiving, linking, and renamed outputs. -- **CMake with Ninja**: the configure step and the Ninja build. -- **Meson with Ninja**: the setup step and the Ninja build. -- **Cargo**: `rustc` invocations and linking; nightly self-profile data with - `--compiler-traces`. -- **Go**: compile, assemble, and link. -- **npm**: offline build scripts, generated JavaScript, and renamed outputs. -- **Bazel**: local actions and generated-file dependencies in batch mode. -- **Buck2**: local actions and artifact flow through an isolated build daemon. - -Anything else that runs as a child process is recorded the same way: shell -scripts, code generators, wrapper scripts, and tools launched by the build. - -## Troubleshooting - -Buildprof only records the process tree under the command it runs. Work -outside that tree is not recorded. It usually appears as one long command -with nothing underneath it. - - -### Bazel, Gradle, and other daemon build systems - -Buildprof cannot see work sent to an existing daemon or a remote executor. -An existing daemon leaves only the client in the recording. If the recorded -command starts the daemon, recording continues until the daemon exits. -Disable the daemon or stop it before and after the build: - -```sh -bazel shutdown && buildprof -- sh -c 'bazel build //... ; bazel shutdown' -buildprof -- ./gradlew --no-daemon build -buck2 kill && buildprof -- sh -c 'buck2 build //... ; buck2 kill' -sccache --stop-server && buildprof -- sh -c 'cargo build; sccache --stop-server' -``` - - -### Builds inside Docker containers - -`docker run` hands the container to the Docker daemon. The container's -processes are outside the recorded tree. You can record inside the container -or use Podman. - -#### Record inside the container - -Download the static musl build on the host and mount it into the container. -It runs in any Linux image without changing the image. Use `aarch64` instead -of `x86_64` for ARM images, including Docker Desktop on Apple Silicon: - -```sh -mkdir -p ~/.cache/buildprof-linux -curl -fsSL https://github.com/LalitMaganti/buildprof/releases/latest/download/buildprof-x86_64-unknown-linux-musl.tar.xz \ - | tar -xJ -C ~/.cache/buildprof-linux --strip-components=1 -``` - -Run the build through the mounted binary. Docker needs `--cap-add SYS_PTRACE` -to allow tracing. The recording goes in the mounted project directory: - -```sh -docker run --rm --cap-add SYS_PTRACE \ - -v ~/.cache/buildprof-linux:/opt/buildprof:ro \ - -v "$PWD:/src" -w /src my-build-image \ - /opt/buildprof/buildprof -o build.buildprof --no-open -- cargo build -buildprof open build.buildprof -``` - -Use `--no-open` because the host browser cannot reach the container's -localhost, where the recording would be served. Open the recording with -`buildprof open` on the host. To make the binary a permanent part of the image, -copy it in the Dockerfile. - - -#### Use Podman - -Podman has no daemon. Container processes are descendants of `podman run`, -so Buildprof records them without image changes. Podman accepts the same -`run` options as Docker: - -```sh -podman info > /dev/null -buildprof -- podman run --rm -v "$PWD:/src" -w /src my-build-image cargo build -``` - -> [!IMPORTANT] -> Run `podman info` before recording. The first Podman command after a reboot -> or `podman system migrate` sets up the rootless user namespace using setuid -> `newuidmap`. It cannot gain privileges while traced, causing -> `newuidmap: write to uid_map failed: Operation not permitted`. -> Any Podman command run outside the recording sets up the namespace. -> Later commands reuse it. - -Commands and file paths from inside the container are recorded as the -container sees them, such as `/src/target/...` rather than the host path. - -## How it works - -On Linux, Buildprof launches the command under `ptrace` and follows process -creation, execution, and exit through the complete descendant tree. A seccomp -filter lets it stop only for the filesystem operations it records instead of -paying the cost of intercepting every system call. - -The recorder writes a Perfetto protobuf trace directly. Perfetto provides the -storage format, query engine, and core timeline interactions; Buildprof adds -the build-specific view on top, including process ancestry, command types, -concurrency, file relationships, and optional compiler timing data. - -### Backwards compatibility - -Before 1.0, the CLI and the UI move in lockstep at the minor version: a -recording is meant to be viewed in a UI from the same 0.x series, and patch -releases never change the trace format. Every UI version stays deployed under -its own path, the CLI opens the one matching its version, and the UI links to -the matching series when it is handed a recording from a different one, so -nothing stops working, but the trace format may change between minor -releases. - -From 1.0 onward, the trace format is stable and compatibility is permanent: -any recording opens in every later UI, and newer UIs simply add features on -top of older recordings. - -### Self-hosting the UI - -Each release attaches `buildprof-ui-v.tar.zst`, the complete UI as -static files. Serve its contents from any web server and point the CLI at it: - -```bash -buildprof open --url https://ui.example.internal/v0.2.0 clean-build.buildprof -``` - -## Development - -See [CONTRIBUTING.md](CONTRIBUTING.md) for the layout and the UI workflow. -Run the complete conformance suite in the Linux development container: - -```bash -just bootstrap -just test -``` - -For a quick host-side check of formatting, lints, unit tests, and package -contents: +### Requirements -```bash -just release-check -``` +- **Recording needs Linux**, with a kernel and container configuration that + permits tracing child processes: + - Docker needs `--cap-add SYS_PTRACE`. + - `kernel.yama.ptrace_scope` must be below 3. + - gVisor-style sandboxes cannot trace at all. +- **Viewing needs nothing.** Recordings open on any platform in the + [web UI](https://buildprof.lalitm.com), whatever they were recorded on, with + Buildprof installed or not. +- **Building from source needs Rust 1.91 or newer.** + +## Documentation + +- Reading a recording: + - [Investigation guide](docs/investigating-builds.md): finding bottlenecks, + following dependencies, and checking whether a change helped. + - [Guided tour](docs/ripgrep-tutorial.md): the example recording, explained. +- Recording one: + - [Recording and opening builds](docs/usage.md): GitHub Actions, remote + machines, collection options, compiler details. + - [Build systems](docs/build-systems.md): what the conformance suite covers. + - [Troubleshooting](docs/troubleshooting.md): daemon build systems and + containers. +- Under the hood: + - [How it works](docs/internals.md): tracing, the trace format, its + compatibility rules, and self-hosting the UI. + - [CONTRIBUTING.md](CONTRIBUTING.md): building, testing, and the UI + workflow. ## License diff --git a/RELEASING.md b/RELEASING.md index 092c27b..eeaa69b 100644 --- a/RELEASING.md +++ b/RELEASING.md @@ -23,7 +23,7 @@ Also create the empty `homebrew-tap` repository. Re-record `examples/*.buildprof` with this version if the trace format changed (see `examples/README.md`). 2. Run `just release-prepare` to update the action's default recorder version - and README release-tag example from `Cargo.toml`. Run `cargo check` to + and the docs' release-tag example from `Cargo.toml`. Run `cargo check` to refresh `Cargo.lock`. Then run `just release-check`, commit, push, and wait for CI. 3. Tag and push: `git tag v0.2.2 && git push origin v0.2.2`. @@ -60,7 +60,7 @@ Manual follow-ups: ## GitHub Action releases The action ships under the crate's `vX.Y.Z` tags. `just release-prepare` -updates its recorder default and README example from `Cargo.toml`; CI and +updates its recorder default and the docs example from `Cargo.toml`; CI and release checks catch drift. Keep release tags fixed. Consumers can use tags or release SHAs with Dependabot for upgrade PRs. @@ -78,10 +78,10 @@ tests the released action's default recorder on x86_64 and ARM64 after publicati cannot express are edited in by hand, and `allow-dirty = ["ci"]` in `dist-workspace.toml` stops dist from overwriting them: -- the release metadata check validates the action default and README example; +- the release metadata check validates the action default and the docs example; - the GitHub release is created as a draft titled `Buildprof `; - `infra/release-notes` replaces the install section of dist's notes with - the README's commands; the changelog and the download table stay as dist + the documented commands; the changelog and the download table stay as dist wrote them. To upgrade dist: bump `cargo-dist-version`, run `dist generate` with diff --git a/docs/build-systems.md b/docs/build-systems.md new file mode 100644 index 0000000..d2264e9 --- /dev/null +++ b/docs/build-systems.md @@ -0,0 +1,17 @@ +# Build systems + +Buildprof follows the process tree, so it does not need to understand the +build system. These are exercised by the conformance suite on every change: + +- **Make**: compiles, archiving, linking, and renamed outputs. +- **CMake with Ninja**: the configure step and the Ninja build. +- **Meson with Ninja**: the setup step and the Ninja build. +- **Cargo**: `rustc` invocations and linking; nightly self-profile data with + `--compiler-traces`. +- **Go**: compile, assemble, and link. +- **npm**: offline build scripts, generated JavaScript, and renamed outputs. +- **Bazel**: local actions and generated-file dependencies in batch mode. +- **Buck2**: local actions and artifact flow through an isolated build daemon. + +Anything else that runs as a child process is recorded the same way: shell +scripts, code generators, wrapper scripts, and tools launched by the build. diff --git a/docs/internals.md b/docs/internals.md new file mode 100644 index 0000000..bbb5a61 --- /dev/null +++ b/docs/internals.md @@ -0,0 +1,34 @@ +# How Buildprof works + +On Linux, Buildprof launches the command under `ptrace` and follows process +creation, execution, and exit through the complete descendant tree. A seccomp +filter lets it stop only for the filesystem operations it records instead of +paying the cost of intercepting every system call. + +The recorder writes a Perfetto protobuf trace directly. Perfetto provides the +storage format, query engine, and core timeline interactions; Buildprof adds +the build-specific view on top, including process ancestry, command types, +concurrency, file relationships, and optional compiler timing data. + +## Backwards compatibility + +Before 1.0, the CLI and the UI move in lockstep at the minor version: a +recording is meant to be viewed in a UI from the same 0.x series, and patch +releases never change the trace format. Every UI version stays deployed under +its own path, the CLI opens the one matching its version, and the UI links to +the matching series when it is handed a recording from a different one, so +nothing stops working, but the trace format may change between minor +releases. + +From 1.0 onward, the trace format is stable and compatibility is permanent: +any recording opens in every later UI, and newer UIs simply add features on +top of older recordings. + +## Self-hosting the UI + +Each release attaches `buildprof-ui-v.tar.zst`, the complete UI as +static files. Serve its contents from any web server and point the CLI at it: + +```bash +buildprof open --url https://ui.example.internal/v0.2.0 clean-build.buildprof +``` diff --git a/docs/investigating-builds.md b/docs/investigating-builds.md index 4fbe6bb..cac4159 100644 --- a/docs/investigating-builds.md +++ b/docs/investigating-builds.md @@ -29,7 +29,7 @@ Look for long-running commands, gaps before compilation starts, and processes which keep running after everything else finishes. Click a process to inspect its command line, working directory, lifetime, and exit status. For example, a long linker invocation is a reason to inspect its LTO flags and then look -inside the linker with [compiler tracing](#compiler-details). +inside the linker with [compiler tracing](usage.md#compiler-details). A parent's lifetime includes time spent waiting for children: follow the process tree down to see which command is still running. Process duration @@ -160,7 +160,7 @@ inputs, and full argument list, including libraries and any LTO flags. Keep response files referenced by `@file` available too. Both `clang` and `clang++` must be on `PATH` for Buildprof to install its Clang wrappers. -See [compiler support and limitations](../README.md#compiler-details) for +See [compiler support and limitations](usage.md#compiler-details) for supported compilers and the effects on compiler caches. ## Check whether a change helped diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md new file mode 100644 index 0000000..54ced21 --- /dev/null +++ b/docs/troubleshooting.md @@ -0,0 +1,78 @@ +# Troubleshooting + +Buildprof only records the process tree under the command it runs. Work +outside that tree is not recorded. It usually appears as one long command +with nothing underneath it. + + +## Bazel, Gradle, and other daemon build systems + +Buildprof cannot see work sent to an existing daemon or a remote executor. +An existing daemon leaves only the client in the recording. If the recorded +command starts the daemon, recording continues until the daemon exits. +Disable the daemon or stop it before and after the build: + +```sh +bazel shutdown && buildprof -- sh -c 'bazel build //... ; bazel shutdown' +buildprof -- ./gradlew --no-daemon build +buck2 kill && buildprof -- sh -c 'buck2 build //... ; buck2 kill' +sccache --stop-server && buildprof -- sh -c 'cargo build; sccache --stop-server' +``` + + +## Builds inside Docker containers + +`docker run` hands the container to the Docker daemon. The container's +processes are outside the recorded tree. You can record inside the container +or use Podman. + +## Record inside the container + +Download the static musl build on the host and mount it into the container. +It runs in any Linux image without changing the image. Use `aarch64` instead +of `x86_64` for ARM images, including Docker Desktop on Apple Silicon: + +```sh +mkdir -p ~/.cache/buildprof-linux +curl -fsSL https://github.com/LalitMaganti/buildprof/releases/latest/download/buildprof-x86_64-unknown-linux-musl.tar.xz \ + | tar -xJ -C ~/.cache/buildprof-linux --strip-components=1 +``` + +Run the build through the mounted binary. Docker needs `--cap-add SYS_PTRACE` +to allow tracing. The recording goes in the mounted project directory: + +```sh +docker run --rm --cap-add SYS_PTRACE \ + -v ~/.cache/buildprof-linux:/opt/buildprof:ro \ + -v "$PWD:/src" -w /src my-build-image \ + /opt/buildprof/buildprof -o build.buildprof --no-open -- cargo build +buildprof open build.buildprof +``` + +Use `--no-open` because the host browser cannot reach the container's +localhost, where the recording would be served. Open the recording with +`buildprof open` on the host. To make the binary a permanent part of the image, +copy it in the Dockerfile. + + +## Use Podman + +Podman has no daemon. Container processes are descendants of `podman run`, +so Buildprof records them without image changes. Podman accepts the same +`run` options as Docker: + +```sh +podman info > /dev/null +buildprof -- podman run --rm -v "$PWD:/src" -w /src my-build-image cargo build +``` + +> [!IMPORTANT] +> Run `podman info` before recording. The first Podman command after a reboot +> or `podman system migrate` sets up the rootless user namespace using setuid +> `newuidmap`. It cannot gain privileges while traced, causing +> `newuidmap: write to uid_map failed: Operation not permitted`. +> Any Podman command run outside the recording sets up the namespace. +> Later commands reuse it. + +Commands and file paths from inside the container are recorded as the +container sees them, such as `/src/target/...` rather than the host path. diff --git a/docs/usage.md b/docs/usage.md new file mode 100644 index 0000000..0aca291 --- /dev/null +++ b/docs/usage.md @@ -0,0 +1,112 @@ +# Recording and opening builds + +Everything `buildprof` can do beyond `buildprof -- `, which is +covered by the [quick start](../README.md#quick-start). + +## Recording a build + +Put `buildprof --` in front of the build command. Choose another output path +or disable automatic opening when needed: + +```bash +buildprof -o clean-build.buildprof --no-open -- ninja -C out +``` + +## Recording in GitHub Actions + +On a Linux runner, install your build tools, then record the build: + +```yaml +steps: + - uses: actions/checkout@v7 + - uses: LalitMaganti/buildprof@v0.2.6 + with: + command: | + cmake -S . -B build + cmake --build build -j2 +``` + +The job summary links to the recording artifact, including when the build +fails. Download it and open it in [the web UI](https://buildprof.lalitm.com). +See [action inputs](../action.yml) for version and retention settings; use a +unique `artifact-name` for each matrix entry. For upgrades, use release tags +or release SHAs with Dependabot (see [action releases](../RELEASING.md#github-action-releases)). + +## Opening recordings + +Open an existing recording later with: + +```bash +buildprof open clean-build.buildprof +``` + +buildprof.lalitm.com only delivers the UI itself. Your browser fetches the +recording from localhost and processes it entirely in the page; no trace data +leaves your machine. + +Because the page comes from buildprof.lalitm.com and the recording from +localhost, the browser asks once whether the site may access other apps and +services on this device. Allow it. If you block it, the trace never loads. To +recover, allow it again in the site settings next to the address bar, under +"Apps on device" in Chrome or "Access this device" in Firefox, then run +`buildprof open` again. + +Recordings contain command lines and filesystem paths. Review them before +sending them to anyone. + +## Builds on a remote machine + +Over SSH there is no browser to launch, so Buildprof prints the port forward +to run from your own machine instead and waits for the browser to fetch the +trace: + +```bash +ssh -L 9001:127.0.0.1:9001 user@buildhost +``` + +VS Code Remote and JetBrains Gateway forward the port automatically. The wait +gives up after ten minutes; adjust it with `--wait `, where `0` waits +forever. Alternatively, copy the recording to your own machine and open it +in the [web UI](https://buildprof.lalitm.com). No local installation is needed +for viewing. + +## Collection options + +Process creation, commands, and timing are always recorded. File opens and +renames are also recorded by default; to reduce overhead on builds with lots +of filesystem activity, disable that layer: + +```bash +buildprof --no-file-events -- make -j6 +``` + +The process timeline remains available, but file lists and producer/consumer +links are unavailable. This skips filesystem interception itself, rather than +collecting and discarding events. The UI identifies recordings made this way. + +## Compiler details + +Process timing is usually the right level for understanding a build. When a +particular compiler or linker invocation needs a closer look, enable compiler +tracing: + +```bash +buildprof --compiler-traces -- cargo build +``` + +Buildprof currently imports Clang `-ftime-trace`, explicitly selected LLD +`--time-trace`, and nightly Rust self-profile data. These events appear as a +summary of active compiler threads with expandable per-thread phase tracks. +Compiler tracing can be combined with process-only recording: + +```bash +buildprof --no-file-events --compiler-traces -- ninja -C build +``` + +A build which invokes Clang through an absolute path currently bypasses +compiler tracing. Buildprof will still record the compiler process, but its +Clang and LLD internal phases will be absent. + +Compiler tracing can also change compiler cache keys or turn cache hits into +misses. Existing Rust compiler wrappers remain in the invocation chain, but +cache preservation is not guaranteed in this mode. diff --git a/infra/buildprof.lalitm.com/assemble-site b/infra/buildprof.lalitm.com/assemble-site index 8765049..819ba1d 100755 --- a/infra/buildprof.lalitm.com/assemble-site +++ b/infra/buildprof.lalitm.com/assemble-site @@ -123,13 +123,15 @@ def main() -> int: # A short, stable installer URL; GitHub's "latest" redirect keeps it # pointing at the newest release without a redeploy. The /diagnose/ links # are printed by the CLI, so installed binaries keep working when the - # README's headings change; update the anchors here instead. - readme = "https://github.com/LalitMaganti/buildprof" + # troubleshooting headings change; update the anchors here instead. + troubleshooting = ( + "https://github.com/LalitMaganti/buildprof/blob/main/docs/troubleshooting.md" + ) (site / "_redirects").write_text( "/install.sh https://github.com/LalitMaganti/buildprof/releases/latest/download/buildprof-installer.sh 302\n" - f"/diagnose/containers {readme}#builds-inside-docker-containers 302\n" - f"/diagnose/daemons {readme}#bazel-gradle-and-other-daemon-build-systems 302\n" - f"/diagnose/podman {readme}#use-podman 302\n" + f"/diagnose/containers {troubleshooting}#builds-inside-docker-containers 302\n" + f"/diagnose/daemons {troubleshooting}#bazel-gradle-and-other-daemon-build-systems 302\n" + f"/diagnose/podman {troubleshooting}#use-podman 302\n" ) if args.headers is not None: diff --git a/infra/prepare-action-release b/infra/prepare-action-release index 643d86a..776c851 100644 --- a/infra/prepare-action-release +++ b/infra/prepare-action-release @@ -30,17 +30,17 @@ def replace_once(text: str, pattern: str, replacement: str) -> str: def prepare(root: Path, *, check: bool, allow_preview: bool) -> list[str]: package = tomllib.loads((root / "Cargo.toml").read_text())["package"] version = "v" + package["version"] - readme = root / "README.md" + usage = root / "docs/usage.md" reference = version if ( allow_preview and version == PREVIEW_VERSION - and f"{PREVIEW_REF} {PREVIEW_NOTE}" in readme.read_text() + and f"{PREVIEW_REF} {PREVIEW_NOTE}" in usage.read_text() ): reference = f"{PREVIEW_REF} {PREVIEW_NOTE}" replacements = { root / "action.yml": (r"(?<= default: )v[^\n]+", version), - readme: ( + usage: ( r"(?<= - uses: LalitMaganti/buildprof@)[^\n]+", reference, ), diff --git a/justfile b/justfile index 7e5f675..3609c52 100644 --- a/justfile +++ b/justfile @@ -38,7 +38,7 @@ test *args: check: uv run dev/test --collect-only -# Update the action default and README example after changing Cargo.toml's version. +# Update the action default and docs example after changing Cargo.toml's version. release-prepare: python3 infra/prepare-action-release