A Rust CLI and web UI that intercepts an LLM's token stream as it arrives, shows per-token confidence and perplexity from the logprobs, and can rewrite every other token (or any fraction) on the fly, for interpretability research, red-teaming and prompt engineering.
You normally see a model's answer only as finished text. `every-other-token` sits on the OpenAI or Anthropic SSE stream and lets you watch and poke it token by token: color tokens by confidence, mutate them with transforms (reverse, noise, delete, synonym, chained), compare two providers or two system prompts statistically, record and replay runs, and export per-token data to CSV, JSONL or an HTML heatmap. A `--dry-run` mode with a mock provider needs no API key.
-
+## Try it in one minute (no API key)
+
+```bash
+cargo install every-other-token
+every-other-token "What is consciousness?" --provider mock --visual
+```
+
+
+
+The `mock` provider replays a fixed reply with fixed logprobs through the real interception pipeline, so everything above is real output; only the model is canned. Set `OPENAI_API_KEY` or `ANTHROPIC_API_KEY` and drop `--provider mock` to intercept a real model.
+
+## What you get
+
+**Every token carries its numbers.** Each intercepted token is an event with the original text, what was shown, its index, whether it was rewritten, confidence `exp(logprob)`, perplexity `exp(-logprob)` and the top alternatives. Read them from the library, stream them as JSON lines with `--json-stream`, or export CSV, JSONL or a self-contained HTML heatmap.
+
+
+
+`cargo run --example mock_stream` in a clone of this repo.
+
+**A web UI for watching the stream.** `--web` serves a single page (no build step, no external scripts): single, split and quad views, OpenAI vs Anthropic diff, A/B system prompts, a research dashboard, export, and collaborative rooms where several people edit tokens mid-stream. Pick **Mock (no API key)** in the provider menu to try it offline.
+
+
+
+**Compare, test, record.** Diff two providers (`--diff-terminal`), A/B two system prompts with a Welch's t-test (`--significance`), run headless research batches (`--research --runs 20`), and record any session to replay it after a model update.
## Install
@@ -49,10 +82,11 @@ The Linux binary links the system OpenSSL (libssl3, present on Ubuntu 22.04+ and
| Collaborative rooms | No | Yes: WebSocket multi-participant token surgery |
| TF-IDF semantic heatmaps | No | Yes: no embedding service required |
----
-
## Use cases
+
+Interpretability research, red-teaming, prompt engineering
+
### Interpretability research
- Map which input tokens causally drive each output token using
@@ -87,9 +121,9 @@ The Linux binary links the system OpenSSL (libssl3, present on Ubuntu 22.04+ and
- Export a self-contained HTML heatmap with `AttributionExporter::to_html_heatmap`
to share results without requiring a Python environment.
----
+
-## 5-minute quickstart
+## Build and run from source
### Prerequisites
@@ -152,8 +186,6 @@ export ANTHROPIC_API_KEY="sk-ant-..."
./target/release/every-other-token --completions fish > ~/.config/fish/completions/every-other-token.fish
```
----
-
## Transform types
Each token in the stream can be independently mutated before it reaches the output.
@@ -179,13 +211,94 @@ Each token in the stream can be independently mutated before it reaches the outp
`--min-confidence 0.8` only transforms tokens whose API confidence is below the threshold. High-confidence tokens pass through unchanged.
----
+## A/B system prompt testing
+
+Test how two different system prompts affect per-token confidence distributions, with automatic statistical significance testing.
+
+```bash
+./target/release/every-other-token "Explain machine learning" \
+ --research --runs 30 \
+ --system-a "You are a concise technical expert." \
+ --system-b "You are a friendly tutor explaining to a beginner." \
+ --significance \
+ --output ab_results.json
+```
+
+The output JSON includes:
+- Per-run confidence histograms for system A and system B
+- Welch's t-test statistic and p-value
+- Mean confidence delta between the two system prompts
+- Positions where the distributions diverged most
+
+### A/B via the web UI
+
+Launch with `--web` and select the **Experiment** view to see both system prompts streaming side-by-side with a live divergence map.
+
+## Web UI guide
+
+Launch with `--web` to open the single-page application at `http://localhost:8888`.
+
+| View | Description |
+|------|-------------|
+| **Single** | Live token stream with per-token confidence bars and perplexity pulse |
+| **Split** | Original vs transformed output side by side |
+| **Quad** | Four transforms applied simultaneously in a 2x2 grid |
+| **Diff** | OpenAI and Anthropic streaming the same prompt; diverging positions highlighted |
+| **Experiment** | A/B mode: two system prompts, live divergence map |
+| **Research** | Aggregate stats dashboard: perplexity histogram, confidence distribution, vocabulary diversity |
+
+Change the port with `--port 9000` (also settable in `~/.eot.toml`); `--no-open` skips opening a browser tab. With no API key, run `every-other-token --web --provider mock` or pick **Mock (no API key)** in the provider menu.
+
+## Configuration file
+
+Create `~/.eot.toml` (global) or `.eot.toml` in the working directory (local wins over global):
+
+```toml
+provider = "anthropic"
+model = "claude-sonnet-4-6"
+transform = "reverse"
+rate = 0.5
+port = 8888
+top_logprobs = 5
+system_a = "You are a concise assistant."
+```
+
+All CLI flags override config file values.
+
+## Building from source
+
+```bash
+git clone https://github.com/Mattbusel/Every-Other-Token
+cd Every-Other-Token
+cargo build --release
+cargo test --lib
+```
+
+Enable optional features:
+
+```bash
+cargo build --release --features sqlite-log,self-tune
+```
+
+## Contributing
+
+1. Fork the repository.
+2. Create a feature branch: `git checkout -b my-feature`.
+3. Ensure `cargo fmt`, `cargo clippy -- -D warnings`, and `cargo test --lib` all pass.
+4. Open a pull request against `main`.
+
+CI enforces formatting, Clippy, the full test suite, and a release build before merging.
+
+## Reference
-## Token attribution export
+The library behind the CLI, module by module. Each section is collapsed; open the one you need.
+
+
+Token attribution export
`attribution::AttributionExporter` exports per-token confidence, perplexity, and attribution scores to three formats compatible with LIME, Captum, and custom dashboards.
-### Data types
+#### Data types
| Type | Description |
|------|-------------|
@@ -193,7 +306,7 @@ Each token in the stream can be independently mutated before it reaches the outp
| `SequenceAttribution` | Full sequence: all tokens + aggregate confidence, aggregate perplexity, semantic drift |
| `AttributionExporter` | Serializes to JSONL, CSV, or self-contained HTML heatmap |
-### Export to JSONL (LIME / Captum compatible)
+#### Export to JSONL (LIME / Captum compatible)
```rust
use every_other_token::attribution::{AttributionExporter, SequenceAttribution, TokenAttribution};
@@ -230,7 +343,7 @@ let html = exporter.to_html_heatmap(&seq);
std::fs::write("heatmap.html", html).unwrap();
```
-### Semantic drift detection
+#### Semantic drift detection
Semantic drift measures the confidence decay from the first half of the generated sequence to the second half. A positive value means the model became less certain toward the end of the response.
@@ -241,7 +354,7 @@ if drift > 0.15 {
}
```
-### Finding the most and least confident tokens
+#### Finding the most and least confident tokens
```rust
let (most_confident, least_confident) = exporter.confidence_extremes(&seq);
@@ -249,7 +362,7 @@ println!("Most confident positions: {:?}", most_confident);
println!("Least confident positions: {:?}", least_confident);
```
-### Python interoperability
+#### Python interoperability
The JSONL format maps directly to a pandas DataFrame:
@@ -273,16 +386,17 @@ with open("attribution.jsonl") as f:
df = pd.DataFrame(rows)
```
----
+
-## Token Attribution Map (causal leave-one-out)
+
+Token Attribution Map (causal leave-one-out)
`AttributionMap` computes per-output-token causal attribution scores using an
approximate leave-one-out (LOO) proxy. For each input token *i*, the model is
re-run with token *i* masked; the drop in confidence of each output token is
recorded as the influence score.
-### Computation model
+#### Computation model
```
score(input_i, output_j) = baseline_confidence(output_j)
@@ -295,7 +409,7 @@ Negative score = input *i* was **hurting** output *j*.
This requires N+1 inferences for N input tokens. Use `max_input_tokens` in your
runner to control cost.
-### Example
+#### Example
```rust
use every_other_token::attribution::{AttributionMap, AttributionRenderer};
@@ -321,7 +435,7 @@ let json = map.to_json().expect("serialization succeeds");
std::fs::write("attribution_map.json", json).ok();
```
-### Terminal renderer
+#### Terminal renderer
`AttributionRenderer` uses ANSI color bands matching the existing heatmap:
@@ -335,15 +449,16 @@ std::fs::write("attribution_map.json", json).ok();
`with_top_k(N)` limits the display to the N most influential input tokens per
output position.
----
+
-## Prompt Mutation Lab
+
+Prompt Mutation Lab
`MutationLab` systematically varies specified tokens, phrases, or parameters in
a base prompt, runs all variants through the model, and produces a ranked
markdown table of results.
-### Mutation targets
+#### Mutation targets
| Target | Description |
|--------|-------------|
@@ -352,7 +467,7 @@ markdown table of results.
| `SystemPrompt` | Replace the entire system prompt |
| `Temperature` | Vary the sampling temperature (variant parsed as `f32`) |
-### Mutation metrics
+#### Mutation metrics
| Metric | Description |
|--------|-------------|
@@ -361,7 +476,7 @@ markdown table of results.
| `TopTokenChange` | Whether any top predicted token changed vs. the baseline |
| `SentimentShift` | Fraction of high-confidence tokens minus baseline fraction |
-### Example
+#### Example
```rust
use every_other_token::mutation_lab::{
@@ -392,10 +507,14 @@ println!("Best variant: {:?}", result.best().map(|r| &r.variant));
println!("Worst variant: {:?}", result.worst().map(|r| &r.variant));
```
-### Output table format
+#### Output table format
```markdown
-## Mutation Experiment Results
+
+
+
+
+Mutation Experiment Results
**Base prompt:** `Explain {SLOT} to a 5-year-old.`
**Target:** Word(1)
@@ -409,69 +528,10 @@ println!("Worst variant: {:?}", result.worst().map(|r| &r.variant));
| 4 | `photosynthesis` | `Explain photosynthesis to a 5-y...` | 7.2981 |
```
----
+
-## A/B system prompt testing
-
-Test how two different system prompts affect per-token confidence distributions, with automatic statistical significance testing.
-
-```bash
-./target/release/every-other-token "Explain machine learning" \
- --research --runs 30 \
- --system-a "You are a concise technical expert." \
- --system-b "You are a friendly tutor explaining to a beginner." \
- --significance \
- --output ab_results.json
-```
-
-The output JSON includes:
-- Per-run confidence histograms for system A and system B
-- Welch's t-test statistic and p-value
-- Mean confidence delta between the two system prompts
-- Positions where the distributions diverged most
-
-### A/B via the web UI
-
-Launch with `--web` and select the **Experiment** view to see both system prompts streaming side-by-side with a live divergence map.
-
----
-
-## Web UI guide
-
-Launch with `--web` to open the single-page application at `http://localhost:8888`.
-
-| View | Description |
-|------|-------------|
-| **Single** | Live token stream with per-token confidence bars and perplexity pulse |
-| **Split** | Original vs transformed output side by side |
-| **Quad** | Four transforms applied simultaneously in a 2x2 grid |
-| **Diff** | OpenAI and Anthropic streaming the same prompt; diverging positions highlighted |
-| **Experiment** | A/B mode: two system prompts, live divergence map |
-| **Research** | Aggregate stats dashboard: perplexity histogram, confidence distribution, vocabulary diversity |
-
-Change the port with `--port 9000`. The port can also be set in `~/.eot.toml`.
-
----
-
-## Configuration file
-
-Create `~/.eot.toml` (global) or `.eot.toml` in the working directory (local wins over global):
-
-```toml
-provider = "anthropic"
-model = "claude-sonnet-4-6"
-transform = "reverse"
-rate = 0.5
-port = 8888
-top_logprobs = 5
-system_a = "You are a concise assistant."
-```
-
-All CLI flags override config file values.
-
----
-
-## CLI reference
+
+CLI reference
```
USAGE:
@@ -488,6 +548,7 @@ OPTIONS:
--heatmap Enable token importance heatmap
--web Launch web UI instead of terminal
--port Web UI port [default: 8888]
+ --no-open With --web, do not open a browser tab
--research Headless research mode
--runs Number of research runs [default: 10]
--output Research output JSON path [default: research_output.json]
@@ -510,13 +571,14 @@ OPTIONS:
--log-db SQLite experiment log (requires sqlite-log feature)
```
----
+
-## Divergence Detection Engine
+
+Divergence Detection Engine
`divergence.rs` runs the same prompt through N model configurations simultaneously and computes a per-position **Jensen-Shannon divergence** score showing exactly where and how much models disagree.
-### Concepts
+#### Concepts
| Term | Description |
|------|-------------|
@@ -526,7 +588,7 @@ OPTIONS:
| `DivergenceResult` | Complete run: all streams, all divergence points, aggregate statistics |
| `DivergenceDetector` | Orchestrates analysis; threshold configurable |
-### Jensen-Shannon divergence
+#### Jensen-Shannon divergence
At each token position `t`, the detector collects a probability distribution `P_i(t)` from each model (from the logprobs API). It then computes:
@@ -543,7 +605,7 @@ normalised by `log(n)` to give a score in `[0, 1]`.
| ~0.5 | Substantial disagreement |
| 1.0 | Models produce completely different tokens |
-### Usage
+#### Usage
```rust
use every_other_token::divergence::{DivergenceDetector, ModelConfig};
@@ -565,7 +627,7 @@ println!("Mean JS: {:.4}", result.mean_divergence());
println!("High-divergence positions: {:?}", result.high_divergence_positions(0.5));
```
-### Coloured diff report
+#### Coloured diff report
`DivergenceResult::render_report()` produces a multi-line ANSI report:
@@ -575,13 +637,14 @@ println!("High-divergence positions: {:?}", result.high_divergence_positions(0.5
Each model's full token stream is printed with its disagreement positions highlighted.
----
+
-## Token Intervention Mode
+
+Token Intervention Mode
`intervention.rs` enables **causal interventions**: pausing generation at any token, injecting a correction, continuing generation from the injected token, and measuring the causal effect downstream.
-### Concepts
+#### Concepts
| Term | Description |
|------|-------------|
@@ -591,7 +654,7 @@ Each model's full token stream is printed with its disagreement positions highli
| `causal_effect()` | Normalised token-level edit distance between original and final streams |
| `causal_influence_map()` | Per-position influence score: how much each position affects downstream tokens |
-### Causal effect measurement
+#### Causal effect measurement
```rust
use every_other_token::intervention::{InterventionHistory, Intervention};
@@ -609,7 +672,7 @@ history.apply(intervention, counterfactual);
println!("Causal effect: {:.3}", history.causal_effect()); // fraction of tokens that changed
```
-### TUI token editor
+#### TUI token editor
`TokenEditor` provides cursor navigation and token-level editing in any VT100 terminal:
@@ -627,7 +690,7 @@ let edit = editor.commit_edit(); // Some((1, " sky", " ocean"))
println!("{}", editor.render()); // ANSI-highlighted stream
```
-### Causal influence map
+#### Causal influence map
```rust
use every_other_token::intervention::causal_influence_map;
@@ -640,9 +703,10 @@ let map = causal_influence_map(&original, &counterfact);
// map[1] = fraction of positions >1 that changed = 0.0 (c and d unchanged)
```
----
+
-## Key modules
+
+Key modules
| Module | Responsibility |
|--------|----------------|
@@ -669,9 +733,10 @@ let map = causal_influence_map(&original, &counterfact);
| `bayesian.rs` | Bayesian confidence interval estimation across runs |
| `checkpoint.rs` | Snapshot/restore for long research sessions |
----
+
-## Feature flags
+
+Feature flags
| Flag | Default | Description |
|------|---------|-------------|
@@ -685,13 +750,14 @@ let map = causal_influence_map(&original, &counterfact);
| `redis-backing` | Off | Write-through Redis persistence for agent memory and snapshots |
| `wasm` | Off | WASM target bindings via `wasm-bindgen` |
----
+
-## Semantic Similarity
+
+Semantic Similarity
`src/similarity.rs` provides TF-IDF based semantic similarity scoring.
-### CLI usage
+#### CLI usage
```bash
# Compare two texts
@@ -701,7 +767,7 @@ every-other-token --similarity "the quick brown fox" "a fast red fox"
every-other-token "$(cat my_prompts.txt)" --diversity-filter
```
-### Programmatic usage
+#### Programmatic usage
```rust
use every_other_token::similarity::{SemanticScorer, DiversityFilter};
@@ -723,7 +789,10 @@ let filter = DiversityFilter::new(&seqs);
let deduped = filter.filter(seqs, 0.85);
```
-## Streaming with Backpressure
+
+
+
+Streaming with Backpressure
`src/stream_compress.rs` provides a streaming token compressor with three
backpressure strategies.
@@ -734,7 +803,7 @@ backpressure strategies.
| `Block` | Signals `Throttled` until the buffer drains below `low_watermark` |
| `Compress` | Aggressively compresses incoming tokens (keeps every other) |
-### Example
+#### Example
```rust
use every_other_token::stream_compress::{
@@ -755,41 +824,14 @@ let stats = sc.stats();
println!("ratio: {:.2}", stats.ratio);
```
----
+
-## Building from source
-
-```bash
-git clone https://github.com/Mattbusel/Every-Other-Token
-cd Every-Other-Token
-cargo build --release
-cargo test --lib
-```
-
-Enable optional features:
-
-```bash
-cargo build --release --features sqlite-log,self-tune
-```
-
----
-
-## Contributing
-
-1. Fork the repository.
-2. Create a feature branch: `git checkout -b my-feature`.
-3. Ensure `cargo fmt`, `cargo clippy -- -D warnings`, and `cargo test --lib` all pass.
-4. Open a pull request against `main`.
-
-CI enforces formatting, Clippy, the full test suite, and a release build before merging.
-
----
-
-## Batch Processing
+
+Batch Processing
The `batch` module (`src/batch.rs`) provides a priority-queue-based concurrent pipeline for compressing many token sequences at once.
-### Data types
+#### Data types
| Type | Description |
|------|-------------|
@@ -798,7 +840,7 @@ The `batch` module (`src/batch.rs`) provides a priority-queue-based concurrent p
| `BatchConfig` | `max_concurrent`, `queue_capacity`, `timeout` |
| `BatchStats` | Aggregate: `jobs_submitted`, `jobs_completed`, `jobs_failed`, `avg_ratio`, `throughput_tokens_per_sec` |
-### Usage
+#### Usage
```bash
# Process a JSONL file where each line is a JSON array of tokens
@@ -813,7 +855,7 @@ Each line of `tokens.jsonl` should be a JSON array:
Results are streamed to stdout as JSONL. Aggregate stats are printed to stderr.
-### API example
+#### API example
```rust
use every_other_token::batch::{BatchConfig, BatchProcessor};
@@ -831,13 +873,14 @@ while let Some(result) = stream.next().await {
Higher-priority jobs are processed first via a `BinaryHeap`. Concurrent execution is bounded by `max_concurrent`.
----
+
-## Quality-Adaptive Compression
+
+Quality-Adaptive Compression
The `adaptive` module (`src/adaptive.rs`) measures and optimises the quality of token-level compression.
-### Quality model
+#### Quality model
```
overall_score = 0.4 * semantic_preservation
@@ -852,14 +895,14 @@ overall_score = 0.4 * semantic_preservation
| `ratio` | `compressed_len / original_len`, lower means more compressed |
| `overall_score` | Weighted combination (higher is better) |
-### Usage
+#### Usage
```bash
# Print quality metrics for the compression applied to a prompt
every-other-token "The quick brown fox jumps" --quality
```
-### API example
+#### API example
```rust
use every_other_token::adaptive::{AdaptiveCompressor, CompressionTarget, QualityMetric};
@@ -879,9 +922,10 @@ let (compressed, quality) = compressor.compress(&tokens, &target);
println!("compressed to {} tokens with score={:.3}", compressed.len(), quality.overall_score);
```
----
+
-## Context Window Manager
+
+Context Window Manager
The `context` module provides `ContextWindow`, a priority-based token budget manager for LLM context construction.
@@ -925,9 +969,10 @@ println!("evictions: {}", stats.evictions);
- `ContextStats` tracks block counts by type, utilization, and eviction count
- `--context-budget ` CLI flag sets the token budget (default: 4096)
----
+
-## Vocabulary Analyzer
+
+Vocabulary Analyzer
The `vocab` module provides `VocabularyAnalyzer`, measures vocabulary coverage, OOV rates, and Zipf's law fit for token sequences.
@@ -970,18 +1015,15 @@ println!("Zipf exponent: {:.3} (r²={:.3})", params.exponent, params.r_squared);
- `zipf_score`: 1 - |exponent - 1.0|, normalized to [0, 1]
- `--vocab-stats` CLI flag: prints vocabulary statistics for the current prompt
----
+
## Status
-Research tool, actively developed. The CI workflow is currently failing on `main`; `cargo build` works. The screenshot above is from an early interactive version; the current CLI takes the prompt, transform and model as arguments (see the CLI reference).
-
----
+Research tool, actively developed. CI (the full test suite plus a feature-flag matrix) runs on every push and pull request. Tagged releases get prebuilt binaries for Linux, macOS and Windows from the release workflow.
## License
-MIT -- see [LICENSE](LICENSE) for details.
-
+MIT, see [LICENSE](LICENSE) for details.
## Hire the author
diff --git a/Screenshot 2025-07-08 185410.png b/Screenshot 2025-07-08 185410.png
deleted file mode 100644
index 83aa8fc4..00000000
Binary files a/Screenshot 2025-07-08 185410.png and /dev/null differ
diff --git a/Screenshot 2025-07-09 015940.png b/Screenshot 2025-07-09 015940.png
deleted file mode 100644
index 58243e60..00000000
Binary files a/Screenshot 2025-07-09 015940.png and /dev/null differ
diff --git a/Screenshot 2025-07-12 161852.png b/Screenshot 2025-07-12 161852.png
deleted file mode 100644
index 9e5ae423..00000000
Binary files a/Screenshot 2025-07-12 161852.png and /dev/null differ
diff --git a/assets/hero.svg b/assets/hero.svg
new file mode 100644
index 00000000..537ee5bb
--- /dev/null
+++ b/assets/hero.svg
@@ -0,0 +1,12 @@
+
\ No newline at end of file
diff --git a/assets/terminal-table.svg b/assets/terminal-table.svg
new file mode 100644
index 00000000..e92d72af
--- /dev/null
+++ b/assets/terminal-table.svg
@@ -0,0 +1 @@
+
\ No newline at end of file
diff --git a/assets/terminal-visual.svg b/assets/terminal-visual.svg
new file mode 100644
index 00000000..6767878c
--- /dev/null
+++ b/assets/terminal-visual.svg
@@ -0,0 +1 @@
+
\ No newline at end of file
diff --git a/assets/web-ui.png b/assets/web-ui.png
new file mode 100644
index 00000000..8aaea0f8
Binary files /dev/null and b/assets/web-ui.png differ
diff --git a/docs/.nojekyll b/docs/.nojekyll
new file mode 100644
index 00000000..e69de29b
diff --git a/docs/assets/hero.svg b/docs/assets/hero.svg
new file mode 100644
index 00000000..537ee5bb
--- /dev/null
+++ b/docs/assets/hero.svg
@@ -0,0 +1,12 @@
+
\ No newline at end of file
diff --git a/docs/assets/social-preview.png b/docs/assets/social-preview.png
new file mode 100644
index 00000000..e824343c
Binary files /dev/null and b/docs/assets/social-preview.png differ
diff --git a/docs/assets/terminal-table.svg b/docs/assets/terminal-table.svg
new file mode 100644
index 00000000..e92d72af
--- /dev/null
+++ b/docs/assets/terminal-table.svg
@@ -0,0 +1 @@
+
\ No newline at end of file
diff --git a/docs/assets/terminal-visual.svg b/docs/assets/terminal-visual.svg
new file mode 100644
index 00000000..6767878c
--- /dev/null
+++ b/docs/assets/terminal-visual.svg
@@ -0,0 +1 @@
+
\ No newline at end of file
diff --git a/docs/assets/web-ui.png b/docs/assets/web-ui.png
new file mode 100644
index 00000000..8aaea0f8
Binary files /dev/null and b/docs/assets/web-ui.png differ
diff --git a/docs/index.html b/docs/index.html
new file mode 100644
index 00000000..7b2c83cd
--- /dev/null
+++ b/docs/index.html
@@ -0,0 +1,242 @@
+
+
+
+
+
+every-other-token
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Read the stream, not the answer.
+
every-other-token is a Rust CLI and web UI that sits on an OpenAI or Anthropic token stream as it arrives. It rewrites every other token (or any fraction), and it tells you how sure the model was about each one: confidence exp(logprob) and perplexity exp(-logprob), per token, live.
Recorded from the real /stream endpoint of every-other-token --web --provider mock. The mock provider replays a fixed reply with fixed logprobs through the same interception pipeline a real model goes through, so it runs with no API key. Point it at OpenAI or Anthropic and the tokens and numbers are the model's own.
+
+
+
+
01Running in two minutes
+
Prebuilt binaries for Linux x86_64, macOS (Apple Silicon and Intel) and Windows x86_64 are on the releases page. Or build it with Cargo (Rust 1.81+).
+
+
install
+
$ cargo install every-other-token
+
try it, no API key
+
$ every-other-token "What is consciousness?" --provider mock --visual
+
a real model, in the browser
+
$ export OPENAI_API_KEY=sk-... # or ANTHROPIC_API_KEY with --provider anthropic
+$ every-other-token --web # http://localhost:8888
+
+
+
+
+
02In the terminal
+
The stream prints as it arrives. With --visual, rewritten tokens are highlighted; --heatmap colors each token by importance instead.
+
Real output of the command above.
+
+
+
+
03Every token carries its numbers
+
Each intercepted token is an event with the original text, what was shown, its index, whether it was rewritten, confidence, perplexity and the top alternatives. Consume them from the library, as JSON lines with --json-stream, or export them to CSV, JSONL or a self-contained HTML heatmap.
+
cargo run --example mock_stream, from a clone of the repo.
+
+
+
+
04The web UI
+
--web serves a single page with no build step and no external scripts. Single, split, quad (four transforms at once), OpenAI vs Anthropic diff, A/B system prompts, a research dashboard, JSON and CSV export, and collaborative rooms where several people edit tokens mid-stream.
+
Split view, mock provider. The underline under each token is its confidence.
+
+
+
+
05What else is in the box
+
+
9 transforms, chainable
reverse, uppercase, mock, noise, chaos, scramble, delete, synonym, delay:N, or a chain like reverse,uppercase.
+
Rate, seed, gating
--rate 0.3 spreads rewrites evenly at any fraction. --seed makes random transforms reproducible. --min-confidence only touches tokens the model was unsure of.
+
Provider diff
--diff-terminal streams OpenAI and Anthropic side by side and compares their confidence structure.
+
A/B system prompts
Run two system prompts against the same user prompt, many times, and test the confidence shift with Welch's t-test (--significance).
+
Research mode
--research --runs 20 runs headless and writes aggregate perplexity, confidence and vocabulary stats to JSON.
+
Record and replay
Record any session to JSON and replay it later, for example to spot behavior changes after a model update.
+
+
+
+
+
+
+
+
+
diff --git a/examples/mock_stream.rs b/examples/mock_stream.rs
index 8cbe1923..0341d888 100644
--- a/examples/mock_stream.rs
+++ b/examples/mock_stream.rs
@@ -39,14 +39,14 @@ async fn main() -> Result<(), Box> {
println!("prompt: {prompt}");
println!("transform: {transform:?} (applied to every other token)\n");
- println!("{:>3} {:<24} {:<24} {:>6} {:>6}", "#", "original", "shown", "conf", "ppl");
- println!("{}", "-".repeat(68));
+ println!("{:>3} {:<27} {:<27} {:>6} {:>6}", "#", "original", "shown", "conf", "ppl");
+ println!("{}", "-".repeat(74));
let mut original = String::new();
let mut shown = String::new();
while let Ok(ev) = rx.try_recv() {
println!(
- "{:>3} {:<24} {:<24} {:>6} {:>6}{}",
+ "{:>3} {:<27} {:<27} {:>6} {:>6}{}",
ev.index,
format!("{:?}", ev.original),
format!("{:?}", ev.text),
diff --git a/src/cli.rs b/src/cli.rs
index ceeaffce..81c5a0a3 100644
--- a/src/cli.rs
+++ b/src/cli.rs
@@ -48,6 +48,10 @@ pub struct Args {
#[arg(long, default_value = "8888")]
pub port: u16,
+ /// With --web, do not open a browser tab automatically
+ #[arg(long)]
+ pub no_open: bool,
+
/// Enable headless research mode — runs N times and outputs JSON stats
#[arg(long)]
pub research: bool,
diff --git a/src/lib.rs b/src/lib.rs
index f4009e63..20528f1d 100644
--- a/src/lib.rs
+++ b/src/lib.rs
@@ -321,6 +321,10 @@ pub struct TokenInterceptor {
/// Optional stream timeout in seconds. When set, `intercept_stream` will fail
/// with a timeout error if the entire stream does not complete within this duration.
pub timeout_secs: Option,
+ /// Whitespace seen since the last emitted token in web / JSON-stream mode.
+ /// It is prepended to the next token's `text` and `original` so consumers
+ /// that concatenate events get the spaces back (issue #3).
+ pending_ws: String,
}
// ---------------------------------------------------------------------------
@@ -541,6 +545,7 @@ impl TokenInterceptor {
anthropic_max_tokens: 4096,
stream_start_instant: None,
timeout_secs: None,
+ pending_ws: String::new(),
})
}
@@ -742,7 +747,8 @@ impl TokenInterceptor {
}
}
- if self.web_tx.is_none() {
+ // JSON stream mode keeps stdout pure JSONL, so no header or footer.
+ if self.web_tx.is_none() && !self.json_stream {
self.print_header(prompt);
}
@@ -790,7 +796,7 @@ impl TokenInterceptor {
Provider::Mock => self.stream_mock(&effective_prompt).await?,
}
- if self.web_tx.is_none() {
+ if self.web_tx.is_none() && !self.json_stream {
self.print_footer();
}
Ok(())
@@ -1037,7 +1043,7 @@ impl TokenInterceptor {
// Simulates a response to any prompt without hitting any API.
// First 20 characters (not bytes): slicing by byte index panicked on
// prompts where byte 20 falls inside a multi-byte UTF-8 character.
- let prompt_prefix: String = prompt.chars().take(20).collect();
+ let prompt_prefix: String = prompt.chars().take(40).collect();
let fixture: Vec<(String, f32)> = vec![
("The".to_string(), -0.12),
(" quick".to_string(), -0.45),
@@ -1057,18 +1063,10 @@ impl TokenInterceptor {
(" for".to_string(), -0.27),
(" prompt".to_string(), -0.53),
(":".to_string(), -0.18),
- (" \"".to_string(), -0.39),
- (prompt_prefix, -0.71),
+ (format!(" {prompt_prefix}"), -0.71),
];
- // Vary fixture starting position based on prompt content for more realistic tests
- let prompt_hash: usize = prompt
- .bytes()
- .fold(0usize, |acc, b| acc.wrapping_add(b as usize));
- let offset = prompt_hash % fixture.len();
-
- for idx in 0..fixture.len() {
- let (token_text, logprob) = &fixture[(idx + offset) % fixture.len()];
+ for (idx, (token_text, logprob)) in fixture.iter().enumerate() {
let token_text = token_text.clone();
let confidence = logprob.exp().clamp(0.0_f32, 1.0_f32);
let perplexity = (-logprob).exp();
@@ -1076,13 +1074,17 @@ impl TokenInterceptor {
let should_transform = idx % 2 == 1;
let (display_text, chaos_label) = if should_transform {
- let (t, label) = self.transform.apply_with_label(&token_text);
+ // Transform the word and keep its leading space where it was,
+ // so " quick" reversed shows as " kciuq", not "kciuq ".
+ let word = token_text.trim_start();
+ let lead = &token_text[..token_text.len() - word.len()];
+ let (t, label) = self.transform.apply_with_label(word);
let cl = if matches!(self.transform, Transform::Chaos) {
Some(label.to_string())
} else {
None
};
- (t, cl)
+ (format!("{lead}{t}"), cl)
} else {
(token_text.clone(), None)
};
@@ -1289,6 +1291,17 @@ impl TokenInterceptor {
// Delete transform: the result is an empty string (chaos_label="deleted").
let is_deleted = should_transform && display_text.is_empty();
+ // Web / JSON consumers never see whitespace tokens, so carry the
+ // whitespace that preceded this token onto its text. A deleted
+ // token leaves it pending for the next one.
+ let (display_text, original_text) =
+ if (self.web_tx.is_some() || self.json_stream) && !is_deleted {
+ let lead = std::mem::take(&mut self.pending_ws);
+ (format!("{lead}{display_text}"), format!("{lead}{token}"))
+ } else {
+ (display_text, token.clone())
+ };
+
// Web / terminal / json output — skip deleted tokens for display.
if !is_deleted {
// Record per-token arrival latency relative to stream start.
@@ -1297,7 +1310,7 @@ impl TokenInterceptor {
if let Some(tx) = &self.web_tx {
let event = TokenEvent {
text: display_text.clone(),
- original: token.clone(),
+ original: original_text.clone(),
index: i,
transformed: should_transform,
importance,
@@ -1317,7 +1330,7 @@ impl TokenInterceptor {
// JSON stream mode: one line per token
let event = TokenEvent {
text: display_text.clone(),
- original: token.clone(),
+ original: original_text.clone(),
index: i,
transformed: should_transform,
importance,
@@ -1353,6 +1366,10 @@ impl TokenInterceptor {
// transformed), but terminal output still needs it or every
// word runs into the next one.
print!("{}", token);
+ } else {
+ // Web / JSON stream: hold the whitespace and attach it to the
+ // next emitted token so the text keeps its spacing.
+ self.pending_ws.push_str(&token);
}
}
@@ -1376,46 +1393,54 @@ impl TokenInterceptor {
/// `heatmap_mode` is active, additional legend lines are printed.
/// This method is a no-op when `web_tx` is set (web mode handles its own header).
pub fn print_header(&self, prompt: &str) {
- println!("{}", "EVERY OTHER TOKEN INTERCEPTOR".bright_cyan().bold());
+ let label = |k: &str| format!("{:<11}", k).dimmed();
+ let transform = format!("{:?}", self.transform).to_lowercase();
+ println!();
+ println!(
+ "{} {}",
+ "every-other-token".bold(),
+ format!("v{}", env!("CARGO_PKG_VERSION")).dimmed()
+ );
+ println!();
println!(
- "{}: {}",
- "Provider".bright_yellow(),
- self.provider.to_string().bright_white()
+ "{}{} {}",
+ label("provider"),
+ self.provider.to_string().bright_white(),
+ format!("({})", self.model).dimmed()
);
- println!("{}: {:?}", "Transform".bright_yellow(), self.transform);
- println!("{}: {}", "Model".bright_yellow(), self.model);
- println!("{}: {}", "Prompt".bright_yellow(), prompt);
+ println!(
+ "{}{} {}",
+ label("transform"),
+ transform.bright_cyan(),
+ format!("at rate {}", self.rate).dimmed()
+ );
+ println!("{}{}", label("prompt"), prompt);
if self.orchestrator {
println!(
- "{}: {}",
- "Orchestrator".bright_magenta(),
- "ON (MCP pipeline at localhost:3000)".bright_magenta()
+ "{}{}",
+ label("pipeline"),
+ "MCP orchestrator at localhost:3000".bright_magenta()
);
}
if self.visual_mode {
println!(
- "{}: {}",
- "Visual Mode".bright_green(),
- "ON (even=normal, odd=cyan+bold)".bright_green()
+ "{}{} {}",
+ label("legend"),
+ "original",
+ "transformed".bright_cyan().bold()
);
}
if self.heatmap_mode {
println!(
- "{}: {}",
- "Heatmap Mode".bright_magenta(),
- "ON (color intensity = token importance)".bright_magenta()
- );
- println!(
- "{}: {} {} {} {}",
- "Legend".bright_white(),
- "Low".on_blue(),
- "Medium".on_yellow(),
- "High".on_red(),
- "Critical".on_bright_red().bright_white()
+ "{}{} {} {} {}",
+ label("heatmap"),
+ " low ".on_blue(),
+ " medium ".on_yellow(),
+ " high ".on_red(),
+ " critical ".on_bright_red().bright_white()
);
}
- println!("{}", "=".repeat(50).bright_blue());
- println!("{}", "Response (with transformations):".bright_green());
+ println!("{}", "─".repeat(56).dimmed());
println!();
}
@@ -1423,9 +1448,15 @@ impl TokenInterceptor {
///
/// Reports total token count and how many tokens were transformed.
pub fn print_footer(&self) {
- println!("\n{}", "=".repeat(50).bright_blue());
- println!("Complete! Processed {} tokens.", self.token_count);
- println!("Transform applied to {} tokens.", self.transformed_count);
+ println!();
+ println!();
+ println!("{}", "─".repeat(56).dimmed());
+ println!(
+ "{} tokens streamed, {} transformed",
+ self.token_count.to_string().bold(),
+ self.transformed_count.to_string().bright_cyan().bold()
+ );
+ println!();
}
}
@@ -1622,6 +1653,7 @@ mod tests {
anthropic_max_tokens: 4096,
stream_start_instant: None,
timeout_secs: None,
+ pending_ws: String::new(),
}
}
@@ -1693,7 +1725,8 @@ mod tests {
assert_eq!(events[0].original, "hello");
assert!(!events[0].transformed);
assert_eq!(events[0].index, 0);
- assert_eq!(events[1].original, "world");
+ // The space before "world" rides on the token (issue #3).
+ assert_eq!(events[1].original, " world");
assert!(events[1].transformed);
assert_eq!(events[1].index, 1);
}
@@ -1710,9 +1743,9 @@ mod tests {
while let Ok(e) = rx.try_recv() {
events.push(e);
}
- // "world" reversed = "dlrow"
- assert_eq!(events[1].text, "dlrow");
- assert_eq!(events[1].original, "world");
+ // "world" reversed = "dlrow"; the transform only touches the word.
+ assert_eq!(events[1].text, " dlrow");
+ assert_eq!(events[1].original, " world");
}
#[test]
@@ -1828,6 +1861,50 @@ mod tests {
}
}
+ // -- issue #3: web / JSON consumers keep the spaces between words --
+ #[test]
+ fn test_web_events_keep_whitespace() {
+ let (tx, mut rx) = mpsc::unbounded_channel::();
+ let mut interceptor = make_test_interceptor();
+ interceptor.web_tx = Some(tx);
+
+ // Chunks as a real provider sends them: leading spaces, and one
+ // chunk that ends in whitespace.
+ interceptor.process_content_logprob(" quick brown", None, vec![]);
+ interceptor.process_content_logprob(" fox ", None, vec![]);
+ interceptor.process_content_logprob("jumps", None, vec![]);
+
+ let mut events = Vec::new();
+ while let Ok(e) = rx.try_recv() {
+ events.push(e);
+ }
+ let original: String = events.iter().map(|e| e.original.as_str()).collect();
+ assert_eq!(original, " quick brown fox jumps");
+ let shown: String = events.iter().map(|e| e.text.as_str()).collect();
+ assert_eq!(shown, " quick nworb fox spmuj");
+ // Counting and alternation are unchanged by the whitespace.
+ assert_eq!(events.iter().map(|e| e.index).collect::>(), vec![0, 1, 2, 3]);
+ assert_eq!(interceptor.token_count, 4);
+ assert_eq!(interceptor.transformed_count, 2);
+ }
+
+ #[test]
+ fn test_web_deleted_token_keeps_whitespace_pending() {
+ let (tx, mut rx) = mpsc::unbounded_channel::();
+ let mut interceptor = make_test_interceptor();
+ interceptor.transform = Transform::Delete;
+ interceptor.web_tx = Some(tx);
+
+ interceptor.process_content("one two three");
+
+ let mut events = Vec::new();
+ while let Ok(e) = rx.try_recv() {
+ events.push(e);
+ }
+ let shown: String = events.iter().map(|e| e.text.as_str()).collect();
+ assert_eq!(shown, "one three");
+ }
+
#[test]
fn test_sidebyside_original_is_raw_token() {
let (tx, mut rx) = mpsc::unbounded_channel::();
@@ -1843,8 +1920,8 @@ mod tests {
let originals: Vec<&str> = events.iter().map(|e| e.original.as_str()).collect();
assert!(originals.contains(&"quick"));
- assert!(originals.contains(&"brown"));
- assert!(originals.contains(&"fox"));
+ assert!(originals.contains(&" brown"));
+ assert!(originals.contains(&" fox"));
}
// -- even/odd alternation for graph --
@@ -2046,8 +2123,8 @@ mod tests {
while let Ok(e) = rx.try_recv() {
events.push(e);
}
- assert_eq!(events[1].text, "WORLD");
- assert_eq!(events[1].original, "world");
+ assert_eq!(events[1].text, " WORLD");
+ assert_eq!(events[1].original, " world");
}
#[test]
@@ -2063,7 +2140,7 @@ mod tests {
while let Ok(e) = rx.try_recv() {
events.push(e);
}
- assert_eq!(events[1].text, "wOrLd");
+ assert_eq!(events[1].text, " wOrLd");
}
#[test]
@@ -2079,8 +2156,8 @@ mod tests {
while let Ok(e) = rx.try_recv() {
events.push(e);
}
- assert!(events[1].text.starts_with("world"));
- assert_eq!(events[1].text.len(), 6); // "world" + 1 noise char
+ assert!(events[1].text.starts_with(" world"));
+ assert_eq!(events[1].text.len(), 7); // " world" + 1 noise char
}
// -- web_tx none falls back to terminal mode --
@@ -2578,6 +2655,7 @@ mod research_tests {
anthropic_max_tokens: 4096,
stream_start_instant: None,
timeout_secs: None,
+ pending_ws: String::new(),
}
}
@@ -2792,7 +2870,8 @@ mod research_tests {
while let Ok(ev) = rx.try_recv() {
originals.push(ev.original);
}
- let expected: String = prompt.chars().take(20).collect();
+ // The mock echoes the prompt (up to 40 characters) as one token.
+ let expected: String = format!(" {}", prompt.chars().take(40).collect::());
assert!(originals.contains(&expected), "prompt prefix token missing");
}
@@ -2813,9 +2892,9 @@ mod research_tests {
.intercept_stream("hello")
.await
.expect("mock stream should succeed");
- // 19 fixture tokens plus the echoed prompt, one word each.
- assert_eq!(interceptor.token_count, 20);
- assert_eq!(interceptor.transformed_count, 10);
+ // 18 fixture tokens plus the echoed prompt, one word each.
+ assert_eq!(interceptor.token_count, 19);
+ assert_eq!(interceptor.transformed_count, 9);
}
// -- run_research_headless tests (Mock provider, no API key required) --
diff --git a/src/web.rs b/src/web.rs
index e903141f..0af95285 100644
--- a/src/web.rs
+++ b/src/web.rs
@@ -251,21 +251,21 @@ pub async fn serve(port: u16, default_args: &Args) -> Result<(), Box Provider::Anthropic,
+ "mock" => Provider::Mock,
_ => Provider::Openai,
};
@@ -750,6 +751,7 @@ async fn handle_connection(
let ab_provider = match provider_str.as_str() {
"anthropic" => Provider::Anthropic,
+ "mock" => Provider::Mock,
_ => Provider::Openai,
};
let transform = Transform::from_str_loose(&transform_str).unwrap_or(Transform::Reverse);
@@ -1173,7 +1175,7 @@ mod tests {
#[test]
fn test_index_html_has_dark_theme() {
- assert!(INDEX_HTML.contains("background:#0d1117"));
+ assert!(INDEX_HTML.contains("--bg:#0c0d0b"));
}
#[test]
diff --git a/static/index.html b/static/index.html
index dda6d8d2..c7c214e6 100644
--- a/static/index.html
+++ b/static/index.html
@@ -6,26 +6,26 @@
Every Other Token
-
Every Other Token
+
every-other-token
@@ -227,14 +253,14 @@
Every Other Token
-
-
+
+
-
-
-
+
+
+
@@ -256,17 +282,17 @@
Every Other Token
-
none
+
none
-
+
-
- 0%
+
+ 0%
@@ -297,100 +323,100 @@
Every Other Token
System Prompt A
System Prompt B
-
Perplexity comparison (A=blue, B=purple) — run streams to populate
-
Divergence map — run streams to populate
+
Perplexity comparison (A=gold, B=purple): run streams to populate
+
Divergence map: run streams to populate
-
-
-
Token Agreement — first 50 positions (green=match, red=diverge)
+
+
+
Token Agreement: first 50 positions (green=match, red=diverge)
-
Research Dashboard
+
Research Dashboard
Research runs:
-
+
-
Stats computed from the most recent stream. Stream tokens first.
+
Stats computed from the most recent stream. Stream tokens first.
-
Top 10 High-Perplexity Tokens
-
+
Top 10 High-Perplexity Tokens
+
-
Perplexity Distribution
-
-
Confidence Distribution
+
Confidence Distribution
-
0.00.51.0
+
0.00.51.0
-
A/B Significance Test (Even vs Odd Runs)
-
- Run at least 4 tokens from two alternating runs to see results.
+
A/B Significance Test (Even vs Odd Runs)
+
+ Run at least 4 tokens from two alternating runs to see results.
return s;
}
+/* Append a token span; its leading whitespace goes in as plain text so the
+ confidence underline and line wrapping only apply to the word itself. */
+function appendTok(container,sp){
+ const ws=sp.dataset.ws;
+ if(ws){container.appendChild(document.createTextNode(ws));delete sp.dataset.ws;}
+ container.appendChild(sp);
+}
+
/* ---- Token search/filter ---- */
$('#token-search').addEventListener('input',function(){
const q=this.value.toLowerCase();
@@ -583,7 +621,7 @@
Keyboard Shortcuts
if (!svg) return;
const tip = document.createElement('div');
tip.id = 'spark-tip';
- tip.style.cssText = 'display:none;position:fixed;background:#1c2333;color:#a371f7;padding:2px 8px;border-radius:4px;font-size:.7rem;pointer-events:none;z-index:100;border:1px solid #30363d';
+ tip.style.cssText = 'display:none;position:fixed;background:#1c2333;color:#a371f7;padding:2px 8px;border-radius:4px;font-size:.7rem;pointer-events:none;z-index:100;border:1px solid #36382f';
document.body.appendChild(tip);
svg.addEventListener('mousemove', e => {
if (!perpWindow.length) return;
@@ -627,17 +665,17 @@
Keyboard Shortcuts
if (!el) return;
const r = welchT(evenConf, oddConf);
if (!r) {
- el.innerHTML = 'Not enough confidence data for t-test (need OpenAI provider).';
+ el.innerHTML = 'Not enough confidence data for t-test (need OpenAI provider).';
return;
}
const sig = r.p < 0.05;
const color = sig ? '#f85149' : '#3fb950';
el.innerHTML = `
-
Even tokens mean conf:${r.ma.toFixed(4)} (n=${r.na})
-
Odd tokens mean conf:${r.mb.toFixed(4)} (n=${r.nb})
-
t-statistic:${r.t.toFixed(3)}
-
p-value:${r.p.toFixed(4)}
+
Even tokens mean conf:${r.ma.toFixed(4)} (n=${r.na})
+
Odd tokens mean conf:${r.mb.toFixed(4)} (n=${r.nb})
+
t-statistic:${r.t.toFixed(3)}
+
p-value:${r.p.toFixed(4)}
${sig?'⚠ SIGNIFICANT (p<0.05)':'✓ Not significant'}
+
+ 
+ 
+
+The `mock` provider replays a fixed reply with fixed logprobs through the real interception pipeline, so everything above is real output; only the model is canned. Set `OPENAI_API_KEY` or `ANTHROPIC_API_KEY` and drop `--provider mock` to intercept a real model.
+
+## What you get
+
+**Every token carries its numbers.** Each intercepted token is an event with the original text, what was shown, its index, whether it was rewritten, confidence `exp(logprob)`, perplexity `exp(-logprob)` and the top alternatives. Read them from the library, stream them as JSON lines with `--json-stream`, or export CSV, JSONL or a self-contained HTML heatmap.
+
+
+
+`cargo run --example mock_stream` in a clone of this repo.
+
+**A web UI for watching the stream.** `--web` serves a single page (no build step, no external scripts): single, split and quad views, OpenAI vs Anthropic diff, A/B system prompts, a research dashboard, export, and collaborative rooms where several people edit tokens mid-stream. Pick **Mock (no API key)** in the provider menu to try it offline.
+
+
+
+**Compare, test, record.** Diff two providers (`--diff-terminal`), A/B two system prompts with a Welch's t-test (`--significance`), run headless research batches (`--research --runs 20`), and record any session to replay it after a model update.
## Install
@@ -49,10 +82,11 @@ The Linux binary links the system OpenSSL (libssl3, present on Ubuntu 22.04+ and
| Collaborative rooms | No | Yes: WebSocket multi-participant token surgery |
| TF-IDF semantic heatmaps | No | Yes: no embedding service required |
----
-
## Use cases
+