From 555c508b44a8bb522234aac34e0f2f2af0f9a522 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E8=B5=B5=E8=90=8D?= Date: Thu, 29 Oct 2026 03:41:32 +0000 Subject: [PATCH 1/9] feat(compare): add offline HTML and Markdown comparison reports --- CHANGELOG.md | 6 + README.md | 1 + README.zh.md | 1 + docs/guide/cli-reference.md | 72 ++++- internal/cli/compare.go | 208 +++++++++++++ internal/cli/compare_test.go | 382 ++++++++++++++++++++++++ internal/cli/root.go | 1 + internal/compare/compare.go | 275 +++++++++++++++++ internal/compare/compare_test.go | 282 +++++++++++++++++ internal/compare/gates.go | 34 +++ internal/compare/input.go | 47 +++ internal/compare/render.go | 204 +++++++++++++ internal/compare/render_test.go | 139 +++++++++ internal/compare/templates/compare.html | 71 +++++ internal/compare/text.go | 64 ++++ 15 files changed, 1786 insertions(+), 1 deletion(-) create mode 100644 internal/cli/compare.go create mode 100644 internal/cli/compare_test.go create mode 100644 internal/compare/compare.go create mode 100644 internal/compare/compare_test.go create mode 100644 internal/compare/gates.go create mode 100644 internal/compare/input.go create mode 100644 internal/compare/render.go create mode 100644 internal/compare/render_test.go create mode 100644 internal/compare/templates/compare.html create mode 100644 internal/compare/text.go diff --git a/CHANGELOG.md b/CHANGELOG.md index 02f6d892..b05a1c5c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Added +- Add `skill-up compare ` for offline + comparison with self-contained HTML, shareable Markdown, text, and JSON + reports, case coverage and transitions, explicit baseline/candidate labels, + and optional regression and token-growth gates. + ### Changed - Strengthen `skill-upper` evolution with original/candidate comparisons, evidence-based acceptance and reversion, and regression checks. Keep the diff --git a/README.md b/README.md index 96542523..56193ab5 100644 --- a/README.md +++ b/README.md @@ -408,6 +408,7 @@ skill-up import ./evals/evals.json --output ./evals | `skill-up run [path]` | Run evaluation cases and produce reports | | `skill-up validate [path]` | Validate `eval.yaml` and case files | | `skill-up list-cases [path]` | List all cases referenced by the config | +| `skill-up compare ` | Compare finished runs; export HTML, Markdown, JSON or text | | `skill-up report ` | Generate reports from a previous run | | `skill-up import ` | Import Anthropic `evals.json` to YAML cases | | `skill-up debug judge ` | Debug judge module with a JSON input | diff --git a/README.zh.md b/README.zh.md index a1e82a94..55071710 100644 --- a/README.zh.md +++ b/README.zh.md @@ -315,6 +315,7 @@ Engine 和凭据。它与旧式 `judge.model` 互斥，后者复用被测 Agent | `skill-up run [path]` | 运行评测用例并生成报告 | | `skill-up validate [path]` | 校验 `eval.yaml` 和用例文件 | | `skill-up list-cases [path]` | 列出配置引用的所有用例 | +| `skill-up compare ` | 离线对比评测结果，支持 HTML、Markdown、JSON 和文本 | | `skill-up report ` | 从已有结果生成报告 | | `skill-up import ` | 将 Anthropic `evals.json` 导入为 YAML 用例 | | `skill-up debug judge ` | 使用 JSON 输入调试 judge 模块 | diff --git a/docs/guide/cli-reference.md b/docs/guide/cli-reference.md index b028e2b0..a41d5fe3 100644 --- a/docs/guide/cli-reference.md +++ b/docs/guide/cli-reference.md @@ -234,7 +234,77 @@ skill-up report result.json --format html --output-dir ./reports | `html` | `report.html` | Human-readable visualization; open in a browser | | `markdown` | `report.md` | Concise Markdown summary for CI logs and GitHub PR comments | -`markdown` is intentionally limited to `skill-up report` as an offline conversion format for an existing `result.json`. It is not currently accepted by `skill-up run --format`, `skill-up debug report --format`, `skill-up debug judge --report`, or `eval.yaml report.formats`. +For single-run reports, `markdown` is limited to `skill-up report` as an offline conversion format for an existing `result.json`. It is not currently accepted by `skill-up run --format`, `skill-up debug report --format`, `skill-up debug judge --report`, or `eval.yaml report.formats`. + +--- + +## skill-up compare + +Compare two existing canonical `result.json` (or equivalent `report.json`) +files without running evaluations or calling any Agent or Judge. + +```bash +skill-up compare [flags] +``` + +### Flags + +| Flag | Default | Description | +|---|---|---| +| `--format` | `text` | `text`, `json`, `html`, or `markdown` | +| `--output` | stdout | Write the comparison to this file; its parent directory must exist | +| `--baseline-label` | `Baseline` | Display name for the old run | +| `--candidate-label` | `Candidate` | Display name for the new run | +| `--fail-on-regression` | `false` | Return a non-zero exit code for any shared case moving from PASS to another status | +| `--max-token-increase-percent` | disabled | Maximum permitted increase in run-level total tokens (finite, non-negative percentage) | + +### Examples + +```bash +# Self-contained HTML, with searchable cases and transition filters +skill-up compare old/result.json new/result.json --format html --output comparison.html + +# Markdown tables for PR reviews and CI artifacts +skill-up compare old/result.json new/result.json --format markdown --output comparison.md \ + --baseline-label "Released Skill" --candidate-label "Proposed Skill" + +# Produce a report even if the regression gate fails +skill-up compare old/result.json new/result.json --format html --output comparison.html \ + --fail-on-regression --max-token-increase-percent 20 + +# Stable machine-readable output +skill-up compare old/result.json new/result.json --format json +``` + +All formats use the same comparison and gates. Reports include run metrics, +metadata, original source paths, and fixed, regressed, changed, unchanged, +added, and removed cases. HTML embeds its styles and filtering script, with +no external assets or network requests; all cases remain readable without +JavaScript. Markdown includes tables suitable for GitHub PR descriptions. +Source result artifacts are preserved; output may not overwrite either input. + +Cases align by case ID using the existing primary-result rule: prefer +`with_skill`, then a non-baseline configuration, falling back to `without_skill`. +Benchmark reference entries do not double-count case counts or pass rates. +Configuration identity is retained in case tables. Shared cases flag prompt changes +when both prompts were recorded; otherwise the prompt comparison is unknown. +Requested, applied, and observed agent configurations are compared when recorded. Total tokens are the recorded +run-level total across all configurations; input/output tokens sum primary cases. +Duration is run wall time, and pass-rate changes are percentage points. + +Added and removed cases are shown separately, with shared-case coverage. Aggregate +changes include each run's full primary case set, so different case sets are not +paired samples. A change between two non-PASS statuses is classified as changed, +not fixed or regressed. Recorded zero tokens remain zero; missing token metrics +are `unknown` in human-readable reports and `null` in JSON, including deltas. +The token gate fails when the total-token measurement is unknown, or when a zero +baseline increases to a positive value. + +Reports compare recorded outcomes only and do not regrade scores. Missing +provenance is unknown; matching case IDs do not establish identical prompts, +suites, Skill revisions, or judge/rubric definitions. Review those conditions +before interpreting a change as an improvement. Gates are opt-in, apply equally +to every output format, and write the report before returning a gate failure. --- diff --git a/internal/cli/compare.go b/internal/cli/compare.go new file mode 100644 index 00000000..3fcfda6e --- /dev/null +++ b/internal/cli/compare.go @@ -0,0 +1,208 @@ +package cli + +import ( + "encoding/json" + "errors" + "fmt" + "math" + "os" + "path/filepath" + "strings" + + "github.com/spf13/cobra" + + comparepkg "github.com/alibaba/skill-up/internal/compare" + "github.com/alibaba/skill-up/internal/judge" + "github.com/alibaba/skill-up/internal/report" +) + +var compareCmd = &cobra.Command{ + Use: "compare ", + Short: "Compare two offline evaluation results", + Long: `Compare existing results without running an Agent or Judge. + +Examples: + skill-up compare old.json new.json --format html --output compare.html + skill-up compare old.json new.json --format markdown --output compare.md + skill-up compare old.json new.json --format json --fail-on-regression`, + Args: cobra.ExactArgs(2), + RunE: runCompare, +} + +func init() { + configureCompareFlags(compareCmd) +} + +func configureCompareFlags(cmd *cobra.Command) { + cmd.Flags().String("format", "text", "Output format: text, json, html, markdown") + cmd.Flags().String("output", "", "Write report to a file instead of stdout") + cmd.Flags().String("baseline-label", "Baseline", "Label for the old run") + cmd.Flags().String("candidate-label", "Candidate", "Label for the new run") + cmd.Flags().Bool("fail-on-regression", false, "Fail when any case regresses") + cmd.Flags().Float64("max-token-increase-percent", 0, "Maximum allowed total token increase percentage") +} + +func runCompare(cmd *cobra.Command, args []string) error { + format, err := cmd.Flags().GetString("format") + if err != nil { + return fmt.Errorf("get format flag: %w", err) + } + switch format { + case "text", "json", "html", "markdown": + default: + return fmt.Errorf("unsupported compare format %q; supported formats: text, json, html, markdown", format) + } + + oldInput, oldMissing, err := loadCompareInput("old", args[0]) + if err != nil { + return err + } + newInput, newMissing, err := loadCompareInput("new", args[1]) + if err != nil { + return err + } + + failOnRegression, err := cmd.Flags().GetBool("fail-on-regression") + if err != nil { + return fmt.Errorf("get fail-on-regression flag: %w", err) + } + options := comparepkg.Options{FailOnRegression: failOnRegression, OldMissingMetrics: oldMissing, NewMissingMetrics: newMissing} + maxTokenIncreasePercent, err := maxTokenIncreasePercent(cmd) + if err != nil { + return err + } + if maxTokenIncreasePercent != nil { + options.MaxTokenIncreasePercent = maxTokenIncreasePercent + } + + result := comparepkg.Compare(oldInput, newInput, options) + baselineLabel, _ := cmd.Flags().GetString("baseline-label") + candidateLabel, _ := cmd.Flags().GetString("candidate-label") + result.Baseline = comparepkg.Source{Label: baselineLabel, Path: args[0]} + result.Candidate = comparepkg.Source{Label: candidateLabel, Path: args[1]} + data, err := comparepkg.Render(result, format) + if err != nil { + return err + } + output, _ := cmd.Flags().GetString("output") + if err := writeComparison(cmd, args, output, data); err != nil { + return err + } + + if !result.Gates.Passed { + return fmt.Errorf("comparison gates failed: %v", result.Gates.Failures) + } + return nil +} + +func writeComparison(cmd *cobra.Command, inputs []string, output string, data []byte) error { + if output == "" { + if _, err := cmd.OutOrStdout().Write(data); err != nil { + return fmt.Errorf("write comparison: %w", err) + } + return nil + } + outPath, err := filepath.Abs(output) + if err != nil { + return fmt.Errorf("resolve output path: %w", err) + } + outInfo, statErr := os.Stat(output) + if statErr != nil && !os.IsNotExist(statErr) { + return fmt.Errorf("inspect output: %w", statErr) + } + for _, input := range inputs { + inPath, err := filepath.Abs(input) + if err != nil { + return fmt.Errorf("resolve input path: %w", err) + } + inInfo, err := os.Stat(input) + if err != nil { + return fmt.Errorf("inspect input: %w", err) + } + if outPath == inPath || (outInfo != nil && os.SameFile(outInfo, inInfo)) { + return errors.New("comparison output must not overwrite a source result") + } + } + if err := os.WriteFile(output, data, 0o600); err != nil { + return fmt.Errorf("write comparison %q: %w", output, err) + } + return nil +} + +func maxTokenIncreasePercent(cmd *cobra.Command) (*float64, error) { + if !cmd.Flags().Changed("max-token-increase-percent") { + return nil, nil + } + value, err := cmd.Flags().GetFloat64("max-token-increase-percent") + if err != nil { + return nil, fmt.Errorf("get max-token-increase-percent flag: %w", err) + } + if value < 0 || math.IsNaN(value) || math.IsInf(value, 0) { + return nil, errors.New("max-token-increase-percent must be a finite non-negative number") + } + return &value, nil +} + +func loadCompareInput(role, path string) (report.Input, []string, error) { + data, err := os.ReadFile(path) + if err != nil { + return report.Input{}, nil, fmt.Errorf("read %s result %q: %w", role, path, err) + } + + var input report.Input + if err := json.Unmarshal(data, &input); err != nil { + return report.Input{}, nil, fmt.Errorf("parse %s result %q: %w", role, path, err) + } + if err := validateCompareInput(input); err != nil { + return report.Input{}, nil, fmt.Errorf("validate %s result %q: %w", role, path, err) + } + return input, comparepkg.MissingMetrics(data, input), nil +} + +func validateCompareInput(input report.Input) error { + if input.SkillName == "" || input.SchemaVersion == "" || input.EngineName == "" || input.StartTime.IsZero() || input.EndTime.IsZero() { + return errors.New("missing required result metadata") + } + if input.EndTime.Before(input.StartTime) || input.TotalTokens < 0 { + return errors.New("invalid run duration or total tokens") + } + primary := input.PrimaryCaseResults() + if len(primary) == 0 { + return errors.New("no primary case results") + } + seen := make(map[[2]string]bool) + for _, result := range input.CaseResults { + key := [2]string{result.CaseID, result.Configuration} + if seen[key] { + return fmt.Errorf("duplicate case ID %q for configuration %q", result.CaseID, result.Configuration) + } + seen[key] = true + if err := validateCompareCase(result); err != nil { + return err + } + } + return nil +} + +func validateCompareCase(result report.CaseResult) error { + if result.DurationMs < 0 || result.InputTokens < 0 || result.OutputTokens < 0 { + return fmt.Errorf("case %q has negative metrics", result.CaseID) + } + if strings.TrimSpace(result.CaseID) == "" { + return errors.New("case result has an empty case ID") + } + if !isCompareStatus(result.Status) { + return fmt.Errorf("case %q has invalid status %q", result.CaseID, result.Status) + } + + return nil +} + +func isCompareStatus(status judge.Status) bool { + switch status { + case judge.StatusPass, judge.StatusFail, judge.StatusSkip, judge.StatusError: + return true + default: + return false + } +} diff --git a/internal/cli/compare_test.go b/internal/cli/compare_test.go new file mode 100644 index 00000000..2f269f57 --- /dev/null +++ b/internal/cli/compare_test.go @@ -0,0 +1,382 @@ +package cli + +import ( + "bytes" + "encoding/json" + "math" + "os" + "path/filepath" + "strconv" + "strings" + "testing" + + "github.com/spf13/cobra" + + "github.com/alibaba/skill-up/internal/judge" + "github.com/alibaba/skill-up/internal/report" +) + +// newCompareCmd creates an isolated cobra command wired to runCompare for testing. +func newCompareCmd() *cobra.Command { + cmd := &cobra.Command{RunE: runCompare} + configureCompareFlags(cmd) + return cmd +} + +func TestRunCompare_DefaultTextFormat(t *testing.T) { + t.Parallel() + dir := t.TempDir() + oldPath := writeCompareResultJSON(t, dir, "old.json", makeReportInput(judge.StatusFail)) + newPath := writeCompareResultJSON(t, dir, "new.json", makeReportInput(judge.StatusPass)) + + cmd := newCompareCmd() + var out bytes.Buffer + cmd.SetOut(&out) + + if err := runCompare(cmd, []string{oldPath, newPath}); err != nil { + t.Fatalf("runCompare error: %v", err) + } + + for _, section := range []string{"Run summary", "Metadata differences", "Case transitions", "Gates: passed"} { + if !strings.Contains(out.String(), section) { + t.Errorf("text output should contain %q, got: %s", section, out.String()) + } + } +} + +func TestRunCompare_AllowsEmptyModelName(t *testing.T) { + t.Parallel() + dir := t.TempDir() + oldInput := makeReportInput(judge.StatusFail) + oldInput.ModelName = "" + newInput := makeReportInput(judge.StatusPass) + newInput.ModelName = "" + oldPath := writeCompareResultJSON(t, dir, "old.json", oldInput) + newPath := writeCompareResultJSON(t, dir, "new.json", newInput) + + cmd := newCompareCmd() + var out bytes.Buffer + cmd.SetOut(&out) + + if err := runCompare(cmd, []string{oldPath, newPath}); err != nil { + t.Fatalf("runCompare error: %v", err) + } + if !strings.Contains(out.String(), "Run summary") { + t.Errorf("text output should contain run summary, got: %s", out.String()) + } +} + +func TestRunCompare_JSONFormat(t *testing.T) { + t.Parallel() + dir := t.TempDir() + oldPath := writeCompareResultJSON(t, dir, "old.json", makeReportInput(judge.StatusFail)) + newPath := writeCompareResultJSON(t, dir, "new.json", makeReportInput(judge.StatusPass)) + + cmd := newCompareCmd() + var out bytes.Buffer + cmd.SetOut(&out) + if err := cmd.Flags().Set("format", "json"); err != nil { + t.Fatalf("set format: %v", err) + } + + if err := runCompare(cmd, []string{oldPath, newPath}); err != nil { + t.Fatalf("runCompare error: %v", err) + } + + var result struct { + Run json.RawMessage `json:"run"` + Cases json.RawMessage `json:"cases"` + Gates json.RawMessage `json:"gates"` + } + if err := json.Unmarshal(out.Bytes(), &result); err != nil { + t.Fatalf("unmarshal compare output: %v", err) + } + if result.Run == nil || result.Cases == nil || result.Gates == nil { + t.Fatalf("JSON output missing stable result fields: %s", out.String()) + } +} + +func TestRunCompare_RegressionGateWritesTextBeforeReturningError(t *testing.T) { + t.Parallel() + dir := t.TempDir() + oldPath := writeCompareResultJSON(t, dir, "old.json", makeReportInput(judge.StatusPass)) + newPath := writeCompareResultJSON(t, dir, "new.json", makeReportInput(judge.StatusFail)) + + cmd := newCompareCmd() + var out bytes.Buffer + cmd.SetOut(&out) + if err := cmd.Flags().Set("fail-on-regression", "true"); err != nil { + t.Fatalf("set fail-on-regression flag: %v", err) + } + + err := runCompare(cmd, []string{oldPath, newPath}) + if err == nil || !strings.Contains(err.Error(), "comparison gates failed") { + t.Fatalf("runCompare error = %v, want gate failure", err) + } + for _, want := range []string{"Run summary", "Gates: failed", "1 case(s) regressed"} { + if !strings.Contains(out.String(), want) { + t.Errorf("text output should contain %q before returning error, got: %s", want, out.String()) + } + } +} + +func TestRunCompare_TokenGateWritesJSONBeforeReturningError(t *testing.T) { + t.Parallel() + dir := t.TempDir() + oldInput := makeReportInput(judge.StatusPass) + oldInput.TotalTokens = 100 + newInput := makeReportInput(judge.StatusPass) + newInput.TotalTokens = 150 + oldPath := writeCompareResultJSON(t, dir, "old.json", oldInput) + newPath := writeCompareResultJSON(t, dir, "new.json", newInput) + + cmd := newCompareCmd() + var out bytes.Buffer + cmd.SetOut(&out) + if err := cmd.Flags().Set("format", "json"); err != nil { + t.Fatalf("set format flag: %v", err) + } + if err := cmd.Flags().Set("max-token-increase-percent", "20"); err != nil { + t.Fatalf("set max-token-increase-percent flag: %v", err) + } + + err := runCompare(cmd, []string{oldPath, newPath}) + if err == nil || !strings.Contains(err.Error(), "comparison gates failed") { + t.Fatalf("runCompare error = %v, want gate failure", err) + } + + var result struct { + Gates struct { + Passed bool `json:"passed"` + Failures []string `json:"failures"` + } `json:"gates"` + } + if err := json.Unmarshal(out.Bytes(), &result); err != nil { + t.Fatalf("unmarshal JSON output written before error: %v\noutput: %s", err, out.String()) + } + if result.Gates.Passed || len(result.Gates.Failures) != 1 || !strings.Contains(result.Gates.Failures[0], "50.00%") { + t.Fatalf("JSON gates = %#v, want failed token increase gate", result.Gates) + } +} + +func TestRunCompare_InvalidInputsIncludeRoleAndPath(t *testing.T) { + t.Parallel() + dir := t.TempDir() + validPath := writeCompareResultJSON(t, dir, "valid.json", makeReportInput(judge.StatusPass)) + invalidPath := filepath.Join(dir, "invalid.json") + if err := os.WriteFile(invalidPath, []byte("not-json"), 0o600); err != nil { + t.Fatalf("write invalid input: %v", err) + } + + tests := []struct { + name string + args []string + role string + path string + }{ + {name: "old", args: []string{invalidPath, validPath}, role: "old", path: invalidPath}, + {name: "new", args: []string{validPath, invalidPath}, role: "new", path: invalidPath}, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + cmd := newCompareCmd() + cmd.SetOut(&bytes.Buffer{}) + + err := runCompare(cmd, tc.args) + if err == nil { + t.Fatal("expected invalid JSON error, got nil") + } + if !strings.Contains(err.Error(), tc.role) || !strings.Contains(err.Error(), tc.path) { + t.Errorf("error should identify %s input %q, got: %v", tc.role, tc.path, err) + } + }) + } +} + +func TestRunCompare_RejectsStructurallyInvalidInputs(t *testing.T) { + t.Parallel() + dir := t.TempDir() + validPath := writeCompareResultJSON(t, dir, "valid.json", makeReportInput(judge.StatusPass)) + + tests := []struct { + name string + input report.Input + wantError string + }{ + {name: "empty result", input: report.Input{}, wantError: "missing required result metadata"}, + { + name: "blank case ID", + input: func() report.Input { + input := makeReportInput(judge.StatusPass) + input.CaseResults = []report.CaseResult{{CaseID: " ", Status: judge.StatusPass}} + return input + }(), + wantError: "case result has an empty case ID", + }, + { + name: "invalid status", + input: func() report.Input { + input := makeReportInput(judge.StatusPass) + input.CaseResults = []report.CaseResult{{CaseID: "case-1", Status: "UNKNOWN"}} + return input + }(), + wantError: "case \"case-1\" has invalid status \"UNKNOWN\"", + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + invalidPath := writeCompareResultJSON(t, dir, tc.name+".json", tc.input) + cmd := newCompareCmd() + cmd.SetOut(&bytes.Buffer{}) + + err := runCompare(cmd, []string{invalidPath, validPath}) + if err == nil { + t.Fatal("expected structural validation error, got nil") + } + if !strings.Contains(err.Error(), tc.wantError) { + t.Errorf("error = %v, want to contain %q", err, tc.wantError) + } + if !strings.Contains(err.Error(), "old") || !strings.Contains(err.Error(), invalidPath) { + t.Errorf("error should identify old input %q, got: %v", invalidPath, err) + } + }) + } +} + +func TestRunCompare_RejectsInvalidMaxTokenIncreasePercent(t *testing.T) { + t.Parallel() + dir := t.TempDir() + oldPath := writeCompareResultJSON(t, dir, "old.json", makeReportInput(judge.StatusPass)) + newPath := writeCompareResultJSON(t, dir, "new.json", makeReportInput(judge.StatusPass)) + + for _, value := range []float64{-1, math.NaN(), math.Inf(1)} { + t.Run("invalid value", func(t *testing.T) { + t.Parallel() + cmd := newCompareCmd() + cmd.SetOut(&bytes.Buffer{}) + if err := cmd.Flags().Set("max-token-increase-percent", strconv.FormatFloat(value, 'g', -1, 64)); err != nil { + t.Fatalf("set max-token-increase-percent: %v", err) + } + + err := runCompare(cmd, []string{oldPath, newPath}) + if err == nil || !strings.Contains(err.Error(), "max-token-increase-percent") { + t.Fatalf("runCompare error = %v, want invalid token limit error", err) + } + }) + } +} + +func TestRunCompare_UnsupportedFormat(t *testing.T) { + t.Parallel() + dir := t.TempDir() + oldPath := writeCompareResultJSON(t, dir, "old.json", makeReportInput(judge.StatusPass)) + newPath := writeCompareResultJSON(t, dir, "new.json", makeReportInput(judge.StatusPass)) + + cmd := newCompareCmd() + cmd.SetOut(&bytes.Buffer{}) + if err := cmd.Flags().Set("format", "csv"); err != nil { + t.Fatalf("set format: %v", err) + } + + err := runCompare(cmd, []string{oldPath, newPath}) + if err == nil { + t.Fatal("expected unsupported format error, got nil") + } + if !strings.Contains(err.Error(), "csv") { + t.Errorf("error should name unsupported format, got: %v", err) + } +} + +func writeCompareResultJSON(t *testing.T, dir, name string, input report.Input) string { + t.Helper() + data, err := json.Marshal(input) + if err != nil { + t.Fatalf("marshal report input: %v", err) + } + path := filepath.Join(dir, name) + if err := os.WriteFile(path, data, 0o600); err != nil { + t.Fatalf("write report input: %v", err) + } + return path +} + +func TestRunCompareShareableFilesSurviveGateFailure(t *testing.T) { + t.Parallel() + for _, format := range []string{"html", "markdown"} { + t.Run(format, func(t *testing.T) { + t.Parallel() + dir := t.TempDir() + oldPath := writeCompareResultJSON(t, dir, "old.json", makeReportInput(judge.StatusPass)) + newPath := writeCompareResultJSON(t, dir, "new.json", makeReportInput(judge.StatusFail)) + original, err := os.ReadFile(oldPath) + if err != nil { + t.Fatal(err) + } + output := filepath.Join(dir, "comparison."+format) + cmd := newCompareCmd() + var stdout bytes.Buffer + cmd.SetOut(&stdout) + for flag, value := range map[string]string{"format": format, "output": output, "fail-on-regression": "true", "baseline-label": "Before", "candidate-label": "After"} { + if err := cmd.Flags().Set(flag, value); err != nil { + t.Fatal(err) + } + } + if err := runCompare(cmd, []string{oldPath, newPath}); err == nil || !strings.Contains(err.Error(), "comparison gates failed") { + t.Fatalf("expected gate failure: %v", err) + } + data, err := os.ReadFile(output) + if err != nil { + t.Fatal(err) + } + for _, want := range []string{"Before", "After", "Regressed (1)", "1 case(s) regressed"} { + if !strings.Contains(string(data), want) { + t.Errorf("%s missing %q", format, want) + } + } + if stdout.Len() != 0 { + t.Fatalf("file output leaked into stdout: %s", stdout.String()) + } + retained, err := os.ReadFile(oldPath) + if err != nil { + t.Fatal(err) + } + if !bytes.Equal(original, retained) { + t.Fatal("source artifact modified") + } + }) + } +} + +func TestRunCompareProtectsSourceArtifacts(t *testing.T) { + t.Parallel() + dir := t.TempDir() + oldPath := writeCompareResultJSON(t, dir, "old.json", makeReportInput(judge.StatusPass)) + newPath := writeCompareResultJSON(t, dir, "new.json", makeReportInput(judge.StatusPass)) + linked := filepath.Join(dir, "linked.json") + if err := os.Link(oldPath, linked); err != nil { + t.Fatal(err) + } + for _, output := range []string{oldPath, newPath, linked} { + cmd := newCompareCmd() + if err := cmd.Flags().Set("output", output); err != nil { + t.Fatal(err) + } + if err := runCompare(cmd, []string{oldPath, newPath}); err == nil || !strings.Contains(err.Error(), "must not overwrite") { + t.Fatalf("output %q: %v", output, err) + } + } +} + +func TestRootCommandIncludesCompare(t *testing.T) { + command, _, err := rootCmd.Find([]string{"compare"}) + if err != nil { + t.Fatal(err) + } + if command != compareCmd { + t.Fatal("compare command not registered") + } +} diff --git a/internal/cli/root.go b/internal/cli/root.go index 33819030..ab8e7ea7 100644 --- a/internal/cli/root.go +++ b/internal/cli/root.go @@ -175,6 +175,7 @@ func init() { rootCmd.AddCommand(validateCmd) rootCmd.AddCommand(listCasesCmd) rootCmd.AddCommand(reportCmd) + rootCmd.AddCommand(compareCmd) rootCmd.AddCommand(debugCmd) rootCmd.AddCommand(importCmd) } diff --git a/internal/compare/compare.go b/internal/compare/compare.go new file mode 100644 index 00000000..2192c802 --- /dev/null +++ b/internal/compare/compare.go @@ -0,0 +1,275 @@ +// Package compare compares two offline skill-up report results. +package compare + +import ( + "encoding/json" + "slices" + "time" + + "github.com/alibaba/skill-up/internal/judge" + "github.com/alibaba/skill-up/internal/report" +) + +// Options controls compare output and gate evaluation. +type Options struct { + FailOnRegression bool + MaxRegressions *int + MaxTokenIncreasePercent *float64 + OldMissingMetrics []string + NewMissingMetrics []string +} + +// Result is the stable comparison result used by text and JSON output. +type Result struct { + Baseline Source `json:"baseline"` + Candidate Source `json:"candidate"` + Coverage Coverage `json:"coverage"` + Metadata MetadataDiff `json:"metadata"` + Run RunComparison `json:"run"` + Cases CaseTransitions `json:"cases"` + Gates GateResult `json:"gates"` +} + +// Source identifies an original result artifact independently of its configuration. +type Source struct { + Label string `json:"label"` + Path string `json:"path"` +} + +// Coverage describes the shared case set, without treating added or removed cases as pairs. +type Coverage struct { + Shared int `json:"shared"` + Added int `json:"added"` + Removed int `json:"removed"` +} + +// MetadataDiff records old and new run metadata. +type MetadataDiff struct { + RequestedConfiguration FieldDiff[string] `json:"requested_configuration"` + AppliedConfiguration FieldDiff[string] `json:"applied_configuration"` + ObservedConfiguration FieldDiff[string] `json:"observed_configuration"` + SkillName FieldDiff[string] `json:"skill_name"` + SchemaVersion FieldDiff[string] `json:"schema_version"` + EngineName FieldDiff[string] `json:"engine_name"` + ModelName FieldDiff[string] `json:"model_name"` + StartTime FieldDiff[time.Time] `json:"start_time"` + EndTime FieldDiff[time.Time] `json:"end_time"` +} + +// FieldDiff stores old/new values and whether they differ. +type FieldDiff[T comparable] struct { + Old T `json:"old"` + New T `json:"new"` + Changed bool `json:"changed"` +} + +// RunComparison stores old/new/delta aggregate metrics. +type RunComparison struct { + Old RunMetrics `json:"old"` + New RunMetrics `json:"new"` + Delta RunMetrics `json:"delta"` +} + +// RunMetrics stores aggregate metrics for one result set or a delta. +type RunMetrics struct { + Missing []string `json:"-"` + CaseCount int `json:"case_count"` + PassCount int `json:"pass_count"` + PassRate float64 `json:"pass_rate"` + TotalTokens int `json:"total_tokens"` + InputTokens int `json:"input_tokens"` + OutputTokens int `json:"output_tokens"` + DurationMs int64 `json:"duration_ms"` +} + +// MarshalJSON preserves unknown measurements as null rather than measured zero. +func (m RunMetrics) MarshalJSON() ([]byte, error) { + type metrics RunMetrics + data, err := json.Marshal(metrics(m)) + if err != nil { + return nil, err + } + var fields map[string]json.RawMessage + if err := json.Unmarshal(data, &fields); err != nil { + return nil, err + } + for _, name := range m.Missing { + fields[name] = json.RawMessage("null") + } + return json.Marshal(fields) +} + +// CaseTransitions groups case status transitions. +type CaseTransitions struct { + Fixed []CaseTransition `json:"fixed"` + Regressed []CaseTransition `json:"regressed"` + Changed []CaseTransition `json:"changed"` + Unchanged []CaseTransition `json:"unchanged"` + Added []CaseTransition `json:"added"` + Removed []CaseTransition `json:"removed"` +} + +// CaseTransition describes one case movement between old and new results. +type CaseTransition struct { + PromptChanged *bool `json:"prompt_changed"` + OldConfiguration string `json:"old_configuration,omitempty"` + NewConfiguration string `json:"new_configuration,omitempty"` + CaseID string `json:"case_id"` + OldTitle string `json:"old_title,omitempty"` + NewTitle string `json:"new_title,omitempty"` + OldStatus judge.Status `json:"old_status,omitempty"` + NewStatus judge.Status `json:"new_status,omitempty"` +} + +// GateResult records whether CI gates passed and why they failed. +type GateResult struct { + Passed bool `json:"passed"` + Failures []string `json:"failures"` +} + +// Compare compares two offline report inputs. +func Compare(oldInput, newInput report.Input, options Options) Result { + oldMetrics := collectRunMetrics(oldInput) + newMetrics := collectRunMetrics(newInput) + oldMetrics.Missing = options.OldMissingMetrics + newMetrics.Missing = options.NewMissingMetrics + result := Result{ + Baseline: Source{Label: "Baseline"}, + Candidate: Source{Label: "Candidate"}, + Metadata: metadataDiff(oldInput, newInput), + Run: RunComparison{ + Old: oldMetrics, + New: newMetrics, + Delta: diffRunMetrics(oldMetrics, newMetrics), + }, + Cases: compareCases(oldInput.PrimaryCaseResults(), newInput.PrimaryCaseResults()), + } + result.Coverage = Coverage{ + Shared: oldMetrics.CaseCount - len(result.Cases.Removed), + Added: len(result.Cases.Added), Removed: len(result.Cases.Removed), + } + result.Gates = EvaluateGates(result, options) + return result +} + +func collectRunMetrics(input report.Input) RunMetrics { + primary := input.PrimaryCaseResults() + metrics := RunMetrics{ + CaseCount: len(primary), + TotalTokens: input.TotalTokens, + DurationMs: input.TotalDuration().Milliseconds(), + } + for _, cr := range primary { + if cr.Status == judge.StatusPass { + metrics.PassCount++ + } + metrics.InputTokens += cr.InputTokens + metrics.OutputTokens += cr.OutputTokens + } + if metrics.CaseCount > 0 { + metrics.PassRate = float64(metrics.PassCount) / float64(metrics.CaseCount) + } + return metrics +} + +func diffRunMetrics(oldMetrics, newMetrics RunMetrics) RunMetrics { + return RunMetrics{ + Missing: slices.Concat(oldMetrics.Missing, newMetrics.Missing), + CaseCount: newMetrics.CaseCount - oldMetrics.CaseCount, + PassCount: newMetrics.PassCount - oldMetrics.PassCount, + PassRate: newMetrics.PassRate - oldMetrics.PassRate, + TotalTokens: newMetrics.TotalTokens - oldMetrics.TotalTokens, + InputTokens: newMetrics.InputTokens - oldMetrics.InputTokens, + OutputTokens: newMetrics.OutputTokens - oldMetrics.OutputTokens, + DurationMs: newMetrics.DurationMs - oldMetrics.DurationMs, + } +} + +func metadataDiff(oldInput, newInput report.Input) MetadataDiff { + return MetadataDiff{ + RequestedConfiguration: fieldDiff(configurationText(oldInput.RequestedConfiguration), configurationText(newInput.RequestedConfiguration)), + AppliedConfiguration: fieldDiff(configurationText(oldInput.AppliedConfiguration), configurationText(newInput.AppliedConfiguration)), + ObservedConfiguration: fieldDiff(configurationText(oldInput.ObservedConfiguration), configurationText(newInput.ObservedConfiguration)), + SkillName: fieldDiff(oldInput.SkillName, newInput.SkillName), + SchemaVersion: fieldDiff(oldInput.SchemaVersion, newInput.SchemaVersion), + EngineName: fieldDiff(oldInput.EngineName, newInput.EngineName), + ModelName: fieldDiff(oldInput.ModelName, newInput.ModelName), + StartTime: fieldDiff(oldInput.StartTime, newInput.StartTime), + EndTime: fieldDiff(oldInput.EndTime, newInput.EndTime), + } +} + +func configurationText(configuration *report.AgentConfiguration) string { + if configuration == nil { + return "" + } + data, err := json.Marshal(configuration) + if err != nil { + return "" + } + return string(data) +} + +func fieldDiff[T comparable](oldValue, newValue T) FieldDiff[T] { + return FieldDiff[T]{Old: oldValue, New: newValue, Changed: oldValue != newValue} +} + +func compareCases(oldCases, newCases []report.CaseResult) CaseTransitions { + newByID := make(map[string]report.CaseResult, len(newCases)) + for _, newCase := range newCases { + newByID[newCase.CaseID] = newCase + } + + transitions := CaseTransitions{ + Fixed: make([]CaseTransition, 0), + Regressed: make([]CaseTransition, 0), + Changed: make([]CaseTransition, 0), + Unchanged: make([]CaseTransition, 0), + Added: make([]CaseTransition, 0), + Removed: make([]CaseTransition, 0), + } + for _, oldCase := range oldCases { + newCase, exists := newByID[oldCase.CaseID] + if !exists { + transitions.Removed = append(transitions.Removed, CaseTransition{ + OldConfiguration: oldCase.Configuration, + CaseID: oldCase.CaseID, OldTitle: oldCase.Title, OldStatus: oldCase.Status, + }) + continue + } + + transition := CaseTransition{ + OldConfiguration: oldCase.Configuration, NewConfiguration: newCase.Configuration, + CaseID: oldCase.CaseID, OldTitle: oldCase.Title, NewTitle: newCase.Title, + OldStatus: oldCase.Status, NewStatus: newCase.Status, + } + if oldCase.Prompt != "" && newCase.Prompt != "" { + changed := oldCase.Prompt != newCase.Prompt + transition.PromptChanged = &changed + } + switch { + case oldCase.Status != judge.StatusPass && newCase.Status == judge.StatusPass: + transitions.Fixed = append(transitions.Fixed, transition) + case oldCase.Status == judge.StatusPass && newCase.Status != judge.StatusPass: + transitions.Regressed = append(transitions.Regressed, transition) + case oldCase.Status == newCase.Status: + transitions.Unchanged = append(transitions.Unchanged, transition) + default: + transitions.Changed = append(transitions.Changed, transition) + } + } + + oldByID := make(map[string]struct{}, len(oldCases)) + for _, oldCase := range oldCases { + oldByID[oldCase.CaseID] = struct{}{} + } + for _, newCase := range newCases { + if _, exists := oldByID[newCase.CaseID]; !exists { + transitions.Added = append(transitions.Added, CaseTransition{ + NewConfiguration: newCase.Configuration, + CaseID: newCase.CaseID, NewTitle: newCase.Title, NewStatus: newCase.Status, + }) + } + } + return transitions +} diff --git a/internal/compare/compare_test.go b/internal/compare/compare_test.go new file mode 100644 index 00000000..aeb3bab1 --- /dev/null +++ b/internal/compare/compare_test.go @@ -0,0 +1,282 @@ +package compare + +import ( + "encoding/json" + "math" + "reflect" + "strings" + "testing" + "time" + + "github.com/alibaba/skill-up/internal/judge" + "github.com/alibaba/skill-up/internal/report" +) + +func compareFixture() (oldInput, newInput report.Input) { + start := time.Date(2026, 8, 12, 9, 0, 0, 0, time.UTC) + oldInput = report.Input{ + SkillName: "skill-a", + SchemaVersion: "v1alpha1", + EngineName: "codex", + ModelName: "gpt-5", + StartTime: start, + EndTime: start.Add(2 * time.Minute), + TotalTokens: 100, + CaseResults: []report.CaseResult{ + {CaseID: "case-1", Title: "Case One baseline", Status: judge.StatusFail, Configuration: "without_skill", InputTokens: 10, OutputTokens: 5}, + {CaseID: "case-1", Title: "Case One", Status: judge.StatusPass, Configuration: "with_skill", InputTokens: 20, OutputTokens: 10}, + {CaseID: "case-2", Title: "Case Two", Status: judge.StatusFail, InputTokens: 30, OutputTokens: 15}, + }, + } + newInput = report.Input{ + SkillName: "skill-a", + SchemaVersion: "v1alpha1", + EngineName: "codex", + ModelName: "gpt-5.1", + StartTime: start.Add(24 * time.Hour), + EndTime: start.Add(24*time.Hour + 3*time.Minute), + TotalTokens: 140, + CaseResults: []report.CaseResult{ + {CaseID: "case-1", Title: "Case One", Status: judge.StatusPass, Configuration: "with_skill", InputTokens: 25, OutputTokens: 12}, + {CaseID: "case-2", Title: "Case Two", Status: judge.StatusPass, InputTokens: 35, OutputTokens: 18}, + }, + } + return oldInput, newInput +} + +func TestCompareRunMetricsUsePrimaryCaseResults(t *testing.T) { + t.Parallel() + oldInput, newInput := compareFixture() + + result := Compare(oldInput, newInput, Options{}) + + if result.Run.Old.CaseCount != 2 { + t.Fatalf("old case count should use primary results, got %d", result.Run.Old.CaseCount) + } + if result.Run.Old.PassCount != 1 { + t.Fatalf("old pass count should ignore without_skill baseline failure, got %d", result.Run.Old.PassCount) + } + if math.Abs(result.Run.Old.PassRate-0.5) > 0.001 { + t.Fatalf("old pass rate: want 0.5, got %f", result.Run.Old.PassRate) + } + if result.Run.Old.InputTokens != 50 || result.Run.Old.OutputTokens != 25 { + t.Fatalf("old tokens should sum primary results, got input=%d output=%d", result.Run.Old.InputTokens, result.Run.Old.OutputTokens) + } + if result.Run.Old.TotalTokens != 100 || result.Run.New.TotalTokens != 140 || result.Run.Delta.TotalTokens != 40 { + t.Fatalf("total token delta mismatch: old=%d new=%d delta=%d", result.Run.Old.TotalTokens, result.Run.New.TotalTokens, result.Run.Delta.TotalTokens) + } + if result.Run.Old.DurationMs != 120000 || result.Run.New.DurationMs != 180000 || result.Run.Delta.DurationMs != 60000 { + t.Fatalf("duration delta mismatch: old=%d new=%d delta=%d", result.Run.Old.DurationMs, result.Run.New.DurationMs, result.Run.Delta.DurationMs) + } +} + +func TestCompareClassifiesCaseTransitionsInDeterministicOrder(t *testing.T) { + t.Parallel() + oldInput := report.Input{CaseResults: []report.CaseResult{ + {CaseID: "fixed", Title: "Fixed old", Status: judge.StatusFail}, + {CaseID: "regressed", Title: "Regressed old", Status: judge.StatusPass}, + {CaseID: "status-changed", Title: "Status changed old", Status: judge.StatusError}, + {CaseID: "unchanged", Title: "Unchanged old", Status: judge.StatusError}, + {CaseID: "removed", Title: "Removed", Status: judge.StatusFail}, + }} + newInput := report.Input{CaseResults: []report.CaseResult{ + {CaseID: "regressed", Title: "Regressed new", Status: judge.StatusFail}, + {CaseID: "fixed", Title: "Fixed new", Status: judge.StatusPass}, + {CaseID: "status-changed", Title: "Status changed new", Status: judge.StatusSkip}, + {CaseID: "unchanged", Title: "Unchanged new", Status: judge.StatusError}, + {CaseID: "added-first", Title: "Added first", Status: judge.StatusPass}, + {CaseID: "added-second", Title: "Added second", Status: judge.StatusFail}, + }} + + got := Compare(oldInput, newInput, Options{}).Cases + want := CaseTransitions{ + Fixed: []CaseTransition{{CaseID: "fixed", OldTitle: "Fixed old", NewTitle: "Fixed new", OldStatus: judge.StatusFail, NewStatus: judge.StatusPass}}, + Regressed: []CaseTransition{{CaseID: "regressed", OldTitle: "Regressed old", NewTitle: "Regressed new", OldStatus: judge.StatusPass, NewStatus: judge.StatusFail}}, + Changed: []CaseTransition{{CaseID: "status-changed", OldTitle: "Status changed old", NewTitle: "Status changed new", OldStatus: judge.StatusError, NewStatus: judge.StatusSkip}}, + Unchanged: []CaseTransition{{CaseID: "unchanged", OldTitle: "Unchanged old", NewTitle: "Unchanged new", OldStatus: judge.StatusError, NewStatus: judge.StatusError}}, + Added: []CaseTransition{ + {CaseID: "added-first", NewTitle: "Added first", NewStatus: judge.StatusPass}, + {CaseID: "added-second", NewTitle: "Added second", NewStatus: judge.StatusFail}, + }, + Removed: []CaseTransition{{CaseID: "removed", OldTitle: "Removed", OldStatus: judge.StatusFail}}, + } + if !reflect.DeepEqual(got, want) { + t.Fatalf("case transitions mismatch:\nwant: %#v\n got: %#v", want, got) + } +} + +func TestCompareIncludesMetadataDiff(t *testing.T) { + t.Parallel() + oldInput, newInput := compareFixture() + + got := Compare(oldInput, newInput, Options{}).Metadata + if got.SkillName.Changed { + t.Fatal("skill name should be unchanged") + } + if !got.ModelName.Changed || !got.StartTime.Changed || !got.EndTime.Changed { + t.Fatalf("changed metadata mismatch: model=%t start=%t end=%t", got.ModelName.Changed, got.StartTime.Changed, got.EndTime.Changed) + } +} + +func TestCompareFailsRegressionAndTokenIncreaseGates(t *testing.T) { + t.Parallel() + limit := 20.0 + oldInput := report.Input{TotalTokens: 100, CaseResults: []report.CaseResult{{CaseID: "case-1", Status: judge.StatusPass}}} + newInput := report.Input{TotalTokens: 140, CaseResults: []report.CaseResult{{CaseID: "case-1", Status: judge.StatusFail}}} + + got := Compare(oldInput, newInput, Options{FailOnRegression: true, MaxTokenIncreasePercent: &limit}).Gates + if got.Passed || len(got.Failures) != 2 { + t.Fatalf("expected both gates to fail, got %#v", got) + } +} + +func TestCompareDoesNotFailRegressionGateForChangedNonPassStatus(t *testing.T) { + t.Parallel() + oldInput := report.Input{CaseResults: []report.CaseResult{{CaseID: "case-1", Status: judge.StatusError}}} + newInput := report.Input{CaseResults: []report.CaseResult{{CaseID: "case-1", Status: judge.StatusSkip}}} + + result := Compare(oldInput, newInput, Options{FailOnRegression: true}) + if !result.Gates.Passed || len(result.Gates.Failures) != 0 { + t.Fatalf("changed non-PASS status should not fail regression gate, got %#v", result.Gates) + } + if len(result.Cases.Changed) != 1 { + t.Fatalf("changed transitions = %#v, want one ERROR -> SKIP transition", result.Cases.Changed) + } +} + +func TestCompareFailsRegressionGateAboveConfiguredMaximum(t *testing.T) { + t.Parallel() + maxRegressions := 1 + oldInput := report.Input{CaseResults: []report.CaseResult{ + {CaseID: "case-1", Status: judge.StatusPass}, + {CaseID: "case-2", Status: judge.StatusPass}, + }} + newInput := report.Input{CaseResults: []report.CaseResult{ + {CaseID: "case-1", Status: judge.StatusFail}, + {CaseID: "case-2", Status: judge.StatusError}, + }} + + got := Compare(oldInput, newInput, Options{MaxRegressions: &maxRegressions}).Gates + if got.Passed || len(got.Failures) != 1 || !strings.Contains(got.Failures[0], "2 regressions exceeds maximum 1") { + t.Fatalf("expected regression maximum gate failure, got %#v", got) + } +} + +func TestCompareFailsTokenGateWhenOldTotalTokensAreZero(t *testing.T) { + t.Parallel() + limit := 20.0 + oldInput := report.Input{TotalTokens: 0} + newInput := report.Input{TotalTokens: 1, CaseResults: []report.CaseResult{{CaseID: "case-1"}}} + + got := Compare(oldInput, newInput, Options{MaxTokenIncreasePercent: &limit}).Gates + if got.Passed || len(got.Failures) != 1 || !strings.Contains(got.Failures[0], "old total tokens is 0") { + t.Fatalf("expected zero-token gate failure, got %#v", got) + } +} + +func TestCompareTokenGateUsesTotalTokens(t *testing.T) { + t.Parallel() + limit := 10.0 + oldInput := report.Input{TotalTokens: 200, CaseResults: []report.CaseResult{ + {CaseID: "case-1"}, + {CaseID: "case-2"}, + }} + newInput := report.Input{TotalTokens: 150, CaseResults: []report.CaseResult{{CaseID: "case-1"}}} + + got := Compare(oldInput, newInput, Options{MaxTokenIncreasePercent: &limit}).Gates + if !got.Passed || len(got.Failures) != 0 { + t.Fatalf("expected total token gate to pass, got %#v", got) + } +} + +func TestRenderTextIncludesRunMetadataAndCaseTransitionSections(t *testing.T) { + t.Parallel() + oldInput, newInput := compareFixture() + result := Compare(oldInput, newInput, Options{FailOnRegression: true}) + result.Cases = CaseTransitions{ + Fixed: []CaseTransition{{CaseID: "fixed", OldStatus: judge.StatusFail, NewStatus: judge.StatusPass}}, + Regressed: []CaseTransition{{CaseID: "regressed", OldStatus: judge.StatusPass, NewStatus: judge.StatusFail}}, + Changed: []CaseTransition{{CaseID: "changed", OldStatus: judge.StatusError, NewStatus: judge.StatusSkip}}, + Unchanged: []CaseTransition{{CaseID: "unchanged", OldStatus: judge.StatusError, NewStatus: judge.StatusError}}, + Added: []CaseTransition{{CaseID: "added", NewStatus: judge.StatusPass}}, + Removed: []CaseTransition{{CaseID: "removed", OldStatus: judge.StatusFail}}, + } + result.Gates = GateResult{Passed: false, Failures: []string{"1 case(s) regressed"}} + + got := RenderText(result) + for _, want := range []string{ + "Run summary", + "Pass rate: 50.00% -> 100.00% (+50.00 pp)", + "Total tokens (all configurations): 100 -> 140 (+40)", + "Metadata differences", + "Model: gpt-5 -> gpt-5.1", + "Case transitions", + "fixed (1): fixed (FAIL -> PASS)", + "regressed (1): regressed (PASS -> FAIL)", + "changed (1): changed (ERROR -> SKIP)", + "unchanged (1): unchanged (ERROR -> ERROR)", + "added (1): added (-> PASS)", + "removed (1): removed (FAIL ->)", + "Gates: failed", + "- 1 case(s) regressed", + } { + if !strings.Contains(got, want) { + t.Errorf("RenderText() missing %q:\n%s", want, got) + } + } +} + +func TestResultJSONUsesStableFields(t *testing.T) { + t.Parallel() + oldInput, newInput := compareFixture() + data, err := json.Marshal(Compare(oldInput, newInput, Options{})) + if err != nil { + t.Fatalf("marshal result: %v", err) + } + + var got map[string]any + if err := json.Unmarshal(data, &got); err != nil { + t.Fatalf("unmarshal result: %v", err) + } + for _, key := range []string{"metadata", "run", "cases", "gates"} { + if _, ok := got[key]; !ok { + t.Errorf("JSON result missing top-level %q: %s", key, data) + } + } + for _, key := range []string{"old", "new", "delta"} { + if _, ok := got["run"].(map[string]any)[key]; !ok { + t.Errorf("JSON run missing %q: %s", key, data) + } + } + for _, key := range []string{"fixed", "regressed", "changed", "unchanged", "added", "removed"} { + if _, ok := got["cases"].(map[string]any)[key]; !ok { + t.Errorf("JSON cases missing %q: %s", key, data) + } + } + for _, key := range []string{"passed", "failures"} { + if _, ok := got["gates"].(map[string]any)[key]; !ok { + t.Errorf("JSON gates missing %q: %s", key, data) + } + } +} + +func TestResultJSONUsesEmptyArraysForEmptyCaseTransitions(t *testing.T) { + t.Parallel() + input := report.Input{CaseResults: []report.CaseResult{{CaseID: "case-1", Status: judge.StatusPass}}} + data, err := json.Marshal(Compare(input, input, Options{})) + if err != nil { + t.Fatalf("marshal result: %v", err) + } + + var got struct { + Cases map[string]json.RawMessage `json:"cases"` + } + if err := json.Unmarshal(data, &got); err != nil { + t.Fatalf("unmarshal result: %v", err) + } + for _, name := range []string{"fixed", "regressed", "changed", "added", "removed"} { + if string(got.Cases[name]) != "[]" { + t.Errorf("%s transition group = %s, want []", name, got.Cases[name]) + } + } +} diff --git a/internal/compare/gates.go b/internal/compare/gates.go new file mode 100644 index 00000000..c6dfeea9 --- /dev/null +++ b/internal/compare/gates.go @@ -0,0 +1,34 @@ +package compare + +import ( + "fmt" + "slices" +) + +// EvaluateGates evaluates the configured comparison gates. +func EvaluateGates(result Result, options Options) GateResult { + failures := make([]string, 0, 2) + regressions := len(result.Cases.Regressed) + if options.FailOnRegression && regressions > 0 { + failures = append(failures, fmt.Sprintf("%d case(s) regressed", regressions)) + } + if options.MaxRegressions != nil && regressions > *options.MaxRegressions { + failures = append(failures, fmt.Sprintf("%d regressions exceeds maximum %d", regressions, *options.MaxRegressions)) + } + if options.MaxTokenIncreasePercent != nil { + oldTokens := result.Run.Old.TotalTokens + newTokens := result.Run.New.TotalTokens + switch { + case slices.Contains(result.Run.Old.Missing, "total_tokens") || slices.Contains(result.Run.New.Missing, "total_tokens"): + failures = append(failures, "token increase cannot be evaluated: total tokens is unknown") + case oldTokens == 0 && newTokens > 0: + failures = append(failures, "token increase exceeds limit: old total tokens is 0") + case oldTokens > 0: + increasePercent := float64(newTokens-oldTokens) / float64(oldTokens) * 100 + if increasePercent > *options.MaxTokenIncreasePercent { + failures = append(failures, fmt.Sprintf("token increase %.2f%% exceeds limit %.2f%%", increasePercent, *options.MaxTokenIncreasePercent)) + } + } + } + return GateResult{Passed: len(failures) == 0, Failures: failures} +} diff --git a/internal/compare/input.go b/internal/compare/input.go new file mode 100644 index 00000000..a4afaa33 --- /dev/null +++ b/internal/compare/input.go @@ -0,0 +1,47 @@ +package compare + +import ( + "encoding/json" + + "github.com/alibaba/skill-up/internal/report" +) + +// MissingMetrics identifies absent measurements in an already validated JSON input. +// Unmarshalling into report.Input alone would turn absent measurements into zero. +func MissingMetrics(data []byte, input report.Input) []string { + var raw struct { + TotalTokens *int `json:"total_tokens"` + Cases []struct { + CaseID string `json:"case_id"` + Configuration string `json:"configuration"` + Input *int `json:"input_tokens"` + Output *int `json:"output_tokens"` + } `json:"case_results"` + } + if err := json.Unmarshal(data, &raw); err != nil { + return []string{"total_tokens", "input_tokens", "output_tokens", "duration_ms"} + } + var missing []string + if raw.TotalTokens == nil { + missing = append(missing, "total_tokens") + } + selected := make(map[string]string) + for _, cr := range input.PrimaryCaseResults() { + selected[cr.CaseID] = cr.Configuration + } + inputKnown, outputKnown := true, true + for _, cr := range raw.Cases { + if selected[cr.CaseID] != cr.Configuration { + continue + } + inputKnown = inputKnown && cr.Input != nil + outputKnown = outputKnown && cr.Output != nil + } + if !inputKnown { + missing = append(missing, "input_tokens") + } + if !outputKnown { + missing = append(missing, "output_tokens") + } + return missing +} diff --git a/internal/compare/render.go b/internal/compare/render.go new file mode 100644 index 00000000..9831c348 --- /dev/null +++ b/internal/compare/render.go @@ -0,0 +1,204 @@ +package compare + +import ( + "bytes" + _ "embed" + "encoding/json" + "fmt" + "html" + "html/template" + "slices" + "strconv" + "strings" + "time" +) + +const unknownValue = "unknown" + +//go:embed templates/compare.html +var htmlSource string + +var comparisonTemplate = template.Must(template.New("comparison").Funcs(template.FuncMap{"promptChange": promptChange}).Parse(htmlSource)) + +type ( + comparisonRow struct{ Name, Old, New, Delta string } + caseGroup struct { + Name string + Cases []CaseTransition + } +) + +type comparisonView struct { + Result Result + Metrics []comparisonRow + Metadata []comparisonRow + Groups []caseGroup +} + +// Render emits the same comparison in text, JSON, HTML, or Markdown form. +func Render(result Result, format string) ([]byte, error) { + switch format { + case "text": + return []byte(RenderText(result)), nil + case "json": + data, err := json.MarshalIndent(result, "", " ") + if err != nil { + return nil, fmt.Errorf("encode comparison: %w", err) + } + return append(data, '\n'), nil + case "markdown": + return []byte(RenderMarkdown(result)), nil + case "html": + var b bytes.Buffer + if err := comparisonTemplate.Execute(&b, newComparisonView(result)); err != nil { + return nil, fmt.Errorf("render HTML comparison: %w", err) + } + return b.Bytes(), nil + default: + return nil, fmt.Errorf("unsupported comparison format %q", format) + } +} + +func newComparisonView(result Result) comparisonView { + old, candidate, delta := result.Run.Old, result.Run.New, result.Run.Delta + view := comparisonView{ + Result: result, + Metrics: []comparisonRow{ + {"Pass rate", fmt.Sprintf("%.2f%%", old.PassRate*100), fmt.Sprintf("%.2f%%", candidate.PassRate*100), fmt.Sprintf("%+.2f pp", delta.PassRate*100)}, + {"Cases", strconv.Itoa(old.CaseCount), strconv.Itoa(candidate.CaseCount), fmt.Sprintf("%+d", delta.CaseCount)}, + {"Passed", strconv.Itoa(old.PassCount), strconv.Itoa(candidate.PassCount), fmt.Sprintf("%+d", delta.PassCount)}, + metricRow("Total tokens (all configurations)", "total_tokens", int64(old.TotalTokens), int64(candidate.TotalTokens), result.Run), + metricRow("Input tokens (primary cases)", "input_tokens", int64(old.InputTokens), int64(candidate.InputTokens), result.Run), + metricRow("Output tokens (primary cases)", "output_tokens", int64(old.OutputTokens), int64(candidate.OutputTokens), result.Run), + metricRow("Run duration (ms)", "duration_ms", old.DurationMs, candidate.DurationMs, result.Run), + }, + Groups: []caseGroup{ + {"Regressed", result.Cases.Regressed}, + {"Fixed", result.Cases.Fixed}, + {"Changed", result.Cases.Changed}, + {"Added", result.Cases.Added}, + {"Removed", result.Cases.Removed}, + {"Unchanged", result.Cases.Unchanged}, + }, + } + m := result.Metadata + view.Metadata = []comparisonRow{ + metadataRow("Skill", m.SkillName), metadataRow("Schema version", m.SchemaVersion), + metadataRow("Engine", m.EngineName), metadataRow("Model", m.ModelName), + metadataRow("Start time", fieldDiff(formatTime(m.StartTime.Old), formatTime(m.StartTime.New))), + metadataRow("End time", fieldDiff(formatTime(m.EndTime.Old), formatTime(m.EndTime.New))), + metadataRow("Requested configuration", m.RequestedConfiguration), + metadataRow("Applied configuration", m.AppliedConfiguration), + metadataRow("Observed configuration", m.ObservedConfiguration), + } + return view +} + +func promptChange(changed *bool) string { + if changed == nil { + return unknownValue + } + if *changed { + return "changed" + } + return "unchanged" +} + +func formatTime(t time.Time) string { + if t.IsZero() { + return unknownValue + } + return t.Format(time.RFC3339Nano) +} + +func metricRow(label, key string, old, candidate int64, run RunComparison) comparisonRow { + row := comparisonRow{label, strconv.FormatInt(old, 10), strconv.FormatInt(candidate, 10), fmt.Sprintf("%+d", candidate-old)} + if slices.Contains(run.Old.Missing, key) { + row.Old, row.Delta = unknownValue, unknownValue + } + if slices.Contains(run.New.Missing, key) { + row.New, row.Delta = unknownValue, unknownValue + } + return row +} + +func metadataRow(label string, diff FieldDiff[string]) comparisonRow { + row := comparisonRow{Name: label, Old: knownText(diff.Old), New: knownText(diff.New), Delta: "unchanged"} + if diff.Changed { + row.Delta = "changed" + } + if diff.Old == "" || diff.New == "" { + row.Delta = unknownValue + } + return row +} + +func knownText(value string) string { + if value == "" { + return unknownValue + } + return value +} + +// RenderMarkdown creates a shareable comparison for PR descriptions and CI artifacts. +func RenderMarkdown(result Result) string { + view := newComparisonView(result) + var b strings.Builder + b.WriteString("# Evaluation comparison\n\n") + fmt.Fprintf(&b, "**%s** → **%s**\n\n", markdownCell(result.Baseline.Label), markdownCell(result.Candidate.Label)) + fmt.Fprintf(&b, "Sources: %s → %s\n\n", markdownCell(result.Baseline.Path), markdownCell(result.Candidate.Path)) + fmt.Fprintf(&b, "Shared cases: **%d**; added: **%d**; removed: **%d**.\n\n", result.Coverage.Shared, result.Coverage.Added, result.Coverage.Removed) + fmt.Fprintf(&b, "Comparison coverage: %d / %d baseline cases; %d / %d candidate cases.\n\n", result.Coverage.Shared, result.Run.Old.CaseCount, result.Coverage.Shared, result.Run.New.CaseCount) + if result.Coverage.Added+result.Coverage.Removed > 0 { + b.WriteString("> Different case sets: run-level deltas include unpaired cases. Only shared case IDs are classified as fixed or regressed.\n\n") + } + writeMarkdownRows(&b, "Run summary", view.Metrics, result) + writeMarkdownRows(&b, "Metadata differences", view.Metadata, result) + b.WriteString("Missing provenance is unknown. Matching case IDs do not prove identical prompts, suites, Skill revisions, or judge/rubric definitions. Scores are not regraded.\n\n") + b.WriteString("## Case transitions\n\n") + for _, group := range view.Groups { + writeMarkdownGroup(&b, group) + } + b.WriteString("## Gates\n\n") + if result.Gates.Passed { + b.WriteString("Passed (configured checks only).\n") + } else { + b.WriteString("Failed.\n\n") + for _, failure := range result.Gates.Failures { + fmt.Fprintf(&b, "- %s\n", markdownCell(failure)) + } + } + return b.String() +} + +func writeMarkdownRows(b *strings.Builder, heading string, rows []comparisonRow, result Result) { + fmt.Fprintf(b, "## %s\n\n| Metric | %s | %s | Change |\n|---|---|---|---|\n", heading, markdownCell(result.Baseline.Label), markdownCell(result.Candidate.Label)) + for _, row := range rows { + fmt.Fprintf(b, "| %s | %s | %s | %s |\n", markdownCell(row.Name), markdownCell(row.Old), markdownCell(row.New), markdownCell(row.Delta)) + } + b.WriteByte('\n') +} + +func writeMarkdownGroup(b *strings.Builder, group caseGroup) { + fmt.Fprintf(b, "### %s (%d)\n\n", group.Name, len(group.Cases)) + if len(group.Cases) == 0 { + b.WriteString("None.\n\n") + return + } + b.WriteString("| Case ID | Baseline title | Candidate title | Baseline status | Candidate status | Baseline configuration | Candidate configuration | Prompt |\n|---|---|---|---|---|---|---|---|\n") + for _, cr := range group.Cases { + fmt.Fprintf(b, "| %s | %s | %s | %s | %s | %s | %s | %s |\n", markdownCell(cr.CaseID), markdownCell(cr.OldTitle), markdownCell(cr.NewTitle), markdownCell(string(cr.OldStatus)), markdownCell(string(cr.NewStatus)), markdownCell(cr.OldConfiguration), markdownCell(cr.NewConfiguration), promptChange(cr.PromptChanged)) + } + b.WriteByte('\n') +} + +func markdownCell(value string) string { + // Encode all Markdown control characters, pipes, and line breaks so source + // titles cannot add markup, links, or rows to a shared report. + value = html.EscapeString(value) + value = strings.NewReplacer("\r", " ", "\n", " ", "\\", "\", "|", "|", "`", "`", "*", "*", "_", "_", "[", "[", "]", "]").Replace(value) + if strings.TrimSpace(value) == "" { + return "—" + } + return value +} diff --git a/internal/compare/render_test.go b/internal/compare/render_test.go new file mode 100644 index 00000000..ab1a13fa --- /dev/null +++ b/internal/compare/render_test.go @@ -0,0 +1,139 @@ +package compare + +import ( + "encoding/json" + "strings" + "testing" + + "github.com/alibaba/skill-up/internal/judge" + "github.com/alibaba/skill-up/internal/report" +) + +func TestShareableReportsIncludeTransitionsAndCoverage(t *testing.T) { + t.Parallel() + oldInput, newInput := compareFixture() + oldInput.CaseResults = append(oldInput.CaseResults, report.CaseResult{CaseID: "removed", Status: judge.StatusPass}) + newInput.CaseResults[0].Status = judge.StatusFail + newInput.CaseResults = append(newInput.CaseResults, report.CaseResult{CaseID: "added", Status: judge.StatusPass}) + result := Compare(oldInput, newInput, Options{FailOnRegression: true}) + result.Baseline = Source{Label: "Previous Skill", Path: "old.json"} + result.Candidate = Source{Label: "Revised Skill", Path: "new.json"} + for _, format := range []string{"html", "markdown"} { + t.Run(format, func(t *testing.T) { + t.Parallel() + data, err := Render(result, format) + if err != nil { + t.Fatal(err) + } + for _, want := range []string{"Previous Skill", "Revised Skill", "old.json", "new.json", "Run summary", "Metadata differences", "Regressed (1)", "Fixed (1)", "Added (1)", "Removed (1)", "Different case sets", "1 case(s) regressed", "unknown"} { + if !strings.Contains(string(data), want) { + t.Errorf("%s missing %q", format, want) + } + } + }) + } + if result.Coverage != (Coverage{Shared: 2, Added: 1, Removed: 1}) { + t.Fatalf("coverage = %+v", result.Coverage) + } +} + +func TestReportsEscapeUntrustedSourceText(t *testing.T) { + t.Parallel() + input := report.Input{CaseResults: []report.CaseResult{{CaseID: "case", Title: "\n| [link](https://example.com) `code` \\|", Status: judge.StatusPass}}} + result := Compare(input, input, Options{}) + result.Baseline.Label = "" + for _, format := range []string{"html", "markdown"} { + data, err := Render(result, format) + if err != nil { + t.Fatal(err) + } + text := string(data) + for _, unsafe := range []string{" + diff --git a/internal/compare/text.go b/internal/compare/text.go new file mode 100644 index 00000000..fe2d6c80 --- /dev/null +++ b/internal/compare/text.go @@ -0,0 +1,64 @@ +package compare + +import ( + "fmt" + "strings" +) + +// RenderText renders a human-readable comparison result. +func RenderText(result Result) string { + var b strings.Builder + view := newComparisonView(result) + fmt.Fprintf(&b, "%s -> %s\n", result.Baseline.Label, result.Candidate.Label) + fmt.Fprintf(&b, "Sources: %s -> %s\n", result.Baseline.Path, result.Candidate.Path) + b.WriteString("Run summary\n") + for _, row := range view.Metrics { + fmt.Fprintf(&b, "%s: %s -> %s (%s)\n", row.Name, row.Old, row.New, row.Delta) + } + fmt.Fprintf(&b, "Shared cases: %d; added: %d; removed: %d\n", result.Coverage.Shared, result.Coverage.Added, result.Coverage.Removed) + if result.Coverage.Added+result.Coverage.Removed > 0 { + b.WriteString("Different case sets: run-level deltas include unpaired cases.\n") + } + + b.WriteString("\nMetadata differences\n") + for _, row := range view.Metadata { + fmt.Fprintf(&b, "%s: %s -> %s (%s)\n", row.Name, row.Old, row.New, row.Delta) + } + + b.WriteString("\nCase transitions\n") + writeTransitions(&b, "fixed", result.Cases.Fixed) + writeTransitions(&b, "regressed", result.Cases.Regressed) + writeTransitions(&b, "changed", result.Cases.Changed) + writeTransitions(&b, "unchanged", result.Cases.Unchanged) + writeTransitions(&b, "added", result.Cases.Added) + writeTransitions(&b, "removed", result.Cases.Removed) + + if result.Gates.Passed { + b.WriteString("\nGates: passed\n") + } else { + b.WriteString("\nGates: failed\n") + for _, failure := range result.Gates.Failures { + fmt.Fprintf(&b, "- %s\n", failure) + } + } + return b.String() +} + +func writeTransitions(b *strings.Builder, label string, transitions []CaseTransition) { + fmt.Fprintf(b, "%s (%d):", label, len(transitions)) + for _, transition := range transitions { + fmt.Fprintf(b, " %s (%s)", transition.CaseID, transitionStatus(transition)) + } + b.WriteByte('\n') +} + +func transitionStatus(transition CaseTransition) string { + switch { + case transition.OldStatus == "": + return fmt.Sprintf("-> %s", transition.NewStatus) + case transition.NewStatus == "": + return fmt.Sprintf("%s ->", transition.OldStatus) + default: + return fmt.Sprintf("%s -> %s", transition.OldStatus, transition.NewStatus) + } +} From 5501691a79d86a127a88ece07103542ba8a4b875 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E8=B5=B5=E8=90=8D?= Date: Thu, 29 Oct 2026 05:56:36 +0000 Subject: [PATCH 2/9] feat(compare): support multiple runs against a shared baseline --- CHANGELOG.md | 6 +- README.md | 2 +- README.zh.md | 2 +- docs/guide/cli-reference.md | 37 +++- internal/cli/compare.go | 63 +++++-- internal/cli/compare_test.go | 95 +++++++++- internal/compare/render.go | 12 +- internal/compare/set.go | 241 ++++++++++++++++++++++++ internal/compare/set_test.go | 131 +++++++++++++ internal/compare/templates/compare.html | 68 ++++--- internal/compare/templates/set.html | 26 +++ 11 files changed, 616 insertions(+), 67 deletions(-) create mode 100644 internal/compare/set.go create mode 100644 internal/compare/set_test.go create mode 100644 internal/compare/templates/set.html diff --git a/CHANGELOG.md b/CHANGELOG.md index b05a1c5c..464aa78b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,10 +8,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] ### Added -- Add `skill-up compare ` for offline +- Add `skill-up compare ...` for offline comparison with self-contained HTML, shareable Markdown, text, and JSON - reports, case coverage and transitions, explicit baseline/candidate labels, - and optional regression and token-growth gates. + reports, an overview and case matrix across all runs, per-candidate baseline + comparisons, explicit labels, and optional regression and token-growth gates. ### Changed - Strengthen `skill-upper` evolution with original/candidate comparisons, diff --git a/README.md b/README.md index 56193ab5..3986f3d5 100644 --- a/README.md +++ b/README.md @@ -408,7 +408,7 @@ skill-up import ./evals/evals.json --output ./evals | `skill-up run [path]` | Run evaluation cases and produce reports | | `skill-up validate [path]` | Validate `eval.yaml` and case files | | `skill-up list-cases [path]` | List all cases referenced by the config | -| `skill-up compare ` | Compare finished runs; export HTML, Markdown, JSON or text | +| `skill-up compare ...` | Compare multiple finished runs; export HTML, Markdown, JSON or text | | `skill-up report ` | Generate reports from a previous run | | `skill-up import ` | Import Anthropic `evals.json` to YAML cases | | `skill-up debug judge ` | Debug judge module with a JSON input | diff --git a/README.zh.md b/README.zh.md index 55071710..f7742644 100644 --- a/README.zh.md +++ b/README.zh.md @@ -315,7 +315,7 @@ Engine 和凭据。它与旧式 `judge.model` 互斥，后者复用被测 Agent | `skill-up run [path]` | 运行评测用例并生成报告 | | `skill-up validate [path]` | 校验 `eval.yaml` 和用例文件 | | `skill-up list-cases [path]` | 列出配置引用的所有用例 | -| `skill-up compare ` | 离线对比评测结果，支持 HTML、Markdown、JSON 和文本 | +| `skill-up compare ...` | 离线对比多个评测结果，支持 HTML、Markdown、JSON 和文本 | | `skill-up report ` | 从已有结果生成报告 | | `skill-up import ` | 将 Anthropic `evals.json` 导入为 YAML 用例 | | `skill-up debug judge ` | 使用 JSON 输入调试 judge 模块 | diff --git a/docs/guide/cli-reference.md b/docs/guide/cli-reference.md index a41d5fe3..629fec8f 100644 --- a/docs/guide/cli-reference.md +++ b/docs/guide/cli-reference.md @@ -240,34 +240,46 @@ For single-run reports, `markdown` is limited to `skill-up report` as an offline ## skill-up compare -Compare two existing canonical `result.json` (or equivalent `report.json`) +Compare two or more existing canonical `result.json` (or equivalent `report.json`) files without running evaluations or calling any Agent or Judge. ```bash -skill-up compare [flags] +skill-up compare ... [flags] ``` +The first file is the baseline. Every remaining file is compared independently +against it, preserving argument order. Gates apply to every candidate; any failed +comparison makes the command fail. Inputs are never compared as a sequential chain. + ### Flags | Flag | Default | Description | |---|---|---| | `--format` | `text` | `text`, `json`, `html`, or `markdown` | | `--output` | stdout | Write the comparison to this file; its parent directory must exist | -| `--baseline-label` | `Baseline` | Display name for the old run | -| `--candidate-label` | `Candidate` | Display name for the new run | +| `--baseline-label` | `Baseline` | Display name for the first, baseline run | +| `--candidate-label` | automatic | Repeat once per candidate to label candidates in input order; supply all labels or omit the flag | | `--fail-on-regression` | `false` | Return a non-zero exit code for any shared case moving from PASS to another status | | `--max-token-increase-percent` | disabled | Maximum permitted increase in run-level total tokens (finite, non-negative percentage) | ### Examples ```bash -# Self-contained HTML, with searchable cases and transition filters -skill-up compare old/result.json new/result.json --format html --output comparison.html +# Compare several finished runs in one self-contained HTML report +skill-up compare baseline/result.json skill-a/result.json skill-b/result.json \ + --format html --output comparison.html + +# Two-run comparisons use the same command and report structure +skill-up compare old/result.json new/result.json --format html --output pair.html # Markdown tables for PR reviews and CI artifacts skill-up compare old/result.json new/result.json --format markdown --output comparison.md \ --baseline-label "Released Skill" --candidate-label "Proposed Skill" +# Explicit labels for multiple candidates (repeatable, commas remain literal) +skill-up compare baseline.json a.json b.json --format markdown --output variants.md \ + --candidate-label "Skill A" --candidate-label "Skill B" + # Produce a report even if the regression gate fails skill-up compare old/result.json new/result.json --format html --output comparison.html \ --fail-on-regression --max-token-increase-percent 20 @@ -277,8 +289,17 @@ skill-up compare old/result.json new/result.json --format json ``` All formats use the same comparison and gates. Reports include run metrics, -metadata, original source paths, and fixed, regressed, changed, unchanged, -added, and removed cases. HTML embeds its styles and filtering script, with +metadata, original source paths, a side-by-side overview of all runs, a case +status matrix across the union of case IDs, and detailed fixed, regressed, changed, +unchanged, added, and removed cases for each candidate against the baseline. +The case matrix marks cases absent from a run explicitly; JSON uses null matrix +cells. Matrix outcomes follow input order, and baseline-only cases are retained. +JSON always uses the same top-level `baseline`, `runs`, `case_matrix`, +`comparisons`, and `gates` structure, including for exactly two input files. +`comparisons` contains one baseline comparison per candidate, with individual +gate results; top-level `gates` aggregates failures and identifies the candidate. +Default labels are `Candidate` for one candidate, or `Candidate 1`, `Candidate 2`, +and so on for multiple candidates. HTML embeds its styles and filtering script, with no external assets or network requests; all cases remain readable without JavaScript. Markdown includes tables suitable for GitHub PR descriptions. Source result artifacts are preserved; output may not overwrite either input. diff --git a/internal/cli/compare.go b/internal/cli/compare.go index 3fcfda6e..0e00775b 100644 --- a/internal/cli/compare.go +++ b/internal/cli/compare.go @@ -16,16 +16,21 @@ import ( "github.com/alibaba/skill-up/internal/report" ) +const jsonFormat = "json" + var compareCmd = &cobra.Command{ - Use: "compare ", - Short: "Compare two offline evaluation results", + Use: "compare ...", + Short: "Compare two or more offline evaluation results", Long: `Compare existing results without running an Agent or Judge. +The first result is the baseline; every later result is compared against it. + Examples: + skill-up compare baseline.json skill-a.json skill-b.json --format html --output compare.html skill-up compare old.json new.json --format html --output compare.html skill-up compare old.json new.json --format markdown --output compare.md skill-up compare old.json new.json --format json --fail-on-regression`, - Args: cobra.ExactArgs(2), + Args: cobra.MinimumNArgs(2), RunE: runCompare, } @@ -37,7 +42,7 @@ func configureCompareFlags(cmd *cobra.Command) { cmd.Flags().String("format", "text", "Output format: text, json, html, markdown") cmd.Flags().String("output", "", "Write report to a file instead of stdout") cmd.Flags().String("baseline-label", "Baseline", "Label for the old run") - cmd.Flags().String("candidate-label", "Candidate", "Label for the new run") + cmd.Flags().StringArray("candidate-label", nil, "Candidate labels in input order; repeat once per candidate, or omit for automatic labels") cmd.Flags().Bool("fail-on-regression", false, "Fail when any case regresses") cmd.Flags().Float64("max-token-increase-percent", 0, "Maximum allowed total token increase percentage") } @@ -48,16 +53,15 @@ func runCompare(cmd *cobra.Command, args []string) error { return fmt.Errorf("get format flag: %w", err) } switch format { - case "text", "json", "html", "markdown": + case "text", jsonFormat, "html", "markdown": default: return fmt.Errorf("unsupported compare format %q; supported formats: text, json, html, markdown", format) } - oldInput, oldMissing, err := loadCompareInput("old", args[0]) - if err != nil { - return err + if len(args) < 2 { + return errors.New("compare requires at least two result files") } - newInput, newMissing, err := loadCompareInput("new", args[1]) + runs, err := loadBuildSet(cmd, args) if err != nil { return err } @@ -66,7 +70,7 @@ func runCompare(cmd *cobra.Command, args []string) error { if err != nil { return fmt.Errorf("get fail-on-regression flag: %w", err) } - options := comparepkg.Options{FailOnRegression: failOnRegression, OldMissingMetrics: oldMissing, NewMissingMetrics: newMissing} + options := comparepkg.Options{FailOnRegression: failOnRegression} maxTokenIncreasePercent, err := maxTokenIncreasePercent(cmd) if err != nil { return err @@ -75,12 +79,11 @@ func runCompare(cmd *cobra.Command, args []string) error { options.MaxTokenIncreasePercent = maxTokenIncreasePercent } - result := comparepkg.Compare(oldInput, newInput, options) - baselineLabel, _ := cmd.Flags().GetString("baseline-label") - candidateLabel, _ := cmd.Flags().GetString("candidate-label") - result.Baseline = comparepkg.Source{Label: baselineLabel, Path: args[0]} - result.Candidate = comparepkg.Source{Label: candidateLabel, Path: args[1]} - data, err := comparepkg.Render(result, format) + result, err := comparepkg.BuildSet(runs, options) + if err != nil { + return err + } + data, err := comparepkg.RenderSet(result, format) if err != nil { return err } @@ -95,6 +98,34 @@ func runCompare(cmd *cobra.Command, args []string) error { return nil } +func loadBuildSet(cmd *cobra.Command, args []string) ([]comparepkg.RunInput, error) { + baselineLabel, _ := cmd.Flags().GetString("baseline-label") + candidateLabels, _ := cmd.Flags().GetStringArray("candidate-label") + if len(candidateLabels) != 0 && len(candidateLabels) != len(args)-1 { + return nil, fmt.Errorf("--candidate-label must be omitted or supplied once per candidate: got %d labels for %d candidates", len(candidateLabels), len(args)-1) + } + runs := make([]comparepkg.RunInput, 0, len(args)) + for i, path := range args { + role, label := "old baseline", baselineLabel + if i > 0 { + role = fmt.Sprintf("new candidate %d", i) + label = "Candidate" + if len(args) > 2 { + label = fmt.Sprintf("Candidate %d", i) + } + if len(candidateLabels) > 0 { + label = candidateLabels[i-1] + } + } + input, missing, err := loadCompareInput(role, path) + if err != nil { + return nil, err + } + runs = append(runs, comparepkg.RunInput{Source: comparepkg.Source{Label: label, Path: path}, Input: input, MissingMetrics: missing}) + } + return runs, nil +} + func writeComparison(cmd *cobra.Command, inputs []string, output string, data []byte) error { if output == "" { if _, err := cmd.OutOrStdout().Write(data); err != nil { diff --git a/internal/cli/compare_test.go b/internal/cli/compare_test.go index 2f269f57..194425d6 100644 --- a/internal/cli/compare_test.go +++ b/internal/cli/compare_test.go @@ -12,6 +12,7 @@ import ( "github.com/spf13/cobra" + comparepkg "github.com/alibaba/skill-up/internal/compare" "github.com/alibaba/skill-up/internal/judge" "github.com/alibaba/skill-up/internal/report" ) @@ -84,14 +85,14 @@ func TestRunCompare_JSONFormat(t *testing.T) { } var result struct { - Run json.RawMessage `json:"run"` - Cases json.RawMessage `json:"cases"` - Gates json.RawMessage `json:"gates"` + Runs json.RawMessage `json:"runs"` + Comparisons []json.RawMessage `json:"comparisons"` + Gates json.RawMessage `json:"gates"` } if err := json.Unmarshal(out.Bytes(), &result); err != nil { t.Fatalf("unmarshal compare output: %v", err) } - if result.Run == nil || result.Cases == nil || result.Gates == nil { + if result.Runs == nil || len(result.Comparisons) != 1 || result.Gates == nil { t.Fatalf("JSON output missing stable result fields: %s", out.String()) } } @@ -380,3 +381,89 @@ func TestRootCommandIncludesCompare(t *testing.T) { t.Fatal("compare command not registered") } } + +func TestRunCompareMultipleInputsAndLabels(t *testing.T) { + t.Parallel() + dir := t.TempDir() + paths := []string{ + writeCompareResultJSON(t, dir, "baseline.json", makeReportInput(judge.StatusPass)), + writeCompareResultJSON(t, dir, "same.json", makeReportInput(judge.StatusPass)), + writeCompareResultJSON(t, dir, "regressed.json", makeReportInput(judge.StatusFail)), + } + for _, format := range []string{"json", "text", "html", "markdown"} { + t.Run(format, func(t *testing.T) { + t.Parallel() + cmd := newCompareCmd() + var out bytes.Buffer + cmd.SetOut(&out) + for flag, value := range map[string]string{"format": format, "fail-on-regression": "true"} { + if err := cmd.Flags().Set(flag, value); err != nil { + t.Fatal(err) + } + } + for _, label := range []string{"Same Skill", "Revised Skill"} { + if err := cmd.Flags().Set("candidate-label", label); err != nil { + t.Fatal(err) + } + } + if err := runCompare(cmd, paths); err == nil || !strings.Contains(err.Error(), "Revised Skill") { + t.Fatalf("later candidate gate not enforced: %v", err) + } + for _, want := range []string{"Same Skill", "Revised Skill", "regressed.json"} { + if !strings.Contains(out.String(), want) { + t.Errorf("%s output lost %q", format, want) + } + } + if format == "json" { + assertMultipleCompareJSON(t, out.Bytes()) + } + }) + } +} + +func TestRunCompareValidatesEveryCandidateBeforeWriting(t *testing.T) { + t.Parallel() + dir := t.TempDir() + valid := writeCompareResultJSON(t, dir, "valid.json", makeReportInput(judge.StatusPass)) + missing := filepath.Join(dir, "missing-third.json") + cmd := newCompareCmd() + var out bytes.Buffer + cmd.SetOut(&out) + if err := runCompare(cmd, []string{valid, valid, missing}); err == nil || !strings.Contains(err.Error(), missing) { + t.Fatalf("later invalid input ignored: %v", err) + } + if out.Len() != 0 { + t.Fatal("partial report emitted for invalid inputs") + } + if err := cmd.Flags().Set("candidate-label", "only-one-label"); err != nil { + t.Fatal(err) + } + if err := runCompare(cmd, []string{valid, valid, valid}); err == nil || !strings.Contains(err.Error(), "2 candidates") { + t.Fatalf("ambiguous label count accepted: %v", err) + } +} + +func TestRunCompareProtectsLaterCandidate(t *testing.T) { + t.Parallel() + dir := t.TempDir() + base := writeCompareResultJSON(t, dir, "baseline.json", makeReportInput(judge.StatusPass)) + later := writeCompareResultJSON(t, dir, "later.json", makeReportInput(judge.StatusFail)) + cmd := newCompareCmd() + if err := cmd.Flags().Set("output", later); err != nil { + t.Fatal(err) + } + if err := runCompare(cmd, []string{base, base, later}); err == nil || !strings.Contains(err.Error(), "must not overwrite") { + t.Fatalf("later candidate can be overwritten: %v", err) + } +} + +func assertMultipleCompareJSON(t *testing.T, data []byte) { + t.Helper() + var set comparepkg.Set + if err := json.Unmarshal(data, &set); err != nil { + t.Fatal(err) + } + if len(set.Runs) != 3 || len(set.Comparisons) != 2 || !set.Comparisons[0].Gates.Passed || set.Comparisons[1].Gates.Passed { + t.Fatalf("wrong per-candidate gates: %+v", set) + } +} diff --git a/internal/compare/render.go b/internal/compare/render.go index 9831c348..7479d811 100644 --- a/internal/compare/render.go +++ b/internal/compare/render.go @@ -13,12 +13,18 @@ import ( "time" ) -const unknownValue = "unknown" +const ( + unknownValue = "unknown" + jsonFormat = "json" +) //go:embed templates/compare.html var htmlSource string -var comparisonTemplate = template.Must(template.New("comparison").Funcs(template.FuncMap{"promptChange": promptChange}).Parse(htmlSource)) +//go:embed templates/set.html +var setHTMLSource string + +var comparisonTemplate = template.Must(template.New("comparison").Funcs(template.FuncMap{"promptChange": promptChange}).Parse(htmlSource + setHTMLSource)) type ( comparisonRow struct{ Name, Old, New, Delta string } @@ -40,7 +46,7 @@ func Render(result Result, format string) ([]byte, error) { switch format { case "text": return []byte(RenderText(result)), nil - case "json": + case jsonFormat: data, err := json.MarshalIndent(result, "", " ") if err != nil { return nil, fmt.Errorf("encode comparison: %w", err) diff --git a/internal/compare/set.go b/internal/compare/set.go new file mode 100644 index 00000000..90cc4855 --- /dev/null +++ b/internal/compare/set.go @@ -0,0 +1,241 @@ +package compare + +import ( + "bytes" + "encoding/json" + "errors" + "fmt" + "strings" + "text/tabwriter" + + "github.com/alibaba/skill-up/internal/judge" + "github.com/alibaba/skill-up/internal/report" +) + +// RunInput combines a source artifact with its recorded measurements. +type RunInput struct { + Source Source + Input report.Input + MissingMetrics []string +} + +// RunSummary contains a run's identity and primary-case metrics. +type RunSummary struct { + Source Source `json:"source"` + Metrics RunMetrics `json:"metrics"` +} + +// CaseOutcome retains the source result represented by one matrix cell. +type CaseOutcome struct { + Title string `json:"title"` + Status judge.Status `json:"status"` + Configuration string `json:"configuration,omitempty"` +} + +// CaseRow aligns outcomes to Runs; a nil outcome means the case is absent. +type CaseRow struct { + CaseID string `json:"case_id"` + Outcomes []*CaseOutcome `json:"outcomes"` +} + +// Set compares every candidate independently against the first, baseline run. +// Its schema is identical for two inputs and for larger collections. +type Set struct { + Baseline Source `json:"baseline"` + Runs []RunSummary `json:"runs"` + CaseMatrix []CaseRow `json:"case_matrix"` + Comparisons []Result `json:"comparisons"` + Gates GateResult `json:"gates"` +} + +// BuildSet compares two or more runs, preserving argument order and identity. +func BuildSet(runs []RunInput, options Options) (Set, error) { + if len(runs) < 2 { + return Set{}, errors.New("compare requires at least two result files") + } + set := Set{Baseline: runs[0].Source, Comparisons: make([]Result, 0, len(runs)-1), Gates: GateResult{Passed: true, Failures: []string{}}} + for _, run := range runs { + metrics := collectRunMetrics(run.Input) + metrics.Missing = run.MissingMetrics + set.Runs = append(set.Runs, RunSummary{Source: run.Source, Metrics: metrics}) + } + for _, candidate := range runs[1:] { + options.OldMissingMetrics, options.NewMissingMetrics = runs[0].MissingMetrics, candidate.MissingMetrics + result := Compare(runs[0].Input, candidate.Input, options) + result.Baseline, result.Candidate = runs[0].Source, candidate.Source + set.Comparisons = append(set.Comparisons, result) + for _, failure := range result.Gates.Failures { + set.Gates.Failures = append(set.Gates.Failures, fmt.Sprintf("%s (%s): %s", candidate.Source.Label, candidate.Source.Path, failure)) + } + } + set.Gates.Passed = len(set.Gates.Failures) == 0 + set.CaseMatrix = buildCaseMatrix(runs) + return set, nil +} + +func buildCaseMatrix(runs []RunInput) []CaseRow { + rows := make([]CaseRow, 0) + indices := make(map[string]int) + for runIndex, run := range runs { + for _, cr := range run.Input.PrimaryCaseResults() { + index, exists := indices[cr.CaseID] + if !exists { + index = len(rows) + indices[cr.CaseID] = index + rows = append(rows, CaseRow{CaseID: cr.CaseID, Outcomes: make([]*CaseOutcome, len(runs))}) + } + rows[index].Outcomes[runIndex] = &CaseOutcome{Title: cr.Title, Status: cr.Status, Configuration: cr.Configuration} + } + } + return rows +} + +type overviewRow struct { + Name string + Values []string +} + +type setView struct { + Set Set + Metrics []overviewRow + Matrix []overviewRow + Comparisons []comparisonView +} + +func newSetView(set Set) setView { + view := setView{Set: set} + for runIndex, run := range set.Runs { + rows := newComparisonView(Result{Run: RunComparison{Old: run.Metrics, New: run.Metrics}}).Metrics + for rowIndex, row := range rows { + if runIndex == 0 { + view.Metrics = append(view.Metrics, overviewRow{Name: row.Name}) + } + view.Metrics[rowIndex].Values = append(view.Metrics[rowIndex].Values, row.Old) + } + } + for _, row := range set.CaseMatrix { + values := make([]string, len(row.Outcomes)) + for i, outcome := range row.Outcomes { + values[i] = "absent" + if outcome != nil { + values[i] = string(outcome.Status) + } + } + view.Matrix = append(view.Matrix, overviewRow{Name: row.CaseID, Values: values}) + } + for _, result := range set.Comparisons { + view.Comparisons = append(view.Comparisons, newComparisonView(result)) + } + return view +} + +// RenderSet emits one report containing all runs and baseline comparisons. +func RenderSet(set Set, format string) ([]byte, error) { + switch format { + case jsonFormat: + data, err := json.MarshalIndent(set, "", " ") + if err != nil { + return nil, fmt.Errorf("encode comparison set: %w", err) + } + return append(data, '\n'), nil + case "html": + var b bytes.Buffer + if err := comparisonTemplate.ExecuteTemplate(&b, "set", newSetView(set)); err != nil { + return nil, fmt.Errorf("render HTML comparisons: %w", err) + } + return b.Bytes(), nil + case "markdown": + return []byte(renderSetMarkdown(set)), nil + case "text": + return []byte(renderSetText(set)), nil + default: + return nil, fmt.Errorf("unsupported comparison format %q", format) + } +} + +func renderSetMarkdown(set Set) string { + view := newSetView(set) + var b strings.Builder + fmt.Fprintf(&b, "# Evaluation comparison: %d runs\n\nBaseline: **%s**. Each candidate is compared independently against this baseline.\n\n", len(set.Runs), markdownCell(set.Baseline.Label)) + b.WriteString("## Run overview\n\n") + writeSetMarkdownTable(&b, "Metric", set.Runs, view.Metrics) + b.WriteString("\n## Case matrix\n\nAbsent means that the case was not present in that run.\n\n") + writeSetMarkdownTable(&b, "Case ID", set.Runs, view.Matrix) + for _, result := range set.Comparisons { + b.WriteString("\n---\n\n") + for line := range strings.SplitSeq(RenderMarkdown(result), "\n") { + if strings.HasPrefix(line, "#") { + b.WriteByte('#') + } + b.WriteString(line) + b.WriteByte('\n') + } + } + b.WriteString("\n## Overall gates\n\n") + if set.Gates.Passed { + b.WriteString("Passed (configured checks only).\n") + } else { + for _, failure := range set.Gates.Failures { + fmt.Fprintf(&b, "- %s\n", markdownCell(failure)) + } + } + return b.String() +} + +func writeSetMarkdownTable(b *strings.Builder, firstHeading string, runs []RunSummary, rows []overviewRow) { + fmt.Fprintf(b, "| %s |", firstHeading) + for _, run := range runs { + fmt.Fprintf(b, " %s |", markdownCell(run.Source.Label)) + } + b.WriteString("\n|---|") + for range runs { + b.WriteString("---|") + } + b.WriteByte('\n') + for _, row := range rows { + fmt.Fprintf(b, "| %s |", markdownCell(row.Name)) + for _, value := range row.Values { + fmt.Fprintf(b, " %s |", markdownCell(value)) + } + b.WriteByte('\n') + } +} + +func renderSetText(set Set) string { + var b strings.Builder + fmt.Fprintf(&b, "Run overview (%d runs; baseline: %s)\n", len(set.Runs), set.Baseline.Label) + view := newSetView(set) + writeSetTextTable(&b, "Metric", set.Runs, view.Metrics) + b.WriteString("\nCase matrix (absent = case not present in run)\n") + writeSetTextTable(&b, "Case ID", set.Runs, view.Matrix) + for _, result := range set.Comparisons { + b.WriteByte('\n') + b.WriteString(RenderText(result)) + } + if set.Gates.Passed { + b.WriteString("\nOverall gates: passed\n") + } else { + b.WriteString("\nOverall gates: failed\n") + for _, failure := range set.Gates.Failures { + fmt.Fprintf(&b, "- %s\n", failure) + } + } + return b.String() +} + +func writeSetTextTable(b *strings.Builder, firstHeading string, runs []RunSummary, rows []overviewRow) { + w := tabwriter.NewWriter(b, 0, 4, 2, ' ', 0) + _, _ = fmt.Fprint(w, firstHeading) + for _, run := range runs { + _, _ = fmt.Fprintf(w, "\t%s", run.Source.Label) + } + _, _ = fmt.Fprintln(w) + for _, row := range rows { + _, _ = fmt.Fprint(w, row.Name) + for _, value := range row.Values { + _, _ = fmt.Fprintf(w, "\t%s", value) + } + _, _ = fmt.Fprintln(w) + } + _ = w.Flush() +} diff --git a/internal/compare/set_test.go b/internal/compare/set_test.go new file mode 100644 index 00000000..a6881fa2 --- /dev/null +++ b/internal/compare/set_test.go @@ -0,0 +1,131 @@ +package compare + +import ( + "encoding/json" + "strings" + "testing" + + "github.com/alibaba/skill-up/internal/judge" + "github.com/alibaba/skill-up/internal/report" +) + +func multipleRunFixture() []RunInput { + baseline, candidate := compareFixture() + candidate.TotalTokens = 90 + regressed := candidate + regressed.TotalTokens = 140 + regressed.CaseResults = []report.CaseResult{ + {CaseID: "case-1", Status: judge.StatusFail}, + {CaseID: "case-2", Status: judge.StatusFail}, + {CaseID: "candidate-only", Status: judge.StatusPass}, + } + return []RunInput{ + {Source: Source{Label: "Baseline", Path: "base.json"}, Input: baseline}, + {Source: Source{Label: "Improved", Path: "improved.json"}, Input: candidate}, + {Source: Source{Label: "Regressed", Path: "regressed.json"}, Input: regressed}, + {Source: Source{Label: "Unknown usage", Path: "unknown.json"}, Input: candidate, MissingMetrics: []string{"total_tokens"}}, + } +} + +func TestCompareRunsUsesOneBaselineAndAggregatesAllGates(t *testing.T) { + t.Parallel() + limit := 10.0 + set, err := BuildSet(multipleRunFixture(), Options{FailOnRegression: true, MaxTokenIncreasePercent: &limit}) + if err != nil { + t.Fatal(err) + } + if len(set.Runs) != 4 || len(set.Comparisons) != 3 { + t.Fatalf("lost runs: %+v", set) + } + for _, pair := range set.Comparisons { + if pair.Baseline != set.Baseline || pair.Run.Old.TotalTokens != 100 || pair.Run.Old.CaseCount != 2 { + t.Fatalf("candidate used a different baseline or double-counted benchmark: %+v", pair) + } + } + if len(set.Comparisons[0].Cases.Fixed) != 1 || len(set.Comparisons[1].Cases.Regressed) != 1 || len(set.Comparisons[1].Cases.Unchanged) != 1 { + t.Fatal("transitions compared adjacent candidates instead of the baseline") + } + if set.Gates.Passed || len(set.Gates.Failures) != 3 { + t.Fatalf("gates = %+v", set.Gates) + } + for _, want := range []string{"Regressed (regressed.json)", "Unknown usage (unknown.json)"} { + if !strings.Contains(strings.Join(set.Gates.Failures, "\n"), want) { + t.Errorf("gate failure does not identify %s", want) + } + } +} + +func TestCaseMatrixPreservesUnionAndSourceOrder(t *testing.T) { + t.Parallel() + set, err := BuildSet(multipleRunFixture(), Options{}) + if err != nil { + t.Fatal(err) + } + if len(set.CaseMatrix) != 3 { + t.Fatalf("matrix = %+v", set.CaseMatrix) + } + row := set.CaseMatrix[2] + if row.CaseID != "candidate-only" || len(row.Outcomes) != 4 || row.Outcomes[0] != nil || row.Outcomes[1] != nil || row.Outcomes[2].Status != judge.StatusPass || row.Outcomes[3] != nil { + t.Fatalf("absent cases were paired: %+v", row) + } + if first := set.CaseMatrix[0].Outcomes[0]; first.Status != judge.StatusPass || first.Configuration != "with_skill" { + t.Fatalf("matrix did not use primary variant: %+v", first) + } +} + +func TestRenderSetIncludesEveryRunAndComparison(t *testing.T) { + t.Parallel() + runs := multipleRunFixture() + set, err := BuildSet(runs, Options{FailOnRegression: true}) + if err != nil { + t.Fatal(err) + } + for _, format := range []string{"text", "json", "html", "markdown"} { + data, err := RenderSet(set, format) + if err != nil { + t.Fatal(err) + } + for _, run := range runs { + if !strings.Contains(string(data), run.Source.Label) || !strings.Contains(string(data), run.Source.Path) { + t.Errorf("%s lost run %s", format, run.Source.Label) + } + } + if !strings.Contains(string(data), "candidate-only") { + t.Errorf("%s lost candidate-only case", format) + } + if format == "json" { + var output struct { + Runs []json.RawMessage `json:"runs"` + Comparisons []json.RawMessage `json:"comparisons"` + } + if err := json.Unmarshal(data, &output); err != nil { + t.Fatal(err) + } + if len(output.Runs) != 4 || len(output.Comparisons) != 3 { + t.Errorf("JSON lost runs: %s", data) + } + } else if !strings.Contains(string(data), "Case matrix") || !strings.Contains(string(data), "absent") { + t.Errorf("%s lost case matrix", format) + } + } +} + +func TestCompareRunsRejectsTooFewInputs(t *testing.T) { + t.Parallel() + for _, runs := range [][]RunInput{nil, multipleRunFixture()[:1]} { + if _, err := BuildSet(runs, Options{}); err == nil { + t.Fatal("accepted fewer than two runs") + } + } +} + +func TestSetPreservesUnknownUsage(t *testing.T) { + t.Parallel() + set, err := BuildSet(multipleRunFixture(), Options{}) + if err != nil { + t.Fatal(err) + } + if set.Runs[3].Metrics.Missing[0] != "total_tokens" || set.Comparisons[2].Run.Delta.Missing[0] != "total_tokens" { + t.Fatal("unknown measurements lost") + } +} diff --git a/internal/compare/templates/compare.html b/internal/compare/templates/compare.html index e53ef937..08078ba1 100644 --- a/internal/compare/templates/compare.html +++ b/internal/compare/templates/compare.html @@ -4,11 +4,16 @@ Evaluation comparison · {{.Result.Baseline.Label}} → {{.Result.Candidate.Label}} - + -
+
{{template "comparison-body" .}}
+ + +{{define "styles"}} +:root{color-scheme:light;--ink:#172738;--muted:#526476;--line:#dbe4ec;--paper:#fff;--accent:#125d68}*{box-sizing:border-box}[hidden]{display:none!important}body{margin:0;background:#f2f5f8;color:var(--ink);font:15px/1.6 system-ui,sans-serif}main{max-width:1200px;margin:auto;padding:48px 24px}header{margin-bottom:32px}.eyebrow{font:600 12px ui-monospace,monospace;letter-spacing:.15em;color:var(--accent)}h1{font-size:clamp(28px,4vw,44px);letter-spacing:-.04em;margin:8px 0}h2{font-size:21px;margin:0 0 16px}h3{font-size:16px}p{margin:10px 0}.muted,small{color:var(--muted)}.sources{overflow-wrap:anywhere}.cards{display:grid;grid-template-columns:repeat(auto-fit,minmax(160px,1fr));gap:12px;margin:24px 0}.card,section{background:var(--paper);border:1px solid var(--line);border-radius:12px}.card{padding:18px}.card strong{display:block;font-size:30px;line-height:1.3}.card span{font-size:13px;color:var(--muted)}section{padding:24px;margin:20px 0}.scroll{overflow-x:auto}table{border-collapse:collapse;width:100%;font-size:14px}th,td{padding:11px 14px;border-bottom:1px solid var(--line);text-align:left;vertical-align:top;overflow-wrap:anywhere}th{background:#f5f8fa;color:var(--muted);font-weight:600}td:first-child{font-weight:550}tbody tr:last-child td{border-bottom:0}.notice{padding:12px 16px;border-left:3px solid #ba7b21;background:#fff7e8}.passed{color:#176b45}.failed{color:#a32d37}.toolbar{display:flex;flex-wrap:wrap;gap:12px;margin:16px 0}input,select{padding:10px 12px;border:1px solid #aebfca;border-radius:6px;background:white;color:var(--ink);font:inherit}input{min-width:200px;flex:1}label{display:flex;gap:8px;align-items:center}details{margin-top:16px}summary{cursor:pointer;font-weight:600;padding:8px 0}.case-table td:first-child{font-family:ui-monospace,monospace}.case-table th:last-child,.case-table td:last-child{min-width:100px;white-space:nowrap}footer{margin-top:24px;color:var(--muted);font-size:12px}@media print{body{background:#fff}main{padding:0}.toolbar{display:none}section{break-inside:avoid}details{display:block}details::details-content{content-visibility:visible}.scroll{overflow:visible}} +.overview th{min-width:150px}.overview th small{display:block;font-weight:400;overflow-wrap:anywhere}.comparison{margin-top:40px;padding-top:24px;border-top:2px solid var(--line)}nav{display:flex;flex-wrap:wrap;gap:12px}nav a{color:var(--accent)} +{{end}} +{{define "comparison-body"}}
SKILL-UP / OFFLINE COMPARISON

{{.Result.Baseline.Label}} → {{.Result.Candidate.Label}}

Evaluation changes, case by case. Existing results only; no Agent or Judge calls.

@@ -34,8 +39,8 @@

{{.Result.Baseline.Label}} → {{.Result.Candidate.Label}}

Missing provenance is unknown. Matching case IDs do not prove identical prompts, suites, Skill revisions, or judge/rubric definitions. Scores are not regraded.

Case transitions

- -

+ +

{{range .Groups}}
{{.Name}} ({{len .Cases}}) {{if .Cases}}
{{range .Cases}}{{end}}
Case IDBaseline titleCandidate titleBaseline statusCandidate statusBaseline configurationCandidate configurationPrompt
{{.CaseID}}{{if .OldTitle}}{{.OldTitle}}{{else}}—{{end}}{{if .NewTitle}}{{.NewTitle}}{{else}}—{{end}}{{if .OldStatus}}{{.OldStatus}}{{else}}—{{end}}{{if .NewStatus}}{{.NewStatus}}{{else}}—{{end}}{{if .OldConfiguration}}{{.OldConfiguration}}{{else}}—{{end}}{{if .NewConfiguration}}{{.NewConfiguration}}{{else}}—{{end}}{{promptChange .PromptChanged}}
{{else}}

None.

{{end}}
{{end}} @@ -43,29 +48,30 @@

{{.Result.Baseline.Label}} → {{.Result.Candidate.Label}}

Gates: {{if .Result.Gates.Passed}}passed{{else}}failed{{end}}

{{if .Result.Gates.Passed}}

Configured checks passed. No checks are enforced unless requested.

{{else}}
    {{range .Result.Gates.Failures}}
  • {{.}}
    • {{end}}
{{end}}
Generated by skill-up compare. Self-contained report · no external assets · source artifacts preserved.
-
- - + report.querySelector('.match-count').textContent = count + ' matching cases'; + } + search.addEventListener('input', filterCases); + group.addEventListener('change', filterCases); + filterCases(); +}); +{{end}} diff --git a/internal/compare/templates/set.html b/internal/compare/templates/set.html new file mode 100644 index 00000000..ad680bd7 --- /dev/null +++ b/internal/compare/templates/set.html @@ -0,0 +1,26 @@ +{{define "set"}} + + + + +Evaluation comparison · {{len .Set.Runs}} runs + + +
+
SKILL-UP / OFFLINE COMPARISON
+

Evaluation comparison · {{len .Set.Runs}} runs

+

Baseline: {{.Set.Baseline.Label}}. Every candidate is compared independently against this baseline.

+

Existing results only; no Agent or Judge calls. Case outcomes align by stable case ID; absent cases are not paired samples.

+ +
+

Run overview

+{{range .Set.Runs}}{{end}} +{{range .Metrics}}{{range .Values}}{{end}}{{end}} +
Metric{{.Source.Label}}{{.Source.Path}}
{{.Name}}{{.}}
+

Case matrix

Absent means that the case was not present in that run. Hover over an outcome to see its source title.

+
{{range .Set.Runs}}{{end}} +{{range .Set.CaseMatrix}}{{range .Outcomes}}{{end}}{{end}}
Case ID{{.Source.Label}}
{{.CaseID}}{{if .}}{{.Status}}{{if .Configuration}}
{{.Configuration}}{{end}}{{else}}absent{{end}}
+

Overall gates: {{if .Set.Gates.Passed}}passed{{else}}failed{{end}}

+{{if .Set.Gates.Passed}}

Configured checks passed for every candidate. No checks are enforced unless requested.

{{else}}
    {{range .Set.Gates.Failures}}
  • {{.}}
    • {{end}}
{{end}}
+{{range $index, $view := .Comparisons}}
{{template "comparison-body" $view}}
{{end}} +
{{end}} From c8ca8cc4489ae8b688a6ca1db240bbca10bb390d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E8=B5=B5=E8=90=8D?= Date: Thu, 29 Oct 2026 06:26:10 +0000 Subject: [PATCH 3/9] feat(compare): derive run labels from directories with numeric fallback --- CHANGELOG.md | 5 +- README.md | 2 +- README.zh.md | 2 +- docs/guide/cli-reference.md | 35 +++++++++--- internal/cli/compare.go | 55 ++++++++++++++----- internal/cli/compare_test.go | 73 +++++++++++++++++++++++++ internal/compare/compare.go | 1 + internal/compare/render.go | 4 +- internal/compare/set.go | 15 +++-- internal/compare/templates/compare.html | 6 +- internal/compare/templates/set.html | 12 ++-- 11 files changed, 166 insertions(+), 44 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 464aa78b..5c33cacb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,10 +8,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] ### Added -- Add `skill-up compare ...` for offline +- Add `skill-up compare ...` for offline comparison with self-contained HTML, shareable Markdown, text, and JSON reports, an overview and case matrix across all runs, per-candidate baseline - comparisons, explicit labels, and optional regression and token-growth gates. + comparisons, directory-based labels with numeric ID fallback, explicit label + overrides, and optional regression and token-growth gates. ### Changed - Strengthen `skill-upper` evolution with original/candidate comparisons, diff --git a/README.md b/README.md index 3986f3d5..ee7bd15c 100644 --- a/README.md +++ b/README.md @@ -408,7 +408,7 @@ skill-up import ./evals/evals.json --output ./evals | `skill-up run [path]` | Run evaluation cases and produce reports | | `skill-up validate [path]` | Validate `eval.yaml` and case files | | `skill-up list-cases [path]` | List all cases referenced by the config | -| `skill-up compare ...` | Compare multiple finished runs; export HTML, Markdown, JSON or text | +| `skill-up compare ...` | Compare multiple finished runs; export HTML, Markdown, JSON or text | | `skill-up report ` | Generate reports from a previous run | | `skill-up import ` | Import Anthropic `evals.json` to YAML cases | | `skill-up debug judge ` | Debug judge module with a JSON input | diff --git a/README.zh.md b/README.zh.md index f7742644..3ca319fd 100644 --- a/README.zh.md +++ b/README.zh.md @@ -315,7 +315,7 @@ Engine 和凭据。它与旧式 `judge.model` 互斥，后者复用被测 Agent | `skill-up run [path]` | 运行评测用例并生成报告 | | `skill-up validate [path]` | 校验 `eval.yaml` 和用例文件 | | `skill-up list-cases [path]` | 列出配置引用的所有用例 | -| `skill-up compare ...` | 离线对比多个评测结果，支持 HTML、Markdown、JSON 和文本 | +| `skill-up compare ...` | 离线对比多个评测结果，支持 HTML、Markdown、JSON 和文本 | | `skill-up report ` | 从已有结果生成报告 | | `skill-up import ` | 将 Anthropic `evals.json` 导入为 YAML 用例 | | `skill-up debug judge ` | 使用 JSON 输入调试 judge 模块 | diff --git a/docs/guide/cli-reference.md b/docs/guide/cli-reference.md index 629fec8f..28815634 100644 --- a/docs/guide/cli-reference.md +++ b/docs/guide/cli-reference.md @@ -244,12 +244,13 @@ Compare two or more existing canonical `result.json` (or equivalent `report.json files without running evaluations or calling any Agent or Judge. ```bash -skill-up compare ... [flags] +skill-up compare ... [flags] ``` -The first file is the baseline. Every remaining file is compared independently -against it, preserving argument order. Gates apply to every candidate; any failed -comparison makes the command fail. Inputs are never compared as a sequential chain. +The first file is the reference for deltas and gates. Every remaining file is +compared independently against it, preserving argument order. This comparison +role does not determine its display name. Any failed comparison makes the +command fail; inputs are never compared as a sequential chain. ### Flags @@ -257,8 +258,8 @@ comparison makes the command fail. Inputs are never compared as a sequential cha |---|---|---| | `--format` | `text` | `text`, `json`, `html`, or `markdown` | | `--output` | stdout | Write the comparison to this file; its parent directory must exist | -| `--baseline-label` | `Baseline` | Display name for the first, baseline run | -| `--candidate-label` | automatic | Repeat once per candidate to label candidates in input order; supply all labels or omit the flag | +| `--baseline-label` | automatic | Override the first run's display label | +| `--candidate-label` | automatic | Override later runs' display labels in input order; repeat once per later input or omit the flag | | `--fail-on-regression` | `false` | Return a non-zero exit code for any shared case moving from PASS to another status | | `--max-token-increase-percent` | disabled | Maximum permitted increase in run-level total tokens (finite, non-negative percentage) | @@ -298,11 +299,27 @@ JSON always uses the same top-level `baseline`, `runs`, `case_matrix`, `comparisons`, and `gates` structure, including for exactly two input files. `comparisons` contains one baseline comparison per candidate, with individual gate results; top-level `gates` aggregates failures and identifies the candidate. -Default labels are `Candidate` for one candidate, or `Candidate 1`, `Candidate 2`, -and so on for multiple candidates. HTML embeds its styles and filtering script, with +All runs have a one-based numeric ID (`runs[].source.id`) assigned in input +order, including the reference. These IDs are also retained in each comparison's +source identity; display labels do not change them. + +Default display labels use the immediate parent-directory names if those names +are nonempty and unique across **all** inputs. Relative paths are resolved from +the current working directory. If any names collide or cannot identify a +directory, **all** labels fall back to the IDs `1`, `2`, `3`, and so on. For example: + +- `alpha/result.json beta/result.json gamma/result.json` → `alpha`, `beta`, `gamma`. +- `old.json new.json other.json` in one directory → `1`, `2`, `3`. +- `a/run/result.json b/run/result.json` → `1`, `2` (same directory basename). + +Explicit label flags override automatic display labels. The first run is never +automatically named `Baseline`, nor are later runs automatically named `Candidate`. +The JSON keys `baseline` and `candidate` describe comparison roles only. + +HTML embeds its styles and filtering script, with no external assets or network requests; all cases remain readable without JavaScript. Markdown includes tables suitable for GitHub PR descriptions. -Source result artifacts are preserved; output may not overwrite either input. +Source result artifacts are preserved; output may not overwrite any input. Cases align by case ID using the existing primary-result rule: prefer `with_skill`, then a non-baseline configuration, falling back to `without_skill`. diff --git a/internal/cli/compare.go b/internal/cli/compare.go index 0e00775b..95991d06 100644 --- a/internal/cli/compare.go +++ b/internal/cli/compare.go @@ -7,6 +7,7 @@ import ( "math" "os" "path/filepath" + "strconv" "strings" "github.com/spf13/cobra" @@ -19,14 +20,15 @@ import ( const jsonFormat = "json" var compareCmd = &cobra.Command{ - Use: "compare ...", + Use: "compare ...", Short: "Compare two or more offline evaluation results", Long: `Compare existing results without running an Agent or Judge. -The first result is the baseline; every later result is compared against it. +The first input is the reference for changes and gates. Display labels use +unique parent-directory names, or numeric IDs when those names are ambiguous. Examples: - skill-up compare baseline.json skill-a.json skill-b.json --format html --output compare.html + skill-up compare run-a/result.json run-b/result.json run-c/result.json --format html --output compare.html skill-up compare old.json new.json --format html --output compare.html skill-up compare old.json new.json --format markdown --output compare.md skill-up compare old.json new.json --format json --fail-on-regression`, @@ -41,8 +43,8 @@ func init() { func configureCompareFlags(cmd *cobra.Command) { cmd.Flags().String("format", "text", "Output format: text, json, html, markdown") cmd.Flags().String("output", "", "Write report to a file instead of stdout") - cmd.Flags().String("baseline-label", "Baseline", "Label for the old run") - cmd.Flags().StringArray("candidate-label", nil, "Candidate labels in input order; repeat once per candidate, or omit for automatic labels") + cmd.Flags().String("baseline-label", "", "Override the first run's automatic display label") + cmd.Flags().StringArray("candidate-label", nil, "Override later runs' automatic labels; repeat once per later input, or omit") cmd.Flags().Bool("fail-on-regression", false, "Fail when any case regresses") cmd.Flags().Float64("max-token-increase-percent", 0, "Maximum allowed total token increase percentage") } @@ -99,33 +101,58 @@ func runCompare(cmd *cobra.Command, args []string) error { } func loadBuildSet(cmd *cobra.Command, args []string) ([]comparepkg.RunInput, error) { + labels := automaticCompareLabels(args) baselineLabel, _ := cmd.Flags().GetString("baseline-label") + if baselineLabel != "" { + labels[0] = baselineLabel + } candidateLabels, _ := cmd.Flags().GetStringArray("candidate-label") if len(candidateLabels) != 0 && len(candidateLabels) != len(args)-1 { return nil, fmt.Errorf("--candidate-label must be omitted or supplied once per candidate: got %d labels for %d candidates", len(candidateLabels), len(args)-1) } runs := make([]comparepkg.RunInput, 0, len(args)) for i, path := range args { - role, label := "old baseline", baselineLabel + role := "old reference" if i > 0 { - role = fmt.Sprintf("new candidate %d", i) - label = "Candidate" - if len(args) > 2 { - label = fmt.Sprintf("Candidate %d", i) - } - if len(candidateLabels) > 0 { - label = candidateLabels[i-1] + role = fmt.Sprintf("new input %d", i+1) + if len(candidateLabels) > 0 && candidateLabels[i-1] != "" { + labels[i] = candidateLabels[i-1] } } input, missing, err := loadCompareInput(role, path) if err != nil { return nil, err } - runs = append(runs, comparepkg.RunInput{Source: comparepkg.Source{Label: label, Path: path}, Input: input, MissingMetrics: missing}) + runs = append(runs, comparepkg.RunInput{Source: comparepkg.Source{Label: labels[i], Path: path}, Input: input, MissingMetrics: missing}) } return runs, nil } +func automaticCompareLabels(paths []string) []string { + labels := make([]string, len(paths)) + seen := make(map[string]bool, len(paths)) + unique := true + for i, path := range paths { + absolute, err := filepath.Abs(path) + if err != nil { + unique = false + continue + } + name := filepath.Base(filepath.Dir(absolute)) + if name == "." || name == string(filepath.Separator) || strings.TrimSpace(name) == "" || seen[name] { + unique = false + } + seen[name] = true + labels[i] = name + } + if !unique { + for i := range labels { + labels[i] = strconv.Itoa(i + 1) + } + } + return labels +} + func writeComparison(cmd *cobra.Command, inputs []string, output string, data []byte) error { if output == "" { if _, err := cmd.OutOrStdout().Write(data); err != nil { diff --git a/internal/cli/compare_test.go b/internal/cli/compare_test.go index 194425d6..353f1cba 100644 --- a/internal/cli/compare_test.go +++ b/internal/cli/compare_test.go @@ -6,6 +6,7 @@ import ( "math" "os" "path/filepath" + "reflect" "strconv" "strings" "testing" @@ -467,3 +468,75 @@ func assertMultipleCompareJSON(t *testing.T, data []byte) { t.Fatalf("wrong per-candidate gates: %+v", set) } } + +func TestAutomaticCompareLabels(t *testing.T) { + t.Parallel() + root := t.TempDir() + path := func(parts ...string) string { return filepath.Join(append([]string{root}, parts...)...) } + tests := []struct { + name string + paths []string + want []string + }{ + {"distinct directories", []string{path("alpha", "result.json"), path("beta", "result.json"), path("gamma", "result.json")}, []string{"alpha", "beta", "gamma"}}, + {"same directory", []string{path("alpha", "a.json"), path("alpha", "b.json")}, []string{"1", "2"}}, + {"one duplicate switches all labels", []string{path("alpha", "result.json"), path("beta", "a.json"), path("beta", "b.json")}, []string{"1", "2", "3"}}, + {"same directory basename", []string{path("a", "run", "result.json"), path("b", "run", "result.json")}, []string{"1", "2"}}, + {"bare filenames", []string{"old.json", "new.json", "other.json"}, []string{"1", "2", "3"}}, + {"clean relative paths", []string{filepath.Join("alpha", "nested", "..", "result.json"), filepath.Join("beta", "result.json")}, []string{"alpha", "beta"}}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + if got := automaticCompareLabels(tc.paths); !reflect.DeepEqual(got, tc.want) { + t.Fatalf("labels = %v, want %v", got, tc.want) + } + }) + } +} + +func TestAutomaticLabelsPropagateToReportsAndRunIdentity(t *testing.T) { + t.Parallel() + root := t.TempDir() + var paths []string + for _, name := range []string{"alpha", "beta", "gamma"} { + dir := filepath.Join(root, name) + if err := os.Mkdir(dir, 0o700); err != nil { + t.Fatal(err) + } + paths = append(paths, writeCompareResultJSON(t, dir, "result.json", makeReportInput(judge.StatusPass))) + } + for _, format := range []string{"json", "html", "markdown", "text"} { + cmd := newCompareCmd() + var out bytes.Buffer + cmd.SetOut(&out) + if err := cmd.Flags().Set("format", format); err != nil { + t.Fatal(err) + } + if err := runCompare(cmd, paths); err != nil { + t.Fatal(err) + } + if strings.Contains(out.String(), "Candidate 1") || strings.Contains(out.String(), "Baseline") { + t.Errorf("%s still uses role-based display labels", format) + } + if format == jsonFormat { + assertDirectoryRunIdentity(t, out.Bytes()) + } + } +} + +func assertDirectoryRunIdentity(t *testing.T, data []byte) { + t.Helper() + var set comparepkg.Set + if err := json.Unmarshal(data, &set); err != nil { + t.Fatal(err) + } + for i, name := range []string{"alpha", "beta", "gamma"} { + if source := set.Runs[i].Source; source.ID != i+1 || source.Label != name { + t.Fatalf("wrong run identity: %+v", source) + } + } + if set.Baseline != set.Runs[0].Source || set.Comparisons[1].Candidate != set.Runs[2].Source { + t.Fatal("reference or comparison lost run identity") + } +} diff --git a/internal/compare/compare.go b/internal/compare/compare.go index 2192c802..096d00b5 100644 --- a/internal/compare/compare.go +++ b/internal/compare/compare.go @@ -32,6 +32,7 @@ type Result struct { // Source identifies an original result artifact independently of its configuration. type Source struct { + ID int `json:"id,omitempty"` Label string `json:"label"` Path string `json:"path"` } diff --git a/internal/compare/render.go b/internal/compare/render.go index 7479d811..af0e0023 100644 --- a/internal/compare/render.go +++ b/internal/compare/render.go @@ -154,7 +154,7 @@ func RenderMarkdown(result Result) string { fmt.Fprintf(&b, "**%s** → **%s**\n\n", markdownCell(result.Baseline.Label), markdownCell(result.Candidate.Label)) fmt.Fprintf(&b, "Sources: %s → %s\n\n", markdownCell(result.Baseline.Path), markdownCell(result.Candidate.Path)) fmt.Fprintf(&b, "Shared cases: **%d**; added: **%d**; removed: **%d**.\n\n", result.Coverage.Shared, result.Coverage.Added, result.Coverage.Removed) - fmt.Fprintf(&b, "Comparison coverage: %d / %d baseline cases; %d / %d candidate cases.\n\n", result.Coverage.Shared, result.Run.Old.CaseCount, result.Coverage.Shared, result.Run.New.CaseCount) + fmt.Fprintf(&b, "Comparison coverage: %d / %d reference cases; %d / %d compared cases.\n\n", result.Coverage.Shared, result.Run.Old.CaseCount, result.Coverage.Shared, result.Run.New.CaseCount) if result.Coverage.Added+result.Coverage.Removed > 0 { b.WriteString("> Different case sets: run-level deltas include unpaired cases. Only shared case IDs are classified as fixed or regressed.\n\n") } @@ -191,7 +191,7 @@ func writeMarkdownGroup(b *strings.Builder, group caseGroup) { b.WriteString("None.\n\n") return } - b.WriteString("| Case ID | Baseline title | Candidate title | Baseline status | Candidate status | Baseline configuration | Candidate configuration | Prompt |\n|---|---|---|---|---|---|---|---|\n") + b.WriteString("| Case ID | Reference title | Compared title | Reference status | Compared status | Reference configuration | Compared configuration | Prompt |\n|---|---|---|---|---|---|---|---|\n") for _, cr := range group.Cases { fmt.Fprintf(b, "| %s | %s | %s | %s | %s | %s | %s | %s |\n", markdownCell(cr.CaseID), markdownCell(cr.OldTitle), markdownCell(cr.NewTitle), markdownCell(string(cr.OldStatus)), markdownCell(string(cr.NewStatus)), markdownCell(cr.OldConfiguration), markdownCell(cr.NewConfiguration), promptChange(cr.PromptChanged)) } diff --git a/internal/compare/set.go b/internal/compare/set.go index 90cc4855..afdf084d 100644 --- a/internal/compare/set.go +++ b/internal/compare/set.go @@ -54,15 +54,18 @@ func BuildSet(runs []RunInput, options Options) (Set, error) { return Set{}, errors.New("compare requires at least two result files") } set := Set{Baseline: runs[0].Source, Comparisons: make([]Result, 0, len(runs)-1), Gates: GateResult{Passed: true, Failures: []string{}}} - for _, run := range runs { + for index, run := range runs { + source := run.Source + source.ID = index + 1 metrics := collectRunMetrics(run.Input) metrics.Missing = run.MissingMetrics - set.Runs = append(set.Runs, RunSummary{Source: run.Source, Metrics: metrics}) + set.Runs = append(set.Runs, RunSummary{Source: source, Metrics: metrics}) } - for _, candidate := range runs[1:] { + set.Baseline = set.Runs[0].Source + for index, candidate := range runs[1:] { options.OldMissingMetrics, options.NewMissingMetrics = runs[0].MissingMetrics, candidate.MissingMetrics result := Compare(runs[0].Input, candidate.Input, options) - result.Baseline, result.Candidate = runs[0].Source, candidate.Source + result.Baseline, result.Candidate = set.Baseline, set.Runs[index+1].Source set.Comparisons = append(set.Comparisons, result) for _, failure := range result.Gates.Failures { set.Gates.Failures = append(set.Gates.Failures, fmt.Sprintf("%s (%s): %s", candidate.Source.Label, candidate.Source.Path, failure)) @@ -156,7 +159,7 @@ func RenderSet(set Set, format string) ([]byte, error) { func renderSetMarkdown(set Set) string { view := newSetView(set) var b strings.Builder - fmt.Fprintf(&b, "# Evaluation comparison: %d runs\n\nBaseline: **%s**. Each candidate is compared independently against this baseline.\n\n", len(set.Runs), markdownCell(set.Baseline.Label)) + fmt.Fprintf(&b, "# Evaluation comparison: %d runs\n\nReference for changes: **%s**. Every other run is compared independently against this reference.\n\n", len(set.Runs), markdownCell(set.Baseline.Label)) b.WriteString("## Run overview\n\n") writeSetMarkdownTable(&b, "Metric", set.Runs, view.Metrics) b.WriteString("\n## Case matrix\n\nAbsent means that the case was not present in that run.\n\n") @@ -203,7 +206,7 @@ func writeSetMarkdownTable(b *strings.Builder, firstHeading string, runs []RunSu func renderSetText(set Set) string { var b strings.Builder - fmt.Fprintf(&b, "Run overview (%d runs; baseline: %s)\n", len(set.Runs), set.Baseline.Label) + fmt.Fprintf(&b, "Run overview (%d runs; reference: %s)\n", len(set.Runs), set.Baseline.Label) view := newSetView(set) writeSetTextTable(&b, "Metric", set.Runs, view.Metrics) b.WriteString("\nCase matrix (absent = case not present in run)\n") diff --git a/internal/compare/templates/compare.html b/internal/compare/templates/compare.html index 08078ba1..aa58fa8f 100644 --- a/internal/compare/templates/compare.html +++ b/internal/compare/templates/compare.html @@ -17,7 +17,7 @@
SKILL-UP / OFFLINE COMPARISON

{{.Result.Baseline.Label}} → {{.Result.Candidate.Label}}

Evaluation changes, case by case. Existing results only; no Agent or Judge calls.

-

Baseline source: {{.Result.Baseline.Path}}
Candidate source: {{.Result.Candidate.Path}}

+

Reference source: {{.Result.Baseline.Path}}
Compared source: {{.Result.Candidate.Path}}

{{.Result.Coverage.Shared}}Shared cases
@@ -26,7 +26,7 @@

{{.Result.Baseline.Label}} → {{.Result.Candidate.Label}}

{{.Result.Coverage.Added}}Added
{{.Result.Coverage.Removed}}Removed
-

Comparison coverage: {{.Result.Coverage.Shared}} / {{.Result.Run.Old.CaseCount}} baseline cases; {{.Result.Coverage.Shared}} / {{.Result.Run.New.CaseCount}} candidate cases.

+

Comparison coverage: {{.Result.Coverage.Shared}} / {{.Result.Run.Old.CaseCount}} reference cases; {{.Result.Coverage.Shared}} / {{.Result.Run.New.CaseCount}} compared cases.

{{if or .Result.Coverage.Added .Result.Coverage.Removed}}

Different case sets: run-level deltas include unpaired cases. Only shared case IDs are classified as fixed or regressed.

{{end}}

Run summary

@@ -42,7 +42,7 @@

{{.Result.Baseline.Label}} → {{.Result.Candidate.Label}}

{{range .Groups}}
{{.Name}} ({{len .Cases}}) -{{if .Cases}}
Metric{{.Result.Baseline.Label}}{{.Result.Candidate.Label}}Change
+{{if .Cases}}
Case IDBaseline titleCandidate titleBaseline statusCandidate statusBaseline configurationCandidate configurationPrompt
{{range .Cases}}{{end}}
Case IDReference titleCompared titleReference statusCompared statusReference configurationCompared configurationPrompt
{{.CaseID}}{{if .OldTitle}}{{.OldTitle}}{{else}}—{{end}}{{if .NewTitle}}{{.NewTitle}}{{else}}—{{end}}{{if .OldStatus}}{{.OldStatus}}{{else}}—{{end}}{{if .NewStatus}}{{.NewStatus}}{{else}}—{{end}}{{if .OldConfiguration}}{{.OldConfiguration}}{{else}}—{{end}}{{if .NewConfiguration}}{{.NewConfiguration}}{{else}}—{{end}}{{promptChange .PromptChanged}}
{{else}}

None.

{{end}}{{end}}

Gates: {{if .Result.Gates.Passed}}passed{{else}}failed{{end}}

diff --git a/internal/compare/templates/set.html b/internal/compare/templates/set.html index ad680bd7..f030b7c9 100644 --- a/internal/compare/templates/set.html +++ b/internal/compare/templates/set.html @@ -9,18 +9,18 @@
SKILL-UP / OFFLINE COMPARISON

Evaluation comparison · {{len .Set.Runs}} runs

-

Baseline: {{.Set.Baseline.Label}}. Every candidate is compared independently against this baseline.

+

Reference for changes: {{.Set.Baseline.Label}}. Every other run is compared independently against this reference.

Existing results only; no Agent or Judge calls. Case outcomes align by stable case ID; absent cases are not paired samples.

- +

Run overview

-{{range .Set.Runs}}{{end}} +{{range .Set.Runs}}{{end}}{{range .Metrics}}{{range .Values}}{{end}}{{end}}
Metric{{.Source.Label}}{{.Source.Path}}
Metric{{.Source.Label}}{{.Source.Path}}
{{.Name}}{{.}}

Case matrix

Absent means that the case was not present in that run. Hover over an outcome to see its source title.

-
{{range .Set.Runs}}{{end}} +
Case ID{{.Source.Label}}
{{range .Set.Runs}}{{end}}{{range .Set.CaseMatrix}}{{range .Outcomes}}{{end}}{{end}}
Case ID{{.Source.Label}}
{{.CaseID}}{{if .}}{{.Status}}{{if .Configuration}}
{{.Configuration}}{{end}}{{else}}absent{{end}}

Overall gates: {{if .Set.Gates.Passed}}passed{{else}}failed{{end}}

-{{if .Set.Gates.Passed}}

Configured checks passed for every candidate. No checks are enforced unless requested.

{{else}}
    {{range .Set.Gates.Failures}}
  • {{.}}
    • {{end}}
{{end}}
-{{range $index, $view := .Comparisons}}
{{template "comparison-body" $view}}
{{end}} +{{if .Set.Gates.Passed}}

Configured checks passed for every compared run. No checks are enforced unless requested.

{{else}}
    {{range .Set.Gates.Failures}}
  • {{.}}
    • {{end}}
{{end}}
+{{range $index, $view := .Comparisons}}
{{template "comparison-body" $view}}
{{end}} {{end}} From a68e7867c2c0164503ff392b1ac06a12e0e4e697 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E8=B5=B5=E8=90=8D?= Date: Thu, 29 Oct 2026 06:33:09 +0000 Subject: [PATCH 4/9] refactor(report): extend existing HTML report for offline comparisons --- CHANGELOG.md | 4 +- docs/guide/cli-reference.md | 5 ++ internal/compare/render.go | 6 +- internal/compare/set.go | 12 +++- internal/compare/templates/compare.html | 75 ++++++------------------- internal/compare/templates/set.html | 27 +++------ internal/report/html.go | 66 +++++++++++++++++++--- internal/report/html_comparison_test.go | 44 +++++++++++++++ internal/report/templates/report.html | 75 +++++++++++++++++++++++-- 9 files changed, 219 insertions(+), 95 deletions(-) create mode 100644 internal/report/html_comparison_test.go diff --git a/CHANGELOG.md b/CHANGELOG.md index 5c33cacb..20f0d338 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,7 +12,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 comparison with self-contained HTML, shareable Markdown, text, and JSON reports, an overview and case matrix across all runs, per-candidate baseline comparisons, directory-based labels with numeric ID fallback, explicit label - overrides, and optional regression and token-growth gates. + overrides, and optional regression and token-growth gates. HTML extends the + existing report with a comparison tab and run selector, retaining the original + case details and leaving existing report output unchanged. ### Changed - Strengthen `skill-upper` evolution with original/candidate comparisons, diff --git a/docs/guide/cli-reference.md b/docs/guide/cli-reference.md index 28815634..c29f398c 100644 --- a/docs/guide/cli-reference.md +++ b/docs/guide/cli-reference.md @@ -316,6 +316,11 @@ Explicit label flags override automatic display labels. The first run is never automatically named `Baseline`, nor are later runs automatically named `Candidate`. The JSON keys `baseline` and `candidate` describe comparison roles only. +HTML extends the existing evaluation report with a Comparison tab and a run +selector. Selecting a run reuses the original Cases and Benchmark views, including +prompts, responses, formal grades, and evidence. The existing single-run and +with/without-Skill report presentation is unchanged. Comparison-mode feedback is +isolated per run and exported locally, without using the single-run feedback API. HTML embeds its styles and filtering script, with no external assets or network requests; all cases remain readable without JavaScript. Markdown includes tables suitable for GitHub PR descriptions. diff --git a/internal/compare/render.go b/internal/compare/render.go index af0e0023..aa0a17ce 100644 --- a/internal/compare/render.go +++ b/internal/compare/render.go @@ -11,6 +11,8 @@ import ( "strconv" "strings" "time" + + "github.com/alibaba/skill-up/internal/report" ) const ( @@ -56,10 +58,10 @@ func Render(result Result, format string) ([]byte, error) { return []byte(RenderMarkdown(result)), nil case "html": var b bytes.Buffer - if err := comparisonTemplate.Execute(&b, newComparisonView(result)); err != nil { + if err := comparisonTemplate.ExecuteTemplate(&b, "comparison-body", newComparisonView(result)); err != nil { return nil, fmt.Errorf("render HTML comparison: %w", err) } - return b.Bytes(), nil + return report.RenderComparisonHTML([]report.ComparisonRun{{Label: result.Baseline.Label}}, template.HTML(b.String())) //nolint:gosec // html/template escaped all source content. default: return nil, fmt.Errorf("unsupported comparison format %q", format) } diff --git a/internal/compare/set.go b/internal/compare/set.go index afdf084d..4c66c6cd 100644 --- a/internal/compare/set.go +++ b/internal/compare/set.go @@ -5,6 +5,7 @@ import ( "encoding/json" "errors" "fmt" + "html/template" "strings" "text/tabwriter" @@ -41,6 +42,7 @@ type CaseRow struct { // Set compares every candidate independently against the first, baseline run. // Its schema is identical for two inputs and for larger collections. type Set struct { + inputs []report.Input Baseline Source `json:"baseline"` Runs []RunSummary `json:"runs"` CaseMatrix []CaseRow `json:"case_matrix"` @@ -55,6 +57,7 @@ func BuildSet(runs []RunInput, options Options) (Set, error) { } set := Set{Baseline: runs[0].Source, Comparisons: make([]Result, 0, len(runs)-1), Gates: GateResult{Passed: true, Failures: []string{}}} for index, run := range runs { + set.inputs = append(set.inputs, run.Input) source := run.Source source.ID = index + 1 metrics := collectRunMetrics(run.Input) @@ -146,7 +149,14 @@ func RenderSet(set Set, format string) ([]byte, error) { if err := comparisonTemplate.ExecuteTemplate(&b, "set", newSetView(set)); err != nil { return nil, fmt.Errorf("render HTML comparisons: %w", err) } - return b.Bytes(), nil + runs := make([]report.ComparisonRun, len(set.Runs)) + for i, run := range set.Runs { + runs[i].Label = run.Source.Label + if i < len(set.inputs) { + runs[i].Input = set.inputs[i] + } + } + return report.RenderComparisonHTML(runs, template.HTML(b.String())) //nolint:gosec // html/template escaped all source content. case "markdown": return []byte(renderSetMarkdown(set)), nil case "text": diff --git a/internal/compare/templates/compare.html b/internal/compare/templates/compare.html index aa58fa8f..45982014 100644 --- a/internal/compare/templates/compare.html +++ b/internal/compare/templates/compare.html @@ -1,77 +1,36 @@ - - - - - -Evaluation comparison · {{.Result.Baseline.Label}} → {{.Result.Candidate.Label}} - - -
{{template "comparison-body" .}}
- - -{{define "styles"}} -:root{color-scheme:light;--ink:#172738;--muted:#526476;--line:#dbe4ec;--paper:#fff;--accent:#125d68}*{box-sizing:border-box}[hidden]{display:none!important}body{margin:0;background:#f2f5f8;color:var(--ink);font:15px/1.6 system-ui,sans-serif}main{max-width:1200px;margin:auto;padding:48px 24px}header{margin-bottom:32px}.eyebrow{font:600 12px ui-monospace,monospace;letter-spacing:.15em;color:var(--accent)}h1{font-size:clamp(28px,4vw,44px);letter-spacing:-.04em;margin:8px 0}h2{font-size:21px;margin:0 0 16px}h3{font-size:16px}p{margin:10px 0}.muted,small{color:var(--muted)}.sources{overflow-wrap:anywhere}.cards{display:grid;grid-template-columns:repeat(auto-fit,minmax(160px,1fr));gap:12px;margin:24px 0}.card,section{background:var(--paper);border:1px solid var(--line);border-radius:12px}.card{padding:18px}.card strong{display:block;font-size:30px;line-height:1.3}.card span{font-size:13px;color:var(--muted)}section{padding:24px;margin:20px 0}.scroll{overflow-x:auto}table{border-collapse:collapse;width:100%;font-size:14px}th,td{padding:11px 14px;border-bottom:1px solid var(--line);text-align:left;vertical-align:top;overflow-wrap:anywhere}th{background:#f5f8fa;color:var(--muted);font-weight:600}td:first-child{font-weight:550}tbody tr:last-child td{border-bottom:0}.notice{padding:12px 16px;border-left:3px solid #ba7b21;background:#fff7e8}.passed{color:#176b45}.failed{color:#a32d37}.toolbar{display:flex;flex-wrap:wrap;gap:12px;margin:16px 0}input,select{padding:10px 12px;border:1px solid #aebfca;border-radius:6px;background:white;color:var(--ink);font:inherit}input{min-width:200px;flex:1}label{display:flex;gap:8px;align-items:center}details{margin-top:16px}summary{cursor:pointer;font-weight:600;padding:8px 0}.case-table td:first-child{font-family:ui-monospace,monospace}.case-table th:last-child,.case-table td:last-child{min-width:100px;white-space:nowrap}footer{margin-top:24px;color:var(--muted);font-size:12px}@media print{body{background:#fff}main{padding:0}.toolbar{display:none}section{break-inside:avoid}details{display:block}details::details-content{content-visibility:visible}.scroll{overflow:visible}} -.overview th{min-width:150px}.overview th small{display:block;font-weight:400;overflow-wrap:anywhere}.comparison{margin-top:40px;padding-top:24px;border-top:2px solid var(--line)}nav{display:flex;flex-wrap:wrap;gap:12px}nav a{color:var(--accent)} -{{end}} {{define "comparison-body"}}
SKILL-UP / OFFLINE COMPARISON

{{.Result.Baseline.Label}} → {{.Result.Candidate.Label}}

-

Evaluation changes, case by case. Existing results only; no Agent or Judge calls.

+

Evaluation changes, case by case. Existing results only; no Agent or Judge calls.

Reference source: {{.Result.Baseline.Path}}
Compared source: {{.Result.Candidate.Path}}

-
-
{{.Result.Coverage.Shared}}Shared cases
-
{{len .Result.Cases.Regressed}}Regressed
-
{{len .Result.Cases.Fixed}}Fixed
-
{{.Result.Coverage.Added}}Added
-
{{.Result.Coverage.Removed}}Removed
+
+
{{.Result.Coverage.Shared}}
Shared cases
+
{{len .Result.Cases.Regressed}}
Regressed
+
{{len .Result.Cases.Fixed}}
Fixed
+
{{.Result.Coverage.Added}}
Added
+
{{.Result.Coverage.Removed}}
Removed
-

Comparison coverage: {{.Result.Coverage.Shared}} / {{.Result.Run.Old.CaseCount}} reference cases; {{.Result.Coverage.Shared}} / {{.Result.Run.New.CaseCount}} compared cases.

+

Comparison coverage: {{.Result.Coverage.Shared}} / {{.Result.Run.Old.CaseCount}} reference cases; {{.Result.Coverage.Shared}} / {{.Result.Run.New.CaseCount}} compared cases.

{{if or .Result.Coverage.Added .Result.Coverage.Removed}}

Different case sets: run-level deltas include unpaired cases. Only shared case IDs are classified as fixed or regressed.

{{end}} -

Run summary

+

Run summary

{{range .Metrics}}{{end}}
Metric{{.Result.Baseline.Label}}{{.Result.Candidate.Label}}Change
{{.Name}}{{.Old}}{{.New}}{{.Delta}}
Pass rates use one primary result per case ID. Total tokens cover all configurations; input/output tokens cover primary cases. Pass-rate changes are percentage points (pp). Missing measurements are unknown.
-

Metadata differences

+

Metadata differences

{{range .Metadata}}{{end}}
Field{{.Result.Baseline.Label}}{{.Result.Candidate.Label}}Change
{{.Name}}{{.Old}}{{.New}}{{.Delta}}
-

Missing provenance is unknown. Matching case IDs do not prove identical prompts, suites, Skill revisions, or judge/rubric definitions. Scores are not regraded.

+

Missing provenance is unknown. Matching case IDs do not prove identical prompts, suites, Skill revisions, or judge/rubric definitions. Scores are not regraded.

-

Case transitions

- -

+

Case transitions

+ +
{{range .Groups}}
{{.Name}} ({{len .Cases}}) -{{if .Cases}}
-{{range .Cases}}{{end}}
Case IDReference titleCompared titleReference statusCompared statusReference configurationCompared configurationPrompt
{{.CaseID}}{{if .OldTitle}}{{.OldTitle}}{{else}}—{{end}}{{if .NewTitle}}{{.NewTitle}}{{else}}—{{end}}{{if .OldStatus}}{{.OldStatus}}{{else}}—{{end}}{{if .NewStatus}}{{.NewStatus}}{{else}}—{{end}}{{if .OldConfiguration}}{{.OldConfiguration}}{{else}}—{{end}}{{if .NewConfiguration}}{{.NewConfiguration}}{{else}}—{{end}}{{promptChange .PromptChanged}}
{{else}}

None.

{{end}}
{{end}} +{{if .Cases}}
+{{range .Cases}}{{end}}
Case IDReference titleCompared titleReference statusCompared statusReference configurationCompared configurationPrompt
{{.CaseID}}{{if .OldTitle}}{{.OldTitle}}{{else}}—{{end}}{{if .NewTitle}}{{.NewTitle}}{{else}}—{{end}}{{if .OldStatus}}{{.OldStatus}}{{else}}—{{end}}{{if .NewStatus}}{{.NewStatus}}{{else}}—{{end}}{{if .OldConfiguration}}{{.OldConfiguration}}{{else}}—{{end}}{{if .NewConfiguration}}{{.NewConfiguration}}{{else}}—{{end}}{{promptChange .PromptChanged}}
{{else}}

None.

{{end}}{{end}}
-

Gates: {{if .Result.Gates.Passed}}passed{{else}}failed{{end}}

+

Gates: {{if .Result.Gates.Passed}}passed{{else}}failed{{end}}

{{if .Result.Gates.Passed}}

Configured checks passed. No checks are enforced unless requested.

{{else}}
    {{range .Result.Gates.Failures}}
  • {{.}}
    • {{end}}
{{end}}
Generated by skill-up compare. Self-contained report · no external assets · source artifacts preserved.
{{end}} -{{define "scripts"}} -document.querySelectorAll('.comparison').forEach(report => { - const filters = report.querySelector('.filters'); - filters.hidden = false; - const search = report.querySelector('.case-search'); - const group = report.querySelector('.case-filter'); - function filterCases() { - let count = 0; - const query = search.value.trim().toLowerCase(); - report.querySelectorAll('.case-group').forEach(section => { - let visible = 0; - const selected = !group.value || section.dataset.group === group.value; - section.querySelectorAll('tbody tr').forEach(row => { - row.hidden = !selected || !row.textContent.toLowerCase().includes(query); - if (!row.hidden) visible++; - }); - section.hidden = !selected || (!!query && !visible); - count += visible; - }); - report.querySelector('.match-count').textContent = count + ' matching cases'; - } - search.addEventListener('input', filterCases); - group.addEventListener('change', filterCases); - filterCases(); -}); -{{end}} diff --git a/internal/compare/templates/set.html b/internal/compare/templates/set.html index f030b7c9..ad851322 100644 --- a/internal/compare/templates/set.html +++ b/internal/compare/templates/set.html @@ -1,26 +1,17 @@ -{{define "set"}} - - - - -Evaluation comparison · {{len .Set.Runs}} runs - - -
-
SKILL-UP / OFFLINE COMPARISON
+{{define "set"}}
SKILL-UP / OFFLINE COMPARISON

Evaluation comparison · {{len .Set.Runs}} runs

Reference for changes: {{.Set.Baseline.Label}}. Every other run is compared independently against this reference.

-

Existing results only; no Agent or Judge calls. Case outcomes align by stable case ID; absent cases are not paired samples.

+

Existing results only; no Agent or Judge calls. Case outcomes align by stable case ID; absent cases are not paired samples.

-

Run overview

+

Run overview

{{range .Set.Runs}}{{end}}{{range .Metrics}}{{range .Values}}{{end}}{{end}}
Metric{{.Source.Label}}{{.Source.Path}}
{{.Name}}{{.}}
-

Case matrix

Absent means that the case was not present in that run. Hover over an outcome to see its source title.

-
{{range .Set.Runs}}{{end}} -{{range .Set.CaseMatrix}}{{range .Outcomes}}{{end}}{{end}}
Case ID{{.Source.Label}}
{{.CaseID}}{{if .}}{{.Status}}{{if .Configuration}}
{{.Configuration}}{{end}}{{else}}absent{{end}}
-

Overall gates: {{if .Set.Gates.Passed}}passed{{else}}failed{{end}}

+

Case matrix

Absent means that the case was not present in that run. Hover over an outcome to see its source title.

+
{{range .Set.Runs}}{{end}} +{{range .Set.CaseMatrix}}{{range .Outcomes}}{{end}}{{end}}
Case ID{{.Source.Label}}
{{.CaseID}}{{if .}}{{.Status}}{{if .Configuration}}
{{.Configuration}}{{end}}{{else}}absent{{end}}
+

Overall gates: {{if .Set.Gates.Passed}}passed{{else}}failed{{end}}

{{if .Set.Gates.Passed}}

Configured checks passed for every compared run. No checks are enforced unless requested.

{{else}}
    {{range .Set.Gates.Failures}}
  • {{.}}
    • {{end}}
{{end}}
-{{range $index, $view := .Comparisons}}
{{template "comparison-body" $view}}
{{end}} -
{{end}} +{{range $index, $view := .Comparisons}}
{{template "comparison-body" $view}}
{{end}} +{{end}} diff --git a/internal/report/html.go b/internal/report/html.go index 6ab3c7b0..acc20035 100644 --- a/internal/report/html.go +++ b/internal/report/html.go @@ -1,10 +1,12 @@ package report import ( + "bytes" "context" _ "embed" "encoding/base64" "encoding/json" + "errors" "fmt" "html/template" "io" @@ -29,20 +31,64 @@ func (r *HTMLReporter) Write(_ context.Context, in Input) error { return err } + return writeToOutput(r.OutputPath, "html report", func(w io.Writer) error { return executeHTMLReport(w, data) }) +} + +func executeHTMLReport(w io.Writer, data htmlReportData) error { funcMap := SharedTemplateFuncs() funcMap["statusIcon"] = statusIcon - tmpl, err := template.New("report").Funcs(funcMap).Parse(htmlTemplate) if err != nil { return fmt.Errorf("parse html template: %w", err) } + if err := tmpl.Execute(w, data); err != nil { + return fmt.Errorf("execute html template: %w", err) + } + return nil +} + +// ComparisonRun supplies a full original report for the comparison run selector. +type ComparisonRun struct { + Label string + Input Input +} - return writeToOutput(r.OutputPath, "html report", func(w io.Writer) error { - if err := tmpl.Execute(w, data); err != nil { - return fmt.Errorf("execute html template: %w", err) +// RenderComparisonHTML extends the existing report with a trusted, template-rendered +// comparison section and a run selector. Each run retains its original case details. +func RenderComparisonHTML(runs []ComparisonRun, section template.HTML) ([]byte, error) { + if len(runs) == 0 { + return nil, errors.New("comparison requires run details") + } + reporter := &HTMLReporter{} + data, err := reporter.buildTemplateData(runs[0].Input) + if err != nil { + return nil, err + } + var payload []struct { + Label string `json:"label"` + Data json.RawMessage `json:"data"` + } + for _, run := range runs { + runData, err := reporter.buildTemplateData(run.Input) + if err != nil { + return nil, err } - return nil - }) + payload = append(payload, struct { + Label string `json:"label"` + Data json.RawMessage `json:"data"` + }{run.Label, json.RawMessage(runData.EmbeddedDataJSON)}) + } + encoded, err := json.Marshal(payload) + if err != nil { + return nil, fmt.Errorf("encode comparison runs: %w", err) + } + data.ComparisonSection = section + data.ComparisonRuns = template.JS(encoded) //nolint:gosec // json.Marshal escapes untrusted strings, including script delimiters. + var b bytes.Buffer + if err := executeHTMLReport(&b, data); err != nil { + return nil, err + } + return b.Bytes(), nil } // --------------------------------------------------------------------------- @@ -50,9 +96,11 @@ func (r *HTMLReporter) Write(_ context.Context, in Input) error { // --------------------------------------------------------------------------- type htmlReportData struct { - SkillName string - LogoDataURI template.URL - EmbeddedDataJSON template.JS + ComparisonSection template.HTML + ComparisonRuns template.JS + SkillName string + LogoDataURI template.URL + EmbeddedDataJSON template.JS } // -- Embedded JSON types for JavaScript consumption -- diff --git a/internal/report/html_comparison_test.go b/internal/report/html_comparison_test.go new file mode 100644 index 00000000..f1cd7566 --- /dev/null +++ b/internal/report/html_comparison_test.go @@ -0,0 +1,44 @@ +package report + +import ( + "context" + "html/template" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/alibaba/skill-up/internal/judge" +) + +func TestComparisonExtendsOriginalCaseDetails(t *testing.T) { + t.Parallel() + input := Input{SkillName: "example", CaseResults: []CaseResult{ + {CaseID: "shared", Status: judge.StatusPass, Configuration: "with_skill", Prompt: "original prompt", Response: "original response"}, + {CaseID: "shared", Status: judge.StatusFail, Configuration: "without_skill", Response: "reference response"}, + }} + runs := []ComparisonRun{{Label: "first", Input: input}, {Label: "", Input: input}} + output, err := RenderComparisonHTML(runs, template.HTML("

Comparison marker

")) + if err != nil { + t.Fatal(err) + } + for _, want := range []string{"Comparison marker", "comparison-run", "original prompt", "original response", "reference response", "Formal Grades", "Your Feedback", "panel-benchmark"} { + if !strings.Contains(string(output), want) { + t.Errorf("original report feature missing: %s", want) + } + } + if strings.Contains(string(output), " From d644b947132e51822d96d64858972169d8870a08 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E8=B5=B5=E8=90=8D?= Date: Thu, 29 Oct 2026 06:47:28 +0000 Subject: [PATCH 5/9] fix(compare): preserve input paths and align existing report styling --- CHANGELOG.md | 2 +- docs/guide/cli-reference.md | 14 +++----- internal/cli/compare.go | 30 ++--------------- internal/cli/compare_test.go | 44 ++++++++----------------- internal/compare/templates/compare.html | 38 ++++++++++----------- internal/compare/templates/set.html | 18 ++++------ internal/report/templates/report.html | 17 ++++++++-- 7 files changed, 60 insertions(+), 103 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 20f0d338..5b40210d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,7 +11,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Add `skill-up compare ...` for offline comparison with self-contained HTML, shareable Markdown, text, and JSON reports, an overview and case matrix across all runs, per-candidate baseline - comparisons, directory-based labels with numeric ID fallback, explicit label + comparisons, input paths as default labels, explicit label overrides, and optional regression and token-growth gates. HTML extends the existing report with a comparison tab and run selector, retaining the original case details and leaving existing report output unchanged. diff --git a/docs/guide/cli-reference.md b/docs/guide/cli-reference.md index c29f398c..ccbde21d 100644 --- a/docs/guide/cli-reference.md +++ b/docs/guide/cli-reference.md @@ -303,16 +303,12 @@ All runs have a one-based numeric ID (`runs[].source.id`) assigned in input order, including the reference. These IDs are also retained in each comparison's source identity; display labels do not change them. -Default display labels use the immediate parent-directory names if those names -are nonempty and unique across **all** inputs. Relative paths are resolved from -the current working directory. If any names collide or cannot identify a -directory, **all** labels fall back to the IDs `1`, `2`, `3`, and so on. For example: +Default display labels are the input paths exactly as supplied. Relative paths +remain relative (including a leading `./` or `..`); absolute paths remain absolute. +Directory names are not shortened, and numeric IDs are not used as display labels. +The source path is shown once, rather than repeated below the label. -- `alpha/result.json beta/result.json gamma/result.json` → `alpha`, `beta`, `gamma`. -- `old.json new.json other.json` in one directory → `1`, `2`, `3`. -- `a/run/result.json b/run/result.json` → `1`, `2` (same directory basename). - -Explicit label flags override automatic display labels. The first run is never +Explicit label flags override path-based display labels. The first run is never automatically named `Baseline`, nor are later runs automatically named `Candidate`. The JSON keys `baseline` and `candidate` describe comparison roles only. diff --git a/internal/cli/compare.go b/internal/cli/compare.go index 95991d06..0a2932b8 100644 --- a/internal/cli/compare.go +++ b/internal/cli/compare.go @@ -7,7 +7,6 @@ import ( "math" "os" "path/filepath" - "strconv" "strings" "github.com/spf13/cobra" @@ -25,7 +24,7 @@ var compareCmd = &cobra.Command{ Long: `Compare existing results without running an Agent or Judge. The first input is the reference for changes and gates. Display labels use -unique parent-directory names, or numeric IDs when those names are ambiguous. +the input paths exactly as supplied, unless explicitly overridden. Examples: skill-up compare run-a/result.json run-b/result.json run-c/result.json --format html --output compare.html @@ -101,7 +100,7 @@ func runCompare(cmd *cobra.Command, args []string) error { } func loadBuildSet(cmd *cobra.Command, args []string) ([]comparepkg.RunInput, error) { - labels := automaticCompareLabels(args) + labels := append([]string(nil), args...) baselineLabel, _ := cmd.Flags().GetString("baseline-label") if baselineLabel != "" { labels[0] = baselineLabel @@ -128,31 +127,6 @@ func loadBuildSet(cmd *cobra.Command, args []string) ([]comparepkg.RunInput, err return runs, nil } -func automaticCompareLabels(paths []string) []string { - labels := make([]string, len(paths)) - seen := make(map[string]bool, len(paths)) - unique := true - for i, path := range paths { - absolute, err := filepath.Abs(path) - if err != nil { - unique = false - continue - } - name := filepath.Base(filepath.Dir(absolute)) - if name == "." || name == string(filepath.Separator) || strings.TrimSpace(name) == "" || seen[name] { - unique = false - } - seen[name] = true - labels[i] = name - } - if !unique { - for i := range labels { - labels[i] = strconv.Itoa(i + 1) - } - } - return labels -} - func writeComparison(cmd *cobra.Command, inputs []string, output string, data []byte) error { if output == "" { if _, err := cmd.OutOrStdout().Write(data); err != nil { diff --git a/internal/cli/compare_test.go b/internal/cli/compare_test.go index 353f1cba..712da1c7 100644 --- a/internal/cli/compare_test.go +++ b/internal/cli/compare_test.go @@ -6,7 +6,6 @@ import ( "math" "os" "path/filepath" - "reflect" "strconv" "strings" "testing" @@ -469,32 +468,6 @@ func assertMultipleCompareJSON(t *testing.T, data []byte) { } } -func TestAutomaticCompareLabels(t *testing.T) { - t.Parallel() - root := t.TempDir() - path := func(parts ...string) string { return filepath.Join(append([]string{root}, parts...)...) } - tests := []struct { - name string - paths []string - want []string - }{ - {"distinct directories", []string{path("alpha", "result.json"), path("beta", "result.json"), path("gamma", "result.json")}, []string{"alpha", "beta", "gamma"}}, - {"same directory", []string{path("alpha", "a.json"), path("alpha", "b.json")}, []string{"1", "2"}}, - {"one duplicate switches all labels", []string{path("alpha", "result.json"), path("beta", "a.json"), path("beta", "b.json")}, []string{"1", "2", "3"}}, - {"same directory basename", []string{path("a", "run", "result.json"), path("b", "run", "result.json")}, []string{"1", "2"}}, - {"bare filenames", []string{"old.json", "new.json", "other.json"}, []string{"1", "2", "3"}}, - {"clean relative paths", []string{filepath.Join("alpha", "nested", "..", "result.json"), filepath.Join("beta", "result.json")}, []string{"alpha", "beta"}}, - } - for _, tc := range tests { - t.Run(tc.name, func(t *testing.T) { - t.Parallel() - if got := automaticCompareLabels(tc.paths); !reflect.DeepEqual(got, tc.want) { - t.Fatalf("labels = %v, want %v", got, tc.want) - } - }) - } -} - func TestAutomaticLabelsPropagateToReportsAndRunIdentity(t *testing.T) { t.Parallel() root := t.TempDir() @@ -506,6 +479,15 @@ func TestAutomaticLabelsPropagateToReportsAndRunIdentity(t *testing.T) { } paths = append(paths, writeCompareResultJSON(t, dir, "result.json", makeReportInput(judge.StatusPass))) } + current, err := os.Getwd() + if err != nil { + t.Fatal(err) + } + relative, err := filepath.Rel(current, paths[0]) + if err != nil { + t.Fatal(err) + } + paths[0] = "." + string(filepath.Separator) + relative for _, format := range []string{"json", "html", "markdown", "text"} { cmd := newCompareCmd() var out bytes.Buffer @@ -520,19 +502,19 @@ func TestAutomaticLabelsPropagateToReportsAndRunIdentity(t *testing.T) { t.Errorf("%s still uses role-based display labels", format) } if format == jsonFormat { - assertDirectoryRunIdentity(t, out.Bytes()) + assertPathRunIdentity(t, out.Bytes(), paths) } } } -func assertDirectoryRunIdentity(t *testing.T, data []byte) { +func assertPathRunIdentity(t *testing.T, data []byte, paths []string) { t.Helper() var set comparepkg.Set if err := json.Unmarshal(data, &set); err != nil { t.Fatal(err) } - for i, name := range []string{"alpha", "beta", "gamma"} { - if source := set.Runs[i].Source; source.ID != i+1 || source.Label != name { + for i, path := range paths { + if source := set.Runs[i].Source; source.ID != i+1 || source.Label != path || source.Path != path { t.Fatalf("wrong run identity: %+v", source) } } diff --git a/internal/compare/templates/compare.html b/internal/compare/templates/compare.html index 45982014..137d5f3c 100644 --- a/internal/compare/templates/compare.html +++ b/internal/compare/templates/compare.html @@ -1,36 +1,32 @@ {{define "comparison-body"}} -
SKILL-UP / OFFLINE COMPARISON
-

{{.Result.Baseline.Label}} → {{.Result.Candidate.Label}}

-

Evaluation changes, case by case. Existing results only; no Agent or Judge calls.

-

Reference source: {{.Result.Baseline.Path}}
Compared source: {{.Result.Candidate.Path}}

-
+
{{.Result.Baseline.Label}} → {{.Result.Candidate.Label}}
-
{{.Result.Coverage.Shared}}
Shared cases
-
{{len .Result.Cases.Regressed}}
Regressed
-
{{len .Result.Cases.Fixed}}
Fixed
-
{{.Result.Coverage.Added}}
Added
-
{{.Result.Coverage.Removed}}
Removed
+
Shared cases
{{.Result.Coverage.Shared}}
+
Regressed
{{len .Result.Cases.Regressed}}
+
Fixed
{{len .Result.Cases.Fixed}}
+
Added
{{.Result.Coverage.Added}}
+
Removed
{{.Result.Coverage.Removed}}
-

Comparison coverage: {{.Result.Coverage.Shared}} / {{.Result.Run.Old.CaseCount}} reference cases; {{.Result.Coverage.Shared}} / {{.Result.Run.New.CaseCount}} compared cases.

-{{if or .Result.Coverage.Added .Result.Coverage.Removed}}

Different case sets: run-level deltas include unpaired cases. Only shared case IDs are classified as fixed or regressed.

{{end}} +

Comparison coverage: {{.Result.Coverage.Shared}} / {{.Result.Run.Old.CaseCount}} reference cases; {{.Result.Coverage.Shared}} / {{.Result.Run.New.CaseCount}} compared cases.

+{{if or .Result.Coverage.Added .Result.Coverage.Removed}}

Different case sets: run-level deltas include unpaired cases. Only shared case IDs are classified as fixed or regressed.

{{end}}

Run summary

- +{{range .Metrics}}{{end}}
Metric{{.Result.Baseline.Label}}{{.Result.Candidate.Label}}Change
Metric{{.Result.Baseline.Label}}{{.Result.Candidate.Label}}Change
{{.Name}}{{.Old}}{{.New}}{{.Delta}}
-Pass rates use one primary result per case ID. Total tokens cover all configurations; input/output tokens cover primary cases. Pass-rate changes are percentage points (pp). Missing measurements are unknown. +
Pass rates use one primary result per case ID. Total tokens cover all configurations; input/output tokens cover primary cases. Pass-rate changes are percentage points (pp). Missing measurements are unknown.

Metadata differences

- +{{range .Metadata}}{{end}}
Field{{.Result.Baseline.Label}}{{.Result.Candidate.Label}}Change
Field{{.Result.Baseline.Label}}{{.Result.Candidate.Label}}Change
{{.Name}}{{.Old}}{{.New}}{{.Delta}}
-

Missing provenance is unknown. Matching case IDs do not prove identical prompts, suites, Skill revisions, or judge/rubric definitions. Scores are not regraded.

+

Missing provenance is unknown. Matching case IDs do not prove identical prompts, suites, Skill revisions, or judge/rubric definitions. Scores are not regraded.

Case transitions

-
-{{range .Groups}}
{{.Name}} ({{len .Cases}}) +
+{{range .Groups}}
{{.Name}} ({{len .Cases}}) {{if .Cases}}
-{{range .Cases}}{{end}}
Case IDReference titleCompared titleReference statusCompared statusReference configurationCompared configurationPrompt
{{.CaseID}}{{if .OldTitle}}{{.OldTitle}}{{else}}—{{end}}{{if .NewTitle}}{{.NewTitle}}{{else}}—{{end}}{{if .OldStatus}}{{.OldStatus}}{{else}}—{{end}}{{if .NewStatus}}{{.NewStatus}}{{else}}—{{end}}{{if .OldConfiguration}}{{.OldConfiguration}}{{else}}—{{end}}{{if .NewConfiguration}}{{.NewConfiguration}}{{else}}—{{end}}{{promptChange .PromptChanged}}
{{else}}

None.

{{end}}
{{end}} +{{range .Cases}}{{.CaseID}}{{if .OldTitle}}{{.OldTitle}}{{else}}—{{end}}{{if .NewTitle}}{{.NewTitle}}{{else}}—{{end}}{{if .OldStatus}}{{.OldStatus}}{{else}}—{{end}}{{if .NewStatus}}{{.NewStatus}}{{else}}—{{end}}{{if .OldConfiguration}}{{.OldConfiguration}}{{else}}—{{end}}{{if .NewConfiguration}}{{.NewConfiguration}}{{else}}—{{end}}{{promptChange .PromptChanged}}{{end}}
{{else}}

None.

{{end}}{{end}}

Gates: {{if .Result.Gates.Passed}}passed{{else}}failed{{end}}

-{{if .Result.Gates.Passed}}

Configured checks passed. No checks are enforced unless requested.

{{else}}
    {{range .Result.Gates.Failures}}
  • {{.}}
    • {{end}}
{{end}}
-
Generated by skill-up compare. Self-contained report · no external assets · source artifacts preserved.
+
{{if .Result.Gates.Passed}}

Configured checks passed. No checks are enforced unless requested.

{{else}}
    {{range .Result.Gates.Failures}}
  • {{.}}
    • {{end}}
{{end}}
+ {{end}} diff --git a/internal/compare/templates/set.html b/internal/compare/templates/set.html index ad851322..fa26967d 100644 --- a/internal/compare/templates/set.html +++ b/internal/compare/templates/set.html @@ -1,17 +1,13 @@ -{{define "set"}}
SKILL-UP / OFFLINE COMPARISON
-

Evaluation comparison · {{len .Set.Runs}} runs

-

Reference for changes: {{.Set.Baseline.Label}}. Every other run is compared independently against this reference.

-

Existing results only; no Agent or Judge calls. Case outcomes align by stable case ID; absent cases are not paired samples.

- -
+{{define "set"}}
+
Comparison · {{len .Set.Runs}} runs

Reference for changes: {{.Set.Baseline.Label}}

Every other run is compared independently against this reference.

Run overview

-{{range .Set.Runs}}{{end}} +{{range .Set.Runs}}{{end}}{{range .Metrics}}{{range .Values}}{{end}}{{end}}
Metric{{.Source.Label}}{{.Source.Path}}
Metric{{.Source.Label}}
{{.Name}}{{.}}
-

Case matrix

Absent means that the case was not present in that run. Hover over an outcome to see its source title.

-
{{range .Set.Runs}}{{end}} +

Case matrix

Absent means that the case was not present in that run. Hover over an outcome to see its source title.

+
Case ID{{.Source.Label}}
{{range .Set.Runs}}{{end}}{{range .Set.CaseMatrix}}{{range .Outcomes}}{{end}}{{end}}
Case ID{{.Source.Label}}
{{.CaseID}}{{if .}}{{.Status}}{{if .Configuration}}
{{.Configuration}}{{end}}{{else}}absent{{end}}

Overall gates: {{if .Set.Gates.Passed}}passed{{else}}failed{{end}}

-{{if .Set.Gates.Passed}}

Configured checks passed for every compared run. No checks are enforced unless requested.

{{else}}
    {{range .Set.Gates.Failures}}
  • {{.}}
    • {{end}}
{{end}}
+
{{if .Set.Gates.Passed}}

Configured checks passed for every compared run. No checks are enforced unless requested.

{{else}}
    {{range .Set.Gates.Failures}}
  • {{.}}
    • {{end}}
{{end}}
{{range $index, $view := .Comparisons}}
{{template "comparison-body" $view}}
{{end}} -{{end}} +
{{end}} diff --git a/internal/report/templates/report.html b/internal/report/templates/report.html index e01a1847..6e924c65 100644 --- a/internal/report/templates/report.html +++ b/internal/report/templates/report.html @@ -167,6 +167,19 @@ .toast { position: fixed; bottom: 5rem; left: 50%; transform: translateX(-50%); background: var(--header-bg); color: var(--header-text); padding: 0.625rem 1.25rem; border-radius: var(--radius); font-size: 0.875rem; opacity: 0; transition: opacity 0.3s; pointer-events: none; z-index: 200; } .toast.visible { opacity: 1; } + {{if .ComparisonRuns}} + #panel-comparison .section { margin-bottom: 1.25rem; } + #panel-comparison .section-header { margin: 0; font-size: .75rem; } + #panel-comparison .section-body { font-size: .8125rem; line-height: 1.6; } + #panel-comparison .comparison-path { text-transform: none; letter-spacing: normal; overflow-wrap: anywhere; min-width: 0; } + #panel-comparison th.comparison-path { max-width: 22rem; } + #panel-comparison .comparison-note { color: var(--text-muted); font-size: .75rem; } + #panel-comparison .comparison-toolbar { display: flex; align-items: center; gap: .75rem; flex-wrap: wrap; margin-top: .75rem; } + #panel-comparison .comparison-toolbar a { text-decoration: none; } + #panel-comparison .summary-bar { margin-bottom: 1.25rem; } + #comparison-run { min-width: 0; max-width: 100%; flex: 1; text-transform: none; } + .comparison-run-control { display: flex; align-items: center; gap: .75rem; font-size: .8125rem; } + {{end}} @@ -182,7 +195,7 @@

Eval Report:

- {{if .ComparisonRuns}}
{{end}} + {{if .ComparisonRuns}}
{{end}}
- {{if .ComparisonRuns}}
{{.ComparisonSection}}
{{end}} + {{if .ComparisonRuns}}
{{.ComparisonSection}}
{{end}}
Pass rates use one primary result per case ID. Total tokens cover all configurations; input/output tokens cover primary cases. Pass-rate changes are percentage points (pp). Missing measurements are unknown.

Metadata differences

-{{range .Metadata}}{{end}}
Field{{.Result.Baseline.Label}}{{.Result.Candidate.Label}}Change
{{.Name}}{{.Old}}{{.New}}{{.Delta}}
+{{range .Metadata}}{{.Name}}{{.Old}}{{.New}}{{.Delta}}{{end}}

Missing provenance is unknown. Matching case IDs do not prove identical prompts, suites, Skill revisions, or judge/rubric definitions. Scores are not regraded.

Case transitions

@@ -24,9 +24,9 @@
{{range .Groups}}
{{.Name}} ({{len .Cases}}) {{if .Cases}}
-{{range .Cases}}{{end}}
Case IDReference titleCompared titleReference statusCompared statusReference configurationCompared configurationPrompt
{{.CaseID}}{{if .OldTitle}}{{.OldTitle}}{{else}}—{{end}}{{if .NewTitle}}{{.NewTitle}}{{else}}—{{end}}{{if .OldStatus}}{{.OldStatus}}{{else}}—{{end}}{{if .NewStatus}}{{.NewStatus}}{{else}}—{{end}}{{if .OldConfiguration}}{{.OldConfiguration}}{{else}}—{{end}}{{if .NewConfiguration}}{{.NewConfiguration}}{{else}}—{{end}}{{promptChange .PromptChanged}}
{{else}}

None.

{{end}}
{{end}} +{{range .Cases}}{{.CaseID}}{{if .OldTitle}}{{.OldTitle}}{{else}}—{{end}}{{if .NewTitle}}{{.NewTitle}}{{else}}—{{end}}{{if .OldStatus}}{{.OldStatus}}{{else}}absent{{end}}{{if .NewStatus}}{{.NewStatus}}{{else}}absent{{end}}{{if .OldConfiguration}}{{.OldConfiguration}}{{else}}—{{end}}{{if .NewConfiguration}}{{.NewConfiguration}}{{else}}—{{end}}{{promptChange .PromptChanged}}{{end}}{{else}}

None.

{{end}}{{end}}
-{{if not .Interactive}}

Gates: {{if .Result.Gates.Passed}}passed{{else}}failed{{end}}

+{{if not .Interactive}}

Gates: {{if .Result.Gates.Passed}}passed{{else}}failed{{end}}

{{if .Result.Gates.Passed}}

Configured checks passed. No checks are enforced unless requested.

{{else}}
    {{range .Result.Gates.Failures}}
  • {{.}}
    • {{end}}
{{end}}
{{end}} {{end}} diff --git a/internal/compare/templates/set.html b/internal/compare/templates/set.html index 5d431d61..8601a510 100644 --- a/internal/compare/templates/set.html +++ b/internal/compare/templates/set.html @@ -2,12 +2,12 @@
Comparison · {{len .Set.Runs}} runs

CI gate reference: {{.Set.Baseline.Label}}

CI gates compare other runs against this first input. The interactive pair below can use any two runs.

Run overview

{{range .Set.Runs}}{{end}} -{{range .Metrics}}{{range .Values}}{{end}}{{end}} -
Metric{{.Source.Label}}
{{.Name}}{{.}}
+{{range .Metrics}}{{$metric := .Name}}{{.Name}}{{range .Values}}{{.}}{{end}}{{end}} +

Pass rate: green = all passed, amber = partially passed, red = none passed. Change: green = higher pass rate or lower token/time usage; red = the reverse; blue = descriptive change; gray = unchanged or unknown.

Case matrix

Absent means that the case was not present in that run. Hover over an outcome to see its source title.

{{range .Set.Runs}}{{end}} -{{range .Set.CaseMatrix}}{{range .Outcomes}}{{end}}{{end}}
Case ID{{.Source.Label}}
{{.CaseID}}{{if .}}{{.Status}}{{if .Configuration}}
{{.Configuration}}{{end}}{{else}}absent{{end}}
-

Overall gates: {{if .Set.Gates.Passed}}passed{{else}}failed{{end}}

+{{range .Set.CaseMatrix}}{{.CaseID}}{{range .Outcomes}}{{if .}}{{.Status}}{{if .Configuration}}
{{.Configuration}}{{end}}{{else}}absent{{end}}{{end}}{{end}}
+

Overall gates: {{if .Set.Gates.Passed}}passed{{else}}failed{{end}}

{{if .Set.Gates.Passed}}

Configured checks passed for every compared run. No checks are enforced unless requested.

{{else}}
    {{range .Set.Gates.Failures}}
  • {{.}}
    • {{end}}
{{end}}
Compare any two results
Cases align by case ID. Each file's primary result is compared here; its internal with/without Skill details remain available under Run details.
{{range .Pairs}}
{{template "comparison-body" .View}}
{{end}} diff --git a/internal/report/templates/report.html b/internal/report/templates/report.html index bf7c0228..62b6fb48 100644 --- a/internal/report/templates/report.html +++ b/internal/report/templates/report.html @@ -176,6 +176,13 @@ #panel-comparison .section-body { font-size: .8125rem; line-height: 1.6; } #panel-comparison .comparison-path { text-transform: none; letter-spacing: normal; overflow-wrap: anywhere; min-width: 0; } #panel-comparison th.comparison-path { max-width: 22rem; } + #panel-comparison .comparison-positive { color: #047857; font-weight: 600; } + #panel-comparison .comparison-negative { color: var(--red); font-weight: 600; } + #panel-comparison .comparison-warning { color: #a16207; font-weight: 600; } + #panel-comparison .comparison-neutral { color: #64748b; } + #panel-comparison .comparison-changed { color: #1d4ed8; font-weight: 600; } + #panel-comparison .comparison-status.sb-pass { color: #047857; } + #panel-comparison .comparison-absent { color: #64748b; background: var(--bg); } #panel-comparison .comparison-note { color: var(--text-muted); font-size: .75rem; } #panel-comparison .comparison-toolbar { display: flex; align-items: center; gap: .75rem; flex-wrap: wrap; margin-top: .75rem; } #panel-comparison .comparison-toolbar a { text-decoration: none; } @@ -344,7 +351,14 @@

Review Complete

errorSection.style.display = sides.some(function(side) { return side && side.error; }) ? 'block' : 'none'; document.getElementById('error-text').innerHTML = columns(sides.map(function(side) { return esc(side ? (side.error || 'No recorded error') : 'Case absent from this run.'); })); var badge = document.getElementById('case-badge'); badge.className = 'status-badge'; - badge.textContent = sides.map(function(side) { return side ? side.status : 'absent'; }).join(' → '); + badge.replaceChildren(); + sides.forEach(function(side, index) { + if (index) badge.appendChild(document.createTextNode(' → ')); + var status = document.createElement('span'); + var value = side ? side.status : 'absent'; + status.className = 'status-badge ' + (side ? 'sb-' + value.toLowerCase() + ' comparison-status' : 'comparison-absent'); + status.textContent = value; badge.appendChild(status); + }); document.getElementById('case-header-metrics').textContent = 'Cross-run comparison of primary results'; } From a2d38270c910ae8a8413301ec4498c4ae6cc65b9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E8=B5=B5=E8=90=8D?= Date: Thu, 29 Oct 2026 09:51:49 +0000 Subject: [PATCH 8/9] fix(report): clarify primary results and improve multi-run layout --- CHANGELOG.md | 3 ++ docs/guide/cli-reference.md | 13 ++++++++- internal/compare/templates/set.html | 6 ++-- internal/report/html.go | 17 ++++++----- internal/report/html_comparison_test.go | 38 +++++++++++++++++++++++++ internal/report/templates/report.html | 19 +++++++++++-- 6 files changed, 82 insertions(+), 14 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index fad34b98..5a014986 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,6 +18,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 selecting any two input files and reviewing aligned prompts, responses, and grades side by side without changing CI gate references. Color-code pass rates, case statuses, and metric changes, with cost/time direction handled separately. + Document primary-result selection for cross-file comparisons; use canonical + primary cases in the interactive viewer, and keep dropdowns and multi-run + tables readable with padding, wrapping, and horizontal scrolling. ### Changed - Strengthen `skill-upper` evolution with original/candidate comparisons, diff --git a/docs/guide/cli-reference.md b/docs/guide/cli-reference.md index a78e6989..2db481e6 100644 --- a/docs/guide/cli-reference.md +++ b/docs/guide/cli-reference.md @@ -333,7 +333,18 @@ JavaScript. Markdown includes tables suitable for GitHub PR descriptions. Source result artifacts are preserved; output may not overwrite any input. Cases align by case ID using the existing primary-result rule: prefer -`with_skill`, then a non-baseline configuration, falling back to `without_skill`. +`with_skill`, then a non-baseline configuration, falling back to `without_skill` +only when no other result for that case is available. In the usual with/without +benchmark, cross-file comparison therefore compares **each revision's with_skill +result**, not all variants against one another. This same selection rule drives +metrics, the case matrix, transitions, and the interactive side-by-side viewer. +Each file's internal with/without binding remains in Run details. The selected +configuration is exposed in the matrix and transition tables. + +For more than three inputs, overview/matrix columns retain readable widths and +scroll horizontally with the first column pinned. Selectors wrap to new rows +on narrow screens and reserve spacing for the dropdown arrow; the pair viewer +always compares two selected inputs, regardless of the total input count. Benchmark reference entries do not double-count case counts or pass rates. Configuration identity is retained in case tables. Shared cases flag prompt changes when both prompts were recorded; otherwise the prompt comparison is unknown. diff --git a/internal/compare/templates/set.html b/internal/compare/templates/set.html index 8601a510..ba3742ff 100644 --- a/internal/compare/templates/set.html +++ b/internal/compare/templates/set.html @@ -1,15 +1,15 @@ {{define "set"}}
Comparison · {{len .Set.Runs}} runs

CI gate reference: {{.Set.Baseline.Label}}

CI gates compare other runs against this first input. The interactive pair below can use any two runs.

-

Run overview

+

Run overview

Multiple runs keep readable column widths; scroll horizontally to see additional files. The metric/case column stays visible.
{{range .Set.Runs}}{{end}}{{range .Metrics}}{{$metric := .Name}}{{range .Values}}{{end}}{{end}}
Metric{{.Source.Label}}
{{.Name}}{{.}}

Pass rate: green = all passed, amber = partially passed, red = none passed. Change: green = higher pass rate or lower token/time usage; red = the reverse; blue = descriptive change; gray = unchanged or unknown.

Case matrix

Absent means that the case was not present in that run. Hover over an outcome to see its source title.

-
{{range .Set.Runs}}{{end}} +
Case ID{{.Source.Label}}
{{range .Set.Runs}}{{end}}{{range .Set.CaseMatrix}}{{range .Outcomes}}{{end}}{{end}}
Case ID{{.Source.Label}}
{{.CaseID}}{{if .}}{{.Status}}{{if .Configuration}}
{{.Configuration}}{{end}}{{else}}absent{{end}}

Overall gates: {{if .Set.Gates.Passed}}passed{{else}}failed{{end}}

{{if .Set.Gates.Passed}}

Configured checks passed for every compared run. No checks are enforced unless requested.

{{else}}
    {{range .Set.Gates.Failures}}
  • {{.}}
    • {{end}}
{{end}}
-
Compare any two results
Cases align by case ID. Each file's primary result is compared here; its internal with/without Skill details remain available under Run details.
+
Compare any two results
Cases align by case ID. For each case ID in each file: use with_skill if available, otherwise a non-baseline result, and fall back to without_skill only when it is the only available result. This is not a cross-file comparison of every configuration. Internal with/without Skill details remain under Run details.
{{range .Pairs}}
{{template "comparison-body" .View}}
{{end}}
{{end}} diff --git a/internal/report/html.go b/internal/report/html.go index acc20035..34e6c7c9 100644 --- a/internal/report/html.go +++ b/internal/report/html.go @@ -64,19 +64,22 @@ func RenderComparisonHTML(runs []ComparisonRun, section template.HTML) ([]byte, if err != nil { return nil, err } - var payload []struct { - Label string `json:"label"` - Data json.RawMessage `json:"data"` + type runPayload struct { + Label string `json:"label"` + Data json.RawMessage `json:"data"` + PrimaryCases []embeddedCase `json:"primary_cases"` } + var payload []runPayload for _, run := range runs { runData, err := reporter.buildTemplateData(run.Input) if err != nil { return nil, err } - payload = append(payload, struct { - Label string `json:"label"` - Data json.RawMessage `json:"data"` - }{run.Label, json.RawMessage(runData.EmbeddedDataJSON)}) + primary := make([]embeddedCase, 0) + for _, cr := range run.Input.PrimaryCaseResults() { + primary = append(primary, caseResultToEmbeddedCase(cr)) + } + payload = append(payload, runPayload{Label: run.Label, Data: json.RawMessage(runData.EmbeddedDataJSON), PrimaryCases: primary}) } encoded, err := json.Marshal(payload) if err != nil { diff --git a/internal/report/html_comparison_test.go b/internal/report/html_comparison_test.go index f1cd7566..618ee765 100644 --- a/internal/report/html_comparison_test.go +++ b/internal/report/html_comparison_test.go @@ -2,6 +2,7 @@ package report import ( "context" + "encoding/json" "html/template" "os" "path/filepath" @@ -42,3 +43,40 @@ func TestComparisonExtendsOriginalCaseDetails(t *testing.T) { t.Fatal("comparison controls leaked into legacy report") } } + +func TestCrossFileDetailsUseCanonicalPrimaryResults(t *testing.T) { + t.Parallel() + const ( + withSkill = "with_skill" + withoutSkill = "without_skill" + ) + input := Input{CaseResults: []CaseResult{ + {CaseID: "shared", Configuration: withSkill, Status: judge.StatusPass, Response: "with skill answer"}, + {CaseID: "shared", Configuration: "custom", Status: judge.StatusFail, Response: "custom answer"}, + {CaseID: "shared", Configuration: withoutSkill, Status: judge.StatusFail, Response: "without skill answer"}, + {CaseID: "fallback", Configuration: withoutSkill, Status: judge.StatusPass}, + }} + output, err := RenderComparisonHTML([]ComparisonRun{{Label: "input.json", Input: input}}, "") + if err != nil { + t.Fatal(err) + } + _, assignment, found := strings.Cut(string(output), "var comparisonRuns = ") + if !found { + t.Fatal("comparison payload missing") + } + var runs []struct { + Primary []embeddedCase `json:"primary_cases"` + } + if err := json.NewDecoder(strings.NewReader(assignment)).Decode(&runs); err != nil { + t.Fatal(err) + } + if len(runs) != 1 || len(runs[0].Primary) != 2 { + t.Fatalf("incorrect primary payload: %+v", runs) + } + if cr := runs[0].Primary[0]; cr.Configuration != withSkill || cr.Response != "with skill answer" { + t.Fatalf("interactive viewer did not prefer with_skill: %+v", cr) + } + if runs[0].Primary[1].Configuration != withoutSkill { + t.Fatal("without_skill-only fallback lost") + } +} diff --git a/internal/report/templates/report.html b/internal/report/templates/report.html index 62b6fb48..acb8910c 100644 --- a/internal/report/templates/report.html +++ b/internal/report/templates/report.html @@ -187,6 +187,18 @@ #panel-comparison .comparison-toolbar { display: flex; align-items: center; gap: .75rem; flex-wrap: wrap; margin-top: .75rem; } #panel-comparison .comparison-toolbar a { text-decoration: none; } #panel-comparison .summary-bar { margin-bottom: 1.25rem; } + #panel-comparison .comparison-toolbar > label { display: flex; flex-direction: column; gap: .5rem; flex: 1 1 20rem; min-width: 0; max-width: 100%; } + #panel-comparison .comparison-toolbar input, #panel-comparison .comparison-toolbar select { width: 100%; box-sizing: border-box; min-width: 0; } + #panel-comparison select, #comparison-run { padding: .625rem 2.75rem .625rem .875rem; line-height: 1.5; appearance: none; background-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' width='16' height='16' viewBox='0 0 24 24' fill='none' stroke='%23475569' stroke-width='2'%3E%3Cpath d='m6 9 6 6 6-6'/%3E%3C/svg%3E"); background-repeat: no-repeat; background-position: right .875rem center; background-size: 1rem; } + #panel-comparison .comparison-table-scroll { max-width: 100%; max-height: 34rem; overflow: auto; } + #panel-comparison .comparison-matrix { width: max-content; min-width: 100%; table-layout: fixed; } + #panel-comparison .comparison-matrix th, #panel-comparison .comparison-matrix td { width: 16rem; min-width: 14rem; max-width: 20rem; vertical-align: top; overflow-wrap: anywhere; } + #panel-comparison .comparison-matrix th:first-child, #panel-comparison .comparison-matrix td:first-child { position: sticky; left: 0; z-index: 1; background: var(--surface); } + #panel-comparison .comparison-matrix th:first-child { background: var(--header-bg); z-index: 2; } + @media (max-width: 600px) { + #panel-comparison .comparison-matrix th:first-child, #panel-comparison .comparison-matrix td:first-child { width: 9rem; min-width: 9rem; max-width: 9rem; } + #panel-comparison .comparison-toolbar > label { flex-basis: 100%; } + } #comparison-run { min-width: 0; max-width: 100%; flex: 1; text-transform: none; } .comparison-run-control { display: flex; align-items: center; gap: .75rem; font-size: .8125rem; } {{end}} @@ -316,9 +328,9 @@

Review Complete

document.getElementById('comparison-run').dataset.selected = String(rightIndex); pairLabels = [left.label, right.label]; pairOpen = true; DATA = JSON.parse(JSON.stringify(right.data)); - var leftCases = new Map(left.data.cases.map(function(c) { return [c.id, c]; })); - var rightCases = new Map(right.data.cases.map(function(c) { return [c.id, c]; })); - var ids = Array.from(new Set(left.data.cases.concat(right.data.cases).map(function(c) { return c.id; }))); + var leftCases = new Map(left.primary_cases.map(function(c) { return [c.id, c]; })); + var rightCases = new Map(right.primary_cases.map(function(c) { return [c.id, c]; })); + var ids = Array.from(new Set(left.primary_cases.concat(right.primary_cases).map(function(c) { return c.id; }))); DATA.cases = ids.map(function(id) { var a = leftCases.get(id), b = rightCases.get(id), c = Object.assign({}, b || a); c.baseline = null; c.turn_results = null; c.pairLeft = a; c.pairRight = b; return c; @@ -856,6 +868,7 @@

Review Complete

} document.addEventListener("keydown", function(e) { + {{if .ComparisonRuns}}if (e.target.closest("input, select, textarea, button")) return;{{end}} if (e.target.tagName === "TEXTAREA") return; if (e.key === "ArrowLeft" || e.key === "ArrowUp") { e.preventDefault(); navigate(-1); } else if (e.key === "ArrowRight" || e.key === "ArrowDown") { e.preventDefault(); navigate(1); } From babe52d246eaa44a3cc60c291b4df4c273a42400 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E8=B5=B5=E8=90=8D?= Date: Thu, 29 Oct 2026 10:01:02 +0000 Subject: [PATCH 9/9] fix(report): streamline comparison controls and contextual help --- CHANGELOG.md | 4 +++- docs/guide/cli-reference.md | 6 +++++- internal/compare/templates/compare.html | 10 +++++----- internal/compare/templates/set.html | 15 +++++++-------- internal/report/templates/report.html | 18 +++++++++++++++++- 5 files changed, 37 insertions(+), 16 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5a014986..03edf376 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -20,7 +20,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 case statuses, and metric changes, with cost/time direction handled separately. Document primary-result selection for cross-file comparisons; use canonical primary cases in the interactive viewer, and keep dropdowns and multi-run - tables readable with padding, wrapping, and horizontal scrolling. + tables readable with padding, wrapping, and horizontal scrolling. Align the + Compare controls, collapse secondary explanations, and omit empty Benchmark + tabs in comparison mode. ### Changed - Strengthen `skill-upper` evolution with original/candidate comparisons, diff --git a/docs/guide/cli-reference.md b/docs/guide/cli-reference.md index 2db481e6..59a9f144 100644 --- a/docs/guide/cli-reference.md +++ b/docs/guide/cli-reference.md @@ -338,7 +338,11 @@ only when no other result for that case is available. In the usual with/without benchmark, cross-file comparison therefore compares **each revision's with_skill result**, not all variants against one another. This same selection rule drives metrics, the case matrix, transitions, and the interactive side-by-side viewer. -Each file's internal with/without binding remains in Run details. The selected +Each file's internal with/without binding remains in Run details. The page keeps +this rule in a collapsed explanation; the full selection semantics are documented +here. Benchmark is the original single-run statistics view (including with/without +deltas when recorded), and is shown in comparison mode only when the selected +run actually contains benchmark data. The selected configuration is exposed in the matrix and transition tables. For more than three inputs, overview/matrix columns retain readable widths and diff --git a/internal/compare/templates/compare.html b/internal/compare/templates/compare.html index 501db205..48617c9c 100644 --- a/internal/compare/templates/compare.html +++ b/internal/compare/templates/compare.html @@ -1,5 +1,5 @@ {{define "comparison-body"}} -
{{.Result.Baseline.Label}} → {{.Result.Candidate.Label}}
+
Shared cases
{{.Result.Coverage.Shared}}
Regressed
{{len .Result.Cases.Regressed}}
@@ -10,14 +10,14 @@

Comparison coverage: {{.Result.Coverage.Shared}} / {{.Result.Run.Old.CaseCount}} reference cases; {{.Result.Coverage.Shared}} / {{.Result.Run.New.CaseCount}} compared cases.

{{if or .Result.Coverage.Added .Result.Coverage.Removed}}

Different case sets: run-level deltas include unpaired cases. Only shared case IDs are classified as fixed or regressed.

{{end}}

Run summary

- +{{range .Metrics}}{{end}}
Metric{{.Result.Baseline.Label}}{{.Result.Candidate.Label}}Change
Metric{{.Result.Baseline.Label}}{{.Result.Candidate.Label}}Change
{{.Name}}{{.Old}}{{.New}}{{.Delta}}
-
Pass rates use one primary result per case ID. Total tokens cover all configurations; input/output tokens cover primary cases. Pass-rate changes are percentage points (pp). Missing measurements are unknown.
+
Metric definitions

Pass rates use one primary result per case ID. Total tokens cover all configurations; input/output tokens cover primary cases. Pass-rate changes are percentage points (pp). Missing measurements are unknown.

Metadata differences

- +{{range .Metadata}}{{end}}
Field{{.Result.Baseline.Label}}{{.Result.Candidate.Label}}Change
Field{{.Result.Baseline.Label}}{{.Result.Candidate.Label}}Change
{{.Name}}{{.Old}}{{.New}}{{.Delta}}
-

Missing provenance is unknown. Matching case IDs do not prove identical prompts, suites, Skill revisions, or judge/rubric definitions. Scores are not regraded.

+
Provenance and grading

Missing provenance is unknown. Matching case IDs do not prove identical prompts, suites, Skill revisions, or judge/rubric definitions. Scores are not regraded.

Case transitions

diff --git a/internal/compare/templates/set.html b/internal/compare/templates/set.html index ba3742ff..0b171527 100644 --- a/internal/compare/templates/set.html +++ b/internal/compare/templates/set.html @@ -1,15 +1,14 @@ {{define "set"}}
-
Comparison · {{len .Set.Runs}} runs

CI gate reference: {{.Set.Baseline.Label}}

CI gates compare other runs against this first input. The interactive pair below can use any two runs.

-

Run overview

Multiple runs keep readable column widths; scroll horizontally to see additional files. The metric/case column stays visible.
+

Run overview · {{len .Set.Runs}} runs

{{range .Set.Runs}}{{end}}{{range .Metrics}}{{$metric := .Name}}{{range .Values}}{{end}}{{end}} -
Metric{{.Source.Label}}
{{.Name}}{{.}}

Pass rate: green = all passed, amber = partially passed, red = none passed. Change: green = higher pass rate or lower token/time usage; red = the reverse; blue = descriptive change; gray = unchanged or unknown.

-

Case matrix

Absent means that the case was not present in that run. Hover over an outcome to see its source title.

+
Color key

Pass rate: green = all passed, amber = partially passed, red = none passed. Change: green = higher pass rate or lower token/time usage; red = the reverse; blue = descriptive change; gray = unchanged or unknown.

+

Case matrix

{{range .Set.Runs}}{{end}} -{{range .Set.CaseMatrix}}{{range .Outcomes}}{{end}}{{end}}
Case ID{{.Source.Label}}
{{.CaseID}}{{if .}}{{.Status}}{{if .Configuration}}
{{.Configuration}}{{end}}{{else}}absent{{end}}
-

Overall gates: {{if .Set.Gates.Passed}}passed{{else}}failed{{end}}

-
{{if .Set.Gates.Passed}}

Configured checks passed for every compared run. No checks are enforced unless requested.

{{else}}
    {{range .Set.Gates.Failures}}
  • {{.}}
    • {{end}}
{{end}}
-
Compare any two results
Cases align by case ID. For each case ID in each file: use with_skill if available, otherwise a non-baseline result, and fall back to without_skill only when it is the only available result. This is not a cross-file comparison of every configuration. Internal with/without Skill details remain under Run details.
+{{range .Set.CaseMatrix}}{{.CaseID}}{{range .Outcomes}}{{if .}}{{.Status}}{{if .Configuration}}
{{.Configuration}}{{end}}{{else}}absent{{end}}{{end}}{{end}}
+
CI gates: {{if .Set.Gates.Passed}}passed{{else}}failed{{end}} +

Reference: {{.Set.Baseline.Label}}

{{if .Set.Gates.Passed}}

Configured checks passed. Gates are opt-in.

{{else}}
    {{range .Set.Gates.Failures}}
  • {{.}}
    • {{end}}
{{end}}
+
Compare results
Primary results by case ID · with_skill preferred

Cases align by case ID. For each case ID in each file: use with_skill if available, otherwise a non-baseline result, and fall back to without_skill only when it is the only available result. This is not a cross-file comparison of every configuration. Internal with/without Skill details remain under Run details.

{{range .Pairs}}
{{template "comparison-body" .View}}
{{end}} {{end}} diff --git a/internal/report/templates/report.html b/internal/report/templates/report.html index acb8910c..055500d6 100644 --- a/internal/report/templates/report.html +++ b/internal/report/templates/report.html @@ -199,6 +199,19 @@ #panel-comparison .comparison-matrix th:first-child, #panel-comparison .comparison-matrix td:first-child { width: 9rem; min-width: 9rem; max-width: 9rem; } #panel-comparison .comparison-toolbar > label { flex-basis: 100%; } } + #panel-comparison .comparison-help { font-size: .75rem; color: var(--text-muted); } + #panel-comparison .comparison-help summary { cursor: pointer; } + #panel-comparison .comparison-help p { margin: .5rem 0 0; } + #panel-comparison .comparison-pair-controls { display: grid; grid-template-columns: minmax(0, 1fr) minmax(0, 1fr) auto; align-items: end; margin-top: 0; } + #panel-comparison .comparison-pair-controls select, #panel-comparison .comparison-pair-controls button { height: 2.75rem; box-sizing: border-box; line-height: 1.5; } + #panel-comparison .comparison-pair-controls button { align-self: end; } + @media (max-width: 900px) { + #panel-comparison .comparison-pair-controls { grid-template-columns: minmax(0, 1fr) minmax(0, 1fr); } + #panel-comparison .comparison-pair-controls button { grid-column: 1 / -1; width: 100%; } + } + @media (max-width: 600px) { + #panel-comparison .comparison-pair-controls { grid-template-columns: minmax(0, 1fr); } + } #comparison-run { min-width: 0; max-width: 100%; flex: 1; text-transform: none; } .comparison-run-control { display: flex; align-items: center; gap: .75rem; font-size: .8125rem; } {{end}} @@ -429,7 +442,10 @@

Review Complete

renderBenchmark(); } - {{if .ComparisonRuns}}document.getElementById('view-tabs').style.display = 'flex';{{end}} + {{if .ComparisonRuns}}document.getElementById('view-tabs').style.display = 'flex'; + var benchmarkTab = document.querySelector('[data-view="benchmark"]'); + benchmarkTab.hidden = !DATA.benchmark; + benchmarkTab.title = 'Statistics for this run, including with/without Skill when recorded';{{end}} renderCaseList(); {{if .ComparisonRuns}}showCase(0);{{else}}