diff --git a/README.md b/README.md index 7bc1b12..86c5627 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # SNodeC control repository -This repository owns the organization catalog and the [profile](profile/README.md) generated from it. Brand tokens define the shared accents, contrast pairs and glyphs. The shared Verify workflow provides reusable, manifest-driven verification. The navigation plan, page-header generator keep documentation paths, generated navigation and tracked-page reachability reviewable from one control point. +This repository owns the organization catalog and the [profile](profile/README.md) generated from it. Brand tokens define the shared accents, contrast pairs and glyphs. The shared Verify workflow provides reusable, manifest-driven verification. The navigation plan, page-header generator and optional `docs-audit` command keep documentation paths, generated navigation and tracked-page reachability reviewable from one control point. ## Documentation @@ -106,11 +106,11 @@ Whitespace checks run against the PR base or previous push commit (empty tree on the initial commit). An independent job also checks changed C/C++ and CMake files against existing repository formatting policies. Documentation and presentation assets are not verification gates. It neither invents a style where none exists nor performs unreliable external-URL probes. Failures are collected rather than stopping at the first file. This job still runs when compilation/tests fail. -### Verify releases and caller pins +### Shared Verify updates -A changed `verify.yml` ships at a new `vX.Y` tag after review and merge. Existing tags are never moved. The current release is `v1.3`, commit `465c7e9398b1c1ab294967153bf855af52b84082`. Callers pin the full release commit SHA with a `# vX.Y` comment; they pass no secrets unless Verify actually reads one (currently none). Product CI remains `workflow_dispatch`-only by owner decision, independent of the existing version-tag package pipelines. +The five product callers and `templates/ci.yml` use `verify.yml@main`, following the API-verified default branch of this control repository. A merged shared-workflow change is used by their next manually started CI runs; it does not start those runs. Callers pass no secrets unless Verify actually reads one (currently none). Product CI remains `workflow_dispatch`-only, independent of version-tag package pipelines. -Every Verify release updates `templates/ci.yml` and all callers to the new tag's full commit SHA in the same release step, once the new tag exists. The template is the canonical onboarding example, not a mutable-tag shortcut. The documented release step keeps the template current without a network-dependent newest-tag test. +Existing tags are historical snapshots and are never moved. Page-header, menu and profile generation are separate from build CI; documentation changes require neither a Verify release nor caller updates. `docs-audit` is optional local/PR review tooling and is never a build, signing or publication gate. Tests execute the actual inline workflow scripts, including ordered source builds and a real CMake/CTest/staged-install consumer, Python commands and @@ -121,4 +121,4 @@ Repository README frames and generated navigation are governed by the [navigatio Historical rollout and design records live only in [history](history/README.md); they are not current release policy. See also [brand values](brand/README.md), [contributing](CONTRIBUTING.md) and [security](SECURITY.md). -Verify `v1.3` removes documentation enforcement. The canonical template and existing callers pin its full commit SHA. Existing `v1.0`, `v1.1` and `v1.2` tags remain unchanged. +Verify no longer enforces documentation. Existing `v1.0`–`v1.3` tags remain unchanged. diff --git a/navigation/README.md b/navigation/README.md index 606860d..8efdda3 100644 --- a/navigation/README.md +++ b/navigation/README.md @@ -7,11 +7,23 @@ python tools/snodec_sync page-headers --repo /path/to/checkout python tools/snodec_sync page-headers --repo /path/to/checkout --check ``` -Apply the plan's `git mv` and declared deletions before generating. `page-headers` does not move/delete authored pages or rewrite unrelated links. It fails if required moved pages are absent or declared deletions remain; `--check` reports drift without writing. For DistributionPackages publication sources, select `site/` to use the Packages plan; the DistributionPackages root uses its own plan. Missing generated status files are supplied by the publisher. The generator installs packages, reads secrets, calls the network or publishes anything. +Apply the plan's `git mv` and declared deletions before generating. `page-headers` does not move/delete authored pages or rewrite unrelated links. It fails if required moved pages are absent or declared deletions remain; `--check` reports drift without writing. For DistributionPackages publication sources, select `site/` to use the Packages plan; the DistributionPackages root uses its own plan. Missing generated status files are supplied by the publisher. The generator does not install packages, read secrets, call the network or publish anything. ## Documentation ownership -Documentation edits are not CI reachability, link, anchor or generated-output gates. The former docs-audit command, its exemptions and its tests have been removed. Use the page-header generator when updating shared navigation; it is not a prerequisite for code verification. +Documentation review is separate from build verification. `docs-audit` is an optional, read-only local/PR review command; it never runs in Verify, package builds or publication. A new planned Markdown page is added to `plan.yml` in the same PR. Existing excluded media notes remain reachable but have no generated header. Narrow `exempt:` declarations identify published-output targets or vendored/generated fixture paths with a reason; they do not excuse ordinary unreachable pages. + +```sh +python tools/snodec_sync docs-audit --repo /path/to/checkout +``` + +The audit reports unplanned enrolled pages, unreachable tracked Markdown, broken local links, missing anchors and generated blocks. It also reports image buttons outside marked navigation blocks, missing parent backlinks and files above 300 KB; files above 100 KB are information only. It does not fetch remote URLs. `section_list: false` disables the generated section list where an existing project table is the section's navigation authority. + +## Design rules + +Button images are allowed only in the banner navigation row (or the root Organization block) and the marked title menu, including the organization area menu. Everything below that menu is a normal text link, including route choices. Wrap title menus in `` / ``. + +Tables grow by rows: one thing per row. A growing project set gets another row or another page, never another project column or a list of projects within a cell. ## Shared rendering @@ -22,10 +34,10 @@ header = render_header('Packages', 'status/snodec.md') The read-only library accepts a catalog name or checkout path. Future PROJECTS-driven status pages inherit their declared section/sibling Up policy; they need no additional project registry. Other undeclared presentation pages fail. Runtime uses Python's standard library and PyYAML only. The checked-in marks and Octicons are inputs, not runtime downloads. -The generator replaces marked headers, preserves historic anchors outside the block and generates section lists from actual H1s in plan order. Root READMEs and the profile get no page headers. Banner plus button row stays unchanged: ↑ parent on the left; Organization first in HTML, then Repository, both right-aligned. The Organization button is labelled Organization. Buttons stay 160 px wide. The existing wider native/OpenWrt route buttons and native wrapping remain unchanged. Structural menus are arrow-free; onward/up-menu arrows retain their approved geometry. +The generator replaces marked headers, preserves historic anchors outside the block and generates section lists from actual H1s in plan order. Root READMEs and the profile get no page headers. Banner plus button row stays unchanged: ↑ parent on the left; Organization first in HTML, then Repository, both right-aligned. The Organization button is labelled Organization. Buttons stay 160 px wide. Native wrapping remains unchanged. Structural title menus are arrow-free; onward and route links are ordinary text. Desktop banners remain 1280×56. At viewport widths up to 640 px, `` selects the existing 480×72 compact SVG, with the same mark, palette and typography. Phone stacking of Organization above Repository is accepted. No CSS workaround or alternate renderer is introduced. DistributionPackages reuses the authoritative Packages gear and infrastructure accent instead of duplicating its graphic. ## Tests and releases -Run `python -m unittest discover -s tests -v` for generator and workflow behavior. Presentation goldens and authored-document drift gates are not maintained. Existing release tags remain unchanged; a changed Verify workflow requires a new approved release and caller pins. +Run `python -m unittest discover -s tests -v` for generator and workflow behavior. Presentation goldens and authored-document drift gates are not maintained. Existing release tags remain unchanged. Verify callers and the template follow `@main` by owner decision; documentation tooling does not require a Verify release. diff --git a/navigation/plan.yml b/navigation/plan.yml index 3b114f2..e3a2052 100644 --- a/navigation/plan.yml +++ b/navigation/plan.yml @@ -139,8 +139,8 @@ snode.c: kind: code title: Express Compatibility Suite (Node.js/Express v4 vs SNode.C/Express) label: Tests - up: README.md - up_label: SNode.C + up: src/apps/README.md + up_label: Examples - old: src/tools/snodec-control/README.md new: src/tools/snodec-control/README.md kind: code @@ -167,8 +167,8 @@ snode.c: kind: code title: Legacy Stream Component Test Coverage label: Tests - up: README.md - up_label: SNode.C + up: docs/guides/README.md + up_label: Guides mqttsuite: display: MQTTSuite source: mqttsuite @@ -881,6 +881,7 @@ Packages: - old: status/README.md new: status/README.md kind: doc + section_list: false title: (generated) label: Package status up: README.md @@ -915,22 +916,33 @@ Packages: remove: true # Visible-label shortening required by the approved 160px menu rule. # Destinations, tooltips and alternative text remain unchanged. -menu_labels: - Native client & server: Client/server - SSE & WebSocket: SSE/WebSocket - Subprotocol source: Subprotocol - Signed packages: Packages - Build from source: From source - Mapping examples: Examples - Mapping schema: Map schema - MQTTStore guide: Store guide - Projection schema: Projection - Grouping & lifecycle: Groups/lifecycle - Browser frontend: Browser - Selected packages: Selected +menu_labels: {} DistributionPackages: display: DistributionPackages + exempt: + - path: site/install/install.sh + reason: Resolves in published Packages. + - path: site/status/** + reason: Resolves in published Packages. + - path: site/keys/*.asc + reason: Resolves in published Packages. + - path: site/keys/*.pub + reason: Resolves in published Packages. + - path: site/keys/*.pem + reason: Resolves in published Packages. + - path: site/openwrt/** + reason: Resolves in published Packages. + - path: site/raspberrypios/** + reason: Resolves in published Packages. + - path: site/debian/** + reason: Resolves in published Packages. + - path: site/ubuntu/** + reason: Resolves in published Packages. + - path: site/rocky/** + reason: Resolves in published Packages. + - path: site/fedora/** + reason: Resolves in published Packages. source: DistributionPackages sections: docs: Documentation @@ -943,7 +955,7 @@ DistributionPackages: intro: Setup and maintenance of package publication. label: Documentation up: README.md - up_label: Distribution + up_label: Dist. Packages - old: docs/maintainers.md new: docs/maintainers.md kind: doc diff --git a/profile/README.md b/profile/README.md index 4bd2190..1d2c0e1 100644 --- a/profile/README.md +++ b/profile/README.md @@ -12,14 +12,16 @@ An ecosystem for building event-driven networked programs in C++ and running MQT - **Learn:** [Book companion](https://github.com/SNodeC/SNode.C-Book-Companion) and [examples](#learn) - **Install signed packages:** [Packages](https://github.com/SNodeC/Packages) · snode.c · mqttsuite +

-Framework -Suites -Applications -Learn -Community projects -Distribution & infrastructure +Framework +Suites +Applications +Learn +Community projects +Distribution & infrastructure

+ ## Framework diff --git a/profile/media/menu/applications-108.svg b/profile/media/menu/applications.svg similarity index 100% rename from profile/media/menu/applications-108.svg rename to profile/media/menu/applications.svg diff --git a/profile/media/menu/community-108.svg b/profile/media/menu/community.svg similarity index 100% rename from profile/media/menu/community-108.svg rename to profile/media/menu/community.svg diff --git a/profile/media/menu/framework-108.svg b/profile/media/menu/framework.svg similarity index 100% rename from profile/media/menu/framework-108.svg rename to profile/media/menu/framework.svg diff --git a/profile/media/menu/infrastructure-108.svg b/profile/media/menu/infrastructure.svg similarity index 100% rename from profile/media/menu/infrastructure-108.svg rename to profile/media/menu/infrastructure.svg diff --git a/profile/media/menu/learn-108.svg b/profile/media/menu/learn.svg similarity index 100% rename from profile/media/menu/learn-108.svg rename to profile/media/menu/learn.svg diff --git a/profile/media/menu/suites-108.svg b/profile/media/menu/suites.svg similarity index 100% rename from profile/media/menu/suites-108.svg rename to profile/media/menu/suites.svg diff --git a/templates/ci.yml b/templates/ci.yml index bc50fd9..dc5ec0d 100644 --- a/templates/ci.yml +++ b/templates/ci.yml @@ -3,4 +3,4 @@ on: [workflow_dispatch] permissions: { contents: read } jobs: snodec: - uses: SNodeC/.github/.github/workflows/verify.yml@465c7e9398b1c1ab294967153bf855af52b84082 # v1.3 + uses: SNodeC/.github/.github/workflows/verify.yml@main diff --git a/templates/readme-frame.md b/templates/readme-frame.md index 672507b..dcbf64d 100644 --- a/templates/readme-frame.md +++ b/templates/readme-frame.md @@ -64,3 +64,7 @@ Supporting pages enrolled in `navigation/plan.yml` use the generated `page-heade Docs indexes retain the common menu as route maps. Individual L2 guides use the generated page header for return navigation and only relevant local navigation, not another complete README menu or a self-linking Build button. Standard menu targets are available on preview branches; preserve meaningful existing anchors when they still name substantive sections. Do not fabricate compatibility or platform support from version numbers alone. The generated Repository, Organization and ↑ parent buttons provide return navigation; do not add duplicate Back blocks. User guides and implementation/qualification records occupy separately labelled groups in docs indexes; historical records do not define current product behavior. + +## Navigation and tables + +Button images belong only in the marked banner navigation / root Organization block and the marked title menu. Below the menu, use ordinary text links, including install routes and further reading. Tables grow by rows: one thing per row; add a project row or a page rather than widening a cell with a growing project list. diff --git a/tests/test_docs_audit.py b/tests/test_docs_audit.py new file mode 100644 index 0000000..de3dd96 --- /dev/null +++ b/tests/test_docs_audit.py @@ -0,0 +1,254 @@ +import copy +import subprocess +import tempfile +import unittest +from pathlib import Path + +import yaml + +from tools.snodec_sync.docs_audit import _document, docs_audit +from tools.snodec_sync.navigation import validate_plan, _config, _banner +from tools.snodec_sync.__main__ import ROOT, render_profile +import json + + +class DocsAuditTests(unittest.TestCase): + def setUp(self): + self.temp = tempfile.TemporaryDirectory() + self.addCleanup(self.temp.cleanup) + self.repo = Path(self.temp.name) / 'checkout' + self.repo.mkdir() + self.git('init', '-q') + self.git('remote', 'add', 'origin', 'git@github.com:SNodeC/snode.c.git') + self.plan = Path(self.temp.name) / 'plan.yml' + self.config = {'snode.c': {'pages': [], 'sections': {}}} + self.write('README.md', '# Root\n') + + def git(self, *args): + return subprocess.check_output(['git', '-C', str(self.repo), *args], text=True) + + def write(self, path, text, tracked=True): + p = self.repo / path + p.parent.mkdir(parents=True, exist_ok=True) + p.write_text(text) + if tracked: + self.git('add', path) + + def audit(self): + self.plan.write_text(yaml.safe_dump(self.config)) + return docs_audit(self.repo, self.plan) + + def declare(self, path, **fields): + page = dict(old=path, new=path, label='Docs', up='README.md', up_label='Root') + self.config['snode.c']['pages'].append(page | fields) + + def test_clean_root_and_untracked_files(self): + self.write('docs/untracked.md', '[missing](no.md)', tracked=False) + self.assertEqual(self.audit(), []) + + def test_existing_untracked_target_is_still_broken(self): + self.write('README.md', '[![button](button.svg)](src/page.md)') + self.write('button.svg', '', tracked=False) + self.write('src/page.md', '# Page', tracked=False) + errors = self.audit() + self.assertEqual(sum('broken-link' in error for error in errors), 2) + + def test_all_violations_collected_deterministically_without_writes(self): + self.write('README.md', '# Root\n[planned](docs/README.md#absent)\n[missing](missing.md)\n![missing](img.svg)\n') + self.write('docs/README.md', '# Docs\n') + self.declare('docs/README.md') + self.config['snode.c']['sections']['docs'] = 'Docs' + self.write('docs/unknown.md', '# Unknown\n') + before = self.git('diff', '--cached') + errors = self.audit() + self.assertEqual(errors, sorted(errors)) + for kind in ['unplanned-page', 'unreachable-page', 'broken-link', 'missing-anchor', 'missing-generated-block']: + self.assertTrue(any(kind in error for error in errors), kind) + self.assertEqual(errors, self.audit()) + self.assertEqual(before, self.git('diff', '--cached')) + self.assertTrue(any('section-pages' in error for error in errors)) + + def test_reachable_but_unplanned_page_is_not_hidden(self): + self.write('README.md', '[notes](docs/notes.md)') + self.write('docs/notes.md', '# Notes') + self.assertEqual(self.audit(), ['docs/notes.md: unplanned-page: not declared in navigation/plan.yml']) + + def test_unreachable_root_and_source_notes(self): + self.write('AGENTS.md', '# Instructions') + self.write('src/README.md', '# Source') + errors = self.audit() + self.assertEqual(len(errors), 2) + self.assertTrue(all('unreachable-page' in error for error in errors)) + self.write('README.md', '[agents](AGENTS.md)\n[source](src/README.md)') + self.assertEqual(self.audit(), []) + + def test_graph_cycles_do_not_make_pages_reachable(self): + self.write('src/a.md', '[b](b.md)') + self.write('src/b.md', '[a](a.md)') + self.assertEqual(len(self.audit()), 2) + self.write('README.md', '[a](src/a.md)') + self.assertEqual(self.audit(), []) + + def test_same_repo_blob_tree_and_slash_branch(self): + self.write('README.md', '[notes](https://github.com/SNodeC/snode.c/blob/feature/nested/src/note.md#héllo-world)\n[folder](https://github.com/SNodeC/snode.c/tree/master/extra)') + self.write('src/note.md', '# Héllo World') + self.write('extra/note.md', '# Extra') + self.assertEqual(self.audit(), []) + + def test_remote_slash_branch_root_links_resolve_to_readme(self): + self.git('config', 'user.name', 'Audit fixture') + self.git('config', 'user.email', 'audit@example.invalid') + self.git('-c', 'commit.gpgsign=false', 'commit', '-qm', 'fixture') + self.git('update-ref', 'refs/remotes/origin/review/nested/branch', 'HEAD') + self.write('README.md', '[review](https://github.com/SNodeC/snode.c/tree/review/nested/branch)') + self.assertEqual(self.audit(), []) + self.write('README.md', '[missing](https://github.com/SNodeC/snode.c/tree/review/nested/branch/missing)') + self.assertTrue(any('broken-link' in e for e in self.audit())) + + def test_published_targets_do_not_exempt_authored_source_pages(self): + self.git('remote', 'set-url', 'origin', 'git@github.com:SNodeC/DistributionPackages.git') + self.config = {'DistributionPackages': {'pages': [], 'sections': {}, 'exempt': [ + {'path': 'site/status/**', 'reason': 'Resolves in published Packages.'}, + {'path': 'site/fedora/**', 'reason': 'Resolves in published Packages.'}]}} + self.write('README.md', '[output](site/status/project.md#page-overview)\n[feed](site/fedora/)\n[bad](site/docs/missing.md)') + self.write('site/status/authored.md', '[broken](../docs/absent.md)') + errors = self.audit() + self.assertFalse(any('project.md' in e for e in errors)) + self.assertTrue(any('site/docs/missing.md' in e for e in errors)) + self.assertTrue(any('site/status/authored.md: unreachable-page' in e for e in errors)) + self.assertTrue(any('site/status/authored.md: broken-link' in e for e in errors)) + for name, path in [('snode.c', 'site/status/**'), ('DistributionPackages', 'docs/status/**'), ('DistributionPackages', 'site/**'), ('DistributionPackages', 'site/docs/**')]: + with self.subTest(name=name, path=path): + with self.assertRaises(ValueError): + validate_plan({name: {'pages': [], 'sections': {}, 'exempt': [ + {'path': path, 'reason': 'Resolves in published Packages.'}]}}) + + def test_commit_pinned_and_other_repository_links_are_not_head_checks(self): + self.write('README.md', '[historic](https://github.com/SNodeC/snode.c/blob/'+'a'*40+'/gone.md#old)\n[other](https://github.com/SNodeC/mqttsuite/blob/master/elsewhere.md#unknown)') + self.assertEqual(self.audit(), []) + self.write('README.md', '[bad](https://github.com/SNodeC/mqttsuite/blob/master)') + self.assertTrue(any('malformed SNodeC URL' in e for e in self.audit())) + + def test_html_picture_reference_links_and_escaped_paths(self): + self.write('README.md', 'Read\n![note][note]\n[note]: src/note.md#named "Title"\n[space](src/space%20name.md)\n[paren](src/a(b).md)') + self.write('src/note.md', '\n# Note') + self.write('src/space name.md', '# Space') + self.write('src/a(b).md', '# Parentheses') + for path in ['a.svg', 'b.svg']: + self.write(path, '') + self.assertEqual(self.audit(), []) + + def test_nested_list_navigation_is_not_indented_code(self): + self.write('README.md', '- A group\n [child](src/child.md)\n\n - Nested group\n [second](src/second.md)\n\n [code example](not-tracked.md)') + self.write('src/child.md', '# Child') + self.write('src/second.md', '# Second') + self.assertEqual(self.audit(), []) + + def test_code_and_comments_are_not_navigation(self): + self.write('README.md', '# Root\n`[link](gone.md)`\n\n```md\n[link](gone.md)\n```\n~~~md\n[link](gone.md)\n~~~\n [link](gone.md)') + self.assertEqual(self.audit(), []) + + def test_github_slugs_duplicate_unicode_setext_and_explicit_ids(self): + _, anchors = _document('# Hello, world!\n## Hello, world!\n## hello-world-1\n## `CMake` & **SDK**\n## 日本語\nSetext title\n===\n') + self.assertTrue({'hello-world', 'hello-world-1', 'hello-world-1-1', 'cmake--sdk', '日本語', 'setext-title', 'specific'} <= anchors) + + def test_heading_reference_links_and_underscore_emphasis(self): + _, anchors = _document('# _Italic_ and __bold__ with snake_case [API][reference]\n[reference]: https://example.invalid') + self.assertIn('italic-and-bold-with-snake_case-api', anchors) + self.write('README.md', '
[not a link](missing.md)
\n[not a link](missing.md)') + self.assertEqual(self.audit(), []) + + def test_fragment_links_and_query_strings(self): + self.write('README.md', '# Root\n[ok](#root)\n[asset](file.txt?raw=true)\n[bad](#missing)') + self.write('file.txt', 'text') + self.assertEqual(self.audit(), ['README.md: missing-anchor: #missing (target README.md)']) + + def test_exemptions_are_narrow_and_do_not_hide_agent_notes(self): + self.config['snode.c']['exempt'] = [{'path': 'vendor/**', 'reason': 'Vendored third-party library.'}, {'path': 'tests/generated/**', 'reason': 'Generated test fixture Markdown.'}] + for p in ['vendor/README.md', 'tests/generated/page.md', 'node_modules/lib/README.md', '.github/PULL_REQUEST_TEMPLATE.md']: + self.write(p, '[bad](gone.md)') + self.write('AGENTS.md', '# Agents') + self.assertEqual(self.audit(), ['AGENTS.md: unreachable-page: no route from README.md']) + for entry in [{'path': '**', 'reason': 'Asset notes'}, {'path': '../outside', 'reason': 'Vendored dependency'}, {'path': 'vendor/**', 'reason': 'Vendored\nsecond line'}]: + with self.subTest(entry=entry): + bad = copy.deepcopy(self.config) + bad['snode.c']['exempt'] = [entry] + with self.assertRaises(ValueError): + validate_plan(bad) + + def test_excluded_asset_note_is_still_audited_and_must_be_reachable(self): + self.write('docs/media/README.md', '# Media') + self.declare('docs/media/README.md', excluded=True) + self.assertEqual(self.audit(), ['docs/media/README.md: unreachable-page: no route from README.md']) + + def test_unenrolled_repository_keeps_graph_checks_without_page_planning(self): + self.git('remote', 'set-url', 'origin', 'git@github.com:SNodeC/AutoTalk.git') + self.write('README.md', '[guide](docs/USER_GUIDE.md)') + self.write('docs/USER_GUIDE.md', '# Guide') + self.assertEqual(self.audit(), []) + self.write('docs/unlinked.md', '[broken](absent.md)') + errors = self.audit() + self.assertTrue(any('broken-link' in e for e in errors)) + self.assertTrue(any('unreachable-page' in e for e in errors)) + self.assertFalse(any('unplanned-page' in e for e in errors)) + + def test_all_requested_unplanned_roots(self): + for p in ['docs/a.md', 'status/a.md', 'keys/a.md', 'web/a.md', 'packages/sdk/a.md']: + self.write(p, '# Page') + self.assertEqual(sum('unplanned-page' in e for e in self.audit()), 5) + + def test_missing_planned_page_and_partial_blocks(self): + self.declare('docs/missing.md') + self.write('docs/incomplete.md', '\n# Incomplete') + self.declare('docs/incomplete.md') + self.assertEqual(sum('missing-generated-block' in e for e in self.audit()), 2) + + def test_cli_requires_repo_and_exit_status(self): + self.git('remote', 'set-url', 'origin', 'git@github.com:SNodeC/AutoTalk.git') + python = __import__('sys').executable + for args in [[], ['--repo', str(self.repo)]]: + result = subprocess.run([python, str(ROOT / 'tools/snodec_sync'), 'docs-audit', *args], capture_output=True, text=True) + self.assertEqual(result.returncode, 1 if not args else 0) + self.write('src/bad.md', '# Unreachable') + result = subprocess.run([python, str(ROOT / 'tools/snodec_sync'), 'docs-audit', '--repo', str(self.repo)], capture_output=True, text=True) + self.assertEqual(result.returncode, 1) + self.assertIn('unreachable-page', (result.stdout + result.stderr)) + + def test_distribution_source_and_site_have_separate_policies(self): + self.git('remote', 'set-url', 'origin', 'git@github.com:SNodeC/DistributionPackages.git') + self.assertEqual(_config(self.repo)[0], 'DistributionPackages') + site = self.repo / 'site' + site.mkdir() + self.assertEqual(_config(site)[0], 'Packages') + self.assertIn('DistributionPackages', _banner('DistributionPackages', 'DistributionPackages', 'Documentation', '#ffffff')) + + + def test_image_buttons_are_allowed_only_inside_navigation_blocks(self): + self.write('docs/menu/go.svg', '') + self.write('README.md', '# Root\n[![Go](docs/menu/go.svg)](#root)\n') + self.assertTrue(any('button-outside-navigation' in e for e in self.audit())) + self.write('README.md', '# Root\n\n[![Go](docs/menu/go.svg)](#root)\n\n') + self.assertEqual(self.audit(), []) + + def test_parent_requires_downward_link(self): + self.declare('docs/page.md') + self.write('docs/page.md', '\n\n\n# Page\n') + self.assertTrue(any('missing-backlink' in e for e in self.audit())) + self.write('README.md', '# Root\n[Page](docs/page.md#page-overview)\n') + self.assertEqual(self.audit(), []) + + def test_page_size_information_and_violation(self): + self.write('README.md', '# Root\n' + 'x' * (101 * 1024)) + self.plan.write_text(yaml.safe_dump(self.config)) + info = [] + self.assertEqual(docs_audit(self.repo, self.plan, info), []) + self.assertEqual(len(info), 1) + self.write('README.md', '# Root\n' + 'x' * (301 * 1024)) + self.assertTrue(any('page-too-large' in e for e in self.audit())) + + def test_section_list_opt_out_uses_existing_project_table(self): + self.config['snode.c']['sections']['docs'] = 'Docs' + self.declare('docs/README.md', section_list=False) + self.write('README.md', '# Root\n[Index](docs/README.md#page-overview)\n') + self.write('docs/README.md', '\n\n\n# Index\n') + self.assertEqual(self.audit(), []) diff --git a/tests/test_navigation.py b/tests/test_navigation.py index b591ef5..f8b1ec9 100644 --- a/tests/test_navigation.py +++ b/tests/test_navigation.py @@ -171,3 +171,16 @@ def test_no_moves_deletions_or_state_mutation(self): with self.assertRaisesRegex(ValueError, 'git rm'): page_headers(repo) self.assertTrue(deleted.exists()) + + def test_section_list_opt_out_preserves_project_table_and_detects_menu_drift(self): + repo = self.fixture('Packages') + page = repo / 'status/README.md' + table = '| Project | Latest release | Status |\n| --- | --- | --- |\n| Third | — | [Third](third.md) |' + page.write_text(page.read_text() + '\n' + table + '\n') + page_headers(repo) + self.assertIn(table, page.read_text()) + self.assertNotIn('section-pages', page.read_text()) + self.assertEqual(page_headers(repo, check=True), []) + asset = repo / 'docs/media/menu/build.svg' + asset.write_text(asset.read_text().replace('font-size="16"', 'font-size="12"')) + self.assertIn('docs/media/menu/build.svg', page_headers(repo, check=True)) diff --git a/tests/test_verify.py b/tests/test_verify.py index c5d3d24..3713388 100644 --- a/tests/test_verify.py +++ b/tests/test_verify.py @@ -151,7 +151,7 @@ def test_template_exact_and_control_checks_independent(self): self.assertEqual((ROOT / "templates/ci.yml").read_text(), "name: CI\non: [workflow_dispatch]\n" "permissions: { contents: read }\njobs:\n snodec:\n" - " uses: SNodeC/.github/.github/workflows/verify.yml@465c7e9398b1c1ab294967153bf855af52b84082 # v1.3\n") + " uses: SNodeC/.github/.github/workflows/verify.yml@main\n") ci = yaml.load((ROOT / ".github/workflows/ci.yml").read_text(), Loader=yaml.BaseLoader) steps = ci["jobs"]["control"]["steps"] self.assertEqual(ci["on"], ["push", "pull_request", "workflow_dispatch"]) diff --git a/tools/snodec_sync/__main__.py b/tools/snodec_sync/__main__.py index cd5428a..7fed799 100644 --- a/tools/snodec_sync/__main__.py +++ b/tools/snodec_sync/__main__.py @@ -108,7 +108,7 @@ def render_profile(catalog, metadata): f"- **Run MQTT infrastructure:** [MQTTSuite]({repo_url('mqttsuite', metadata.get('mqttsuite', {}).get('readme_anchor', ''))})", f"- **Learn:** [Book companion]({repo_url('SNode.C-Book-Companion')}) and [examples](#learn)", "- **Install signed packages:** " + " · ".join([f"[Packages]({repo_url('Packages')})", *(text(entry["name"]) for entry in catalog if entry.get("packaged") and entry["status"] == "active")]), - "\n

\n" + "\n".join(f'{html.escape(title)}' for area, title in AREAS.items() if area != "forks") + "\n

\n", + "\n\n

\n" + "\n".join(f'{html.escape(title)}' for area, title in AREAS.items() if area != "forks") + "\n

\n\n", ] def table(entries, forks=False): @@ -165,7 +165,7 @@ def table(entries, forks=False): def main(argv=None): parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument("command", choices=["check", "profile", "page-headers"]) + parser.add_argument("command", choices=["check", "profile", "page-headers", "docs-audit"]) parser.add_argument("--repo", type=Path) parser.add_argument("--check", action="store_true", help="report header/asset drift without writing") parser.add_argument("--catalog", type=Path, default=ROOT / "catalog.yml") @@ -173,6 +173,19 @@ def main(argv=None): parser.add_argument("--output", type=Path, default=ROOT / "profile/README.md") args = parser.parse_args(argv) try: + if args.command == "docs-audit": + if args.repo is None: + raise ValueError("docs-audit requires --repo ") + sys.path.insert(0, str(ROOT / "tools")) + from snodec_sync.docs_audit import docs_audit + information = [] + errors = docs_audit(args.repo, information=information) + for line in information: + print(line) + if errors: + raise ValueError("\n".join(errors)) + print("docs-audit: 0 violations") + return 0 if args.command == "page-headers": if args.repo is None: raise ValueError("page-headers requires --repo ") diff --git a/tools/snodec_sync/docs_audit.py b/tools/snodec_sync/docs_audit.py new file mode 100644 index 0000000..2211a47 --- /dev/null +++ b/tools/snodec_sync/docs_audit.py @@ -0,0 +1,228 @@ +"""Read-only audit of tracked Markdown, navigation coverage and local link graphs.""" +import fnmatch +import html +from html.parser import HTMLParser +from pathlib import Path, PurePosixPath +import posixpath +import re +import subprocess +import unicodedata +from urllib.parse import unquote, urlsplit + +import yaml + +from .navigation import ROOT, validate_plan + + +def _git(repo, *args): + result = subprocess.run(['git', '-C', str(repo), *args], capture_output=True) + if result.returncode: + raise ValueError(result.stderr.decode('utf-8', errors='replace').strip()) + return result.stdout.decode('utf-8', errors='surrogateescape') + + +def _without_code(text): + lines, fence, list_indent = [], None, None + for line in text.splitlines(): + indent = len(line) - len(line.lstrip(' ')) + bullet = re.match(r'^ *(?:[-+*]|[0-9]+[.)]) +', line) + if not fence and bullet and (indent <= 3 or list_indent is not None and indent < list_indent + 4): + list_indent = bullet.end() + elif not fence and line.strip() and indent == 0: + list_indent = None + content = line[list_indent:] if list_indent is not None and indent >= list_indent else line + marker = re.match(r'^ {0,3}(`{3,}|~{3,})(.*)$', content) + if fence: + if marker and marker[1][0] == fence[0] and len(marker[1]) >= len(fence) and not marker[2].strip(): + fence = None + lines.append('') + elif marker: + fence = marker[1] + lines.append('') + else: + lines.append('' if content.startswith((' ', '\t')) else content) + return '\n'.join(lines) + + +class _Markup(HTMLParser): + def __init__(self): + super().__init__(convert_charrefs=True) + self.links, self.ids = [], set() + + def handle_starttag(self, tag, attributes): + attrs = dict(attributes) + self.ids.update(attrs[k] for k in ('id',) if attrs.get(k)) + if tag == 'a' and attrs.get('name'): + self.ids.add(attrs['name']) + for key in ('href', 'src'): + if attrs.get(key): + self.links.append(attrs[key]) + if attrs.get('srcset'): + self.links.extend(item.strip().split()[0] for item in attrs['srcset'].split(',') if item.strip()) + + +def _label(text): + return ' '.join(text.split()).casefold() + + +def _document(text): + text = _without_code(text) + # Code spans and comments are not links or HTML anchors. + headings = re.sub(r'', '', text, flags=re.S) + text = re.sub(r'(`+)(?:(?!\1).)*?\1|<(pre|code)\b[^>]*>.*?', '', headings, flags=re.S) + markup = _Markup() + markup.feed(text) + definitions = {} + for match in re.finditer(r'^ {0,3}\[([^\]]+)\]:\s*(<[^>]+>|\S+)', text, re.M): + definitions[_label(match[1])] = match[2].strip('<>') + body = re.sub(r'^ {0,3}\[[^\]]+\]:.*$', '', text, flags=re.M) + links = list(markup.links) + # Inspect destinations rather than labels: nested linked images yield both URLs. + for match in re.finditer(r'\]\(\s*(<[^>]+>|(?:\\.|[^\s()]|\([^()]*\))+)(?:\s+[\"\'][^\n]*?[\"\'])?\s*\)', body): + links.append(match[1].strip('<>')) + for match in re.finditer(r'!?\[([^\]\n]+)\](?:\[([^\]\n]*)\])?', body): + target = definitions.get(_label(match[2] or match[1])) + if target: + links.append(target) + links.extend(re.findall(r'<(https?://[^<>\s]+)>', body)) + anchors, used = set(markup.ids), set() + lines = headings.splitlines() + for index, line in enumerate(lines): + heading = re.match(r'^ {0,3}#{1,6}\s+(.+?)(?:\s+#+\s*)?$', line) + value = heading[1] if heading else None + if value is None and index + 1 < len(lines) and line.strip() and re.match(r'^ {0,3}(?:=+|-+)\s*$', lines[index + 1]): + value = line.strip() + if value is None: + continue + value = re.sub(r'!?\[([^\]]*)\](?:\([^)]*\)|\[[^\]]*\])', r'\1', value) + value = html.unescape(re.sub(r'<[^>]+>', '', value)).lower() + value = re.sub(r'\\([!"#$%&\'()*+,\-./:;<=>?@\[\]^_`{|}~])', r'\1', value) + value = re.sub(r'(?= 3 and parts[2] in ('blob', 'tree'): + if len(parts) < 4 or not parts[3] or parts[2] == 'blob' and len(parts) < 5: + report('broken-link', source, f'malformed SNodeC URL {href}') + return None + if parts[1] != name or re.fullmatch(r'[0-9a-fA-F]{40}', parts[3]): + return None + # A branch name can contain slashes: prefer the longest tracked suffix. + candidates = [unquote('/'.join(parts[i:])) for i in range(4, len(parts))] or ['README.md'] + target = 'README.md' if parts[2] == 'tree' and '/'.join(parts[3:]) in refs else next((p for p in candidates if p in tracked or any(t.startswith(p + '/') for t in tracked)), candidates[0]) + elif parts[1] == name and len(parts) == 2: + target = 'README.md' + else: + return None + else: + target = posixpath.normpath(posixpath.join(posixpath.dirname(source), unquote(url.path))) if url.path else source + if url.path.startswith('/'): + target = unquote(url.path).lstrip('/') + if target not in tracked: + children = [p for p in tracked if p.startswith(target.rstrip('/') + '/')] + if children: + # A directory view exposes its tracked documents and subdirectories. + graph[source].update(p for p in markdown if p in children) + target = target.rstrip('/') + '/README.md' + if target not in tracked and not url.fragment: + return None + else: + if any(entry['reason'].lower().startswith('resolves in published packages') and (fnmatch.fnmatchcase(target, entry['path']) or fnmatch.fnmatchcase(target + '/', entry['path'])) for entry in config.get('exempt', [])): + return None + report('broken-link', source, href) + return None + fragment = unquote(url.fragment) + if fragment and target in documents and fragment not in documents[target][1]: + report('missing-anchor', source, f'{href} (target {target})') + return target + + for path in markdown: + parts = PurePosixPath(path).parts + in_scope = parts[0] in ('docs', 'status', 'keys', 'web') or (parts[0] == 'packages' and len(parts) >= 3) + if name in plan and in_scope and path not in pages: + report('unplanned-page', path, 'not declared in navigation/plan.yml') + for href in documents[path][0]: + target = resolve(path, href) + if target in graph: + graph[path].add(target) + for path, page in sorted(pages.items()): + if page.get('excluded'): + continue + if path not in tracked: + report('missing-generated-block', path, 'planned page is not tracked') + continue + text = _without_code((repo / path).read_text()) + if not re.search(r'.*?', text, re.S): + report('missing-generated-block', path, 'page-header') + if path.endswith('/README.md') and posixpath.dirname(path) in config['sections'] and page.get('section_list', True) and not re.search(r'.*?', text, re.S): + report('missing-generated-block', path, 'section-pages') + # Optional review rules: none of these are imported by Verify or publication. + for path in markdown: + text = _without_code((repo / path).read_text()) + allowed = [(m.start(), m.end()) for m in re.finditer( + r'.*?', text, re.S)] + buttons = re.finditer(r']*>\s*(?:.*?)?]*src="[^"\n]*(?:menu|nav)/[^"\n]+"[^>]*>.*?|\[!\[[^\]]*\]\([^)]*(?:menu|nav)/[^)]+\)\]\([^)]+\)', text, re.S) + for button in buttons: + if not any(start <= button.start() < end for start, end in allowed): + report('button-outside-navigation', path, f'line {text.count(chr(10), 0, button.start()) + 1}') + page = pages.get(path) + if page and not page.get('excluded') and page['up'] in graph and path not in graph[page['up']]: + report('missing-backlink', path, f"parent {page['up']} does not link down to this page") + size = (repo / path).stat().st_size + if size > 300 * 1024: + report('page-too-large', path, f'{size} bytes exceeds 300 KB') + elif size > 100 * 1024 and information is not None: + information.append(f'{path}: information: {size} bytes exceeds 100 KB') + reachable, queue = set(), ['README.md'] + while queue: + path = queue.pop() + if path not in reachable: + reachable.add(path) + queue.extend(graph.get(path, set()) - reachable) + for path in markdown: + if path not in reachable: + report('unreachable-page', path, 'no route from README.md') + return sorted(errors) diff --git a/tools/snodec_sync/navigation.py b/tools/snodec_sync/navigation.py index b956865..05aab44 100644 --- a/tools/snodec_sync/navigation.py +++ b/tools/snodec_sync/navigation.py @@ -25,6 +25,15 @@ def validate_plan(plan): if not isinstance(config, dict) or any(not isinstance(k, str) or not isinstance(v, str) or not v.strip() for k, v in config.items()): raise ValueError('menu_labels requires nonempty text labels') continue + for entry in config.get('exempt', []): + if (not isinstance(entry, dict) or set(entry) != {'path', 'reason'} + or not isinstance(entry['path'], str) or not entry['path'] + or PurePosixPath(entry['path']).is_absolute() or '..' in PurePosixPath(entry['path']).parts + or not isinstance(entry['reason'], str) or '\n' in entry['reason'] or '\r' in entry['reason'] + or not entry['reason'].lower().startswith(('vendored', 'third-party', 'generated test fixture', 'resolves in published packages'))): + raise ValueError(f'{name}: exempt requires a safe path glob and approved one-line reason') + if entry['reason'].lower().startswith('resolves in published packages') and (name != 'DistributionPackages' or not entry['path'].startswith('site/') or len(PurePosixPath(entry['path']).parts) < 3 or PurePosixPath(entry['path']).parts[1] == 'docs'): + raise ValueError('Published-output exemptions are limited to DistributionPackages/site targets') active = {} for page in config['pages']: for field in ('old', 'new', 'up'): @@ -33,6 +42,8 @@ def validate_plan(plan): raise ValueError(f'{name}: unsafe {field}: {path}') if len(page['up_label']) > 15 or not page['up_label'].strip(): raise ValueError(f'{name}: Up label exceeds 15 characters: {page["up_label"]}') + if 'section_list' in page and type(page['section_list']) is not bool: + raise ValueError(f'{name}: section_list must be boolean') if page.get('deleted'): continue if page['new'] in active: @@ -200,6 +211,8 @@ def _strip_header(text): def _section_list(config, path, texts): + if not _page(config, path).get('section_list', True): + return '' entries = [] for page in config['pages']: if page.get('deleted') or page.get('excluded') or page['up'] != path or page['new'] not in texts: @@ -325,8 +338,9 @@ def page_headers(repo, check=False): listing = '## Sections\n\n' + _section_list(config, path, texts) pattern = r'(?:## Sections\n\n)?.*?' body = re.sub(pattern, '', body, flags=re.S).rstrip() - prefix, separator, rest = body.partition('\n## ') - body = prefix.rstrip() + '\n\n' + listing + ('\n\n## ' + rest if separator else '') + if page.get('section_list', True): + prefix, separator, rest = body.partition('\n## ') + body = prefix.rstrip() + '\n\n' + listing + ('\n\n## ' + rest if separator else '') body = _header(config, path) + '\n\n' + body texts[path] = (prefixes[path] + '\n\n' if prefixes[path] else '') + body + '\n' outputs.update(texts)