From 7ddcdaf9f99d0e5cb8d1fd1d95c4cb0e3685b1b2 Mon Sep 17 00:00:00 2001
From: Claude
Date: Sun, 27 Sep 2026 11:17:45 +0000
Subject: [PATCH 1/4] Add project skills for quick actions and a CLAUDE.md
Six skills in .claude/skills/: add-collection, triage-leads (fans out to
parallel subagents and merges their drafts), verify-records,
fix-broken-links, release-dataset, and site-audit. The heavier ones run in
a forked subagent context. CLAUDE.md records the ground rules (honesty
rule, verification, review log, permanent IDs), the checks to run, and the
skill index.
Co-Authored-By: Claude Opus 5.5
Claude-Session: https://claude.ai/code/session_0134nrWUakNL58qJMzCVDcn6
---
.claude/skills/add-collection/SKILL.md | 41 +++++++++++++++++++++
.claude/skills/fix-broken-links/SKILL.md | 28 +++++++++++++++
.claude/skills/release-dataset/SKILL.md | 30 ++++++++++++++++
.claude/skills/site-audit/SKILL.md | 29 +++++++++++++++
.claude/skills/triage-leads/SKILL.md | 45 ++++++++++++++++++++++++
.claude/skills/verify-records/SKILL.md | 29 +++++++++++++++
.gitignore | 2 ++
CLAUDE.md | 40 +++++++++++++++++++++
8 files changed, 244 insertions(+)
create mode 100644 .claude/skills/add-collection/SKILL.md
create mode 100644 .claude/skills/fix-broken-links/SKILL.md
create mode 100644 .claude/skills/release-dataset/SKILL.md
create mode 100644 .claude/skills/site-audit/SKILL.md
create mode 100644 .claude/skills/triage-leads/SKILL.md
create mode 100644 .claude/skills/verify-records/SKILL.md
create mode 100644 .gitignore
create mode 100644 CLAUDE.md
diff --git a/.claude/skills/add-collection/SKILL.md b/.claude/skills/add-collection/SKILL.md
new file mode 100644
index 0000000..658d8f4
--- /dev/null
+++ b/.claude/skills/add-collection/SKILL.md
@@ -0,0 +1,41 @@
+---
+name: add-collection
+description: Add one real oral-history or voice-testimony collection to data/collections.json the Mapping Voices way — next MV- id, controlled vocabularies, honest verification status and provenance, review-log entry, regenerated exports. Use when asked to "add", "list", or "put on the map" a specific collection, or to turn a new-collection issue into a record.
+---
+
+# Add a collection
+
+Rules that override everything below: METHODOLOGY.md §1 (inclusion) and the
+honesty rule — never invent a field. Omit what the source doesn't support.
+
+1. **Confirm it qualifies** (METHODOLOGY §1–2): recorded human voice; real
+ holder; the holder's own public page. Search for duplicates first:
+ `grep -i "" data/collections.json` and by URL host.
+2. **Gather facts** from the holder's page (WebFetch; if blocked, WebSearch
+ results pointing at the holder's own pages — then the record stays
+ `partially_verified` and the method is `search-based`). Need: title,
+ holder, place (city), languages *as the source states them*, what it
+ documents, start/end years, access conditions.
+3. **Write the record** — field reference: DATA_DICTIONARY.md §2.
+ - `mv_id`: next unused (`node -e 'const d=require("./data/collections.json");console.log(Math.max(...d.map(r=>+r.mv_id.slice(3)))+1)'`).
+ - `country`, `languages`, `themes` must be terms in `data/vocab/`. Add a
+ missing language (with ISO 639-3) or theme (with group + scope note)
+ to the vocab in the same change. Never infer languages from geography;
+ use `Multiple languages` + `language_note` with the source's wording.
+ - Pin: city-level; subject-country rule in METHODOLOGY §5.
+ - `citation`: `". . Retrieved from ."`
+ - `verification_status`: `partially_verified` unless a person checked
+ every core field against the open source page (then `verified` +
+ `last_reviewed`). Flag inferred fields in `verification_note`.
+ - `provenance`: how it was found (issue number, contributor, search).
+ - `date_added`: today. Keep the canonical key order (see existing records).
+ - `related_ids`: only genuine relations; add the reverse link too.
+4. **Log it**: append to `data/review-log.json` —
+ `{date, mv_id, check:"new record", method, reviewer, finding, evidence:[urls], changes:null, outcome:"added"}`.
+5. **Check and build**:
+ ```bash
+ node scripts/validate-data.mjs && node --test tests/*.test.mjs
+ node scripts/build-data.mjs && node scripts/build-data.mjs --check
+ ```
+6. Add a line under `## [Unreleased]` in CHANGELOG.md (create the section
+ if missing). Commit with a message naming the MV- id.
diff --git a/.claude/skills/fix-broken-links/SKILL.md b/.claude/skills/fix-broken-links/SKILL.md
new file mode 100644
index 0000000..38e40b0
--- /dev/null
+++ b/.claude/skills/fix-broken-links/SKILL.md
@@ -0,0 +1,28 @@
+---
+name: fix-broken-links
+description: Follow up the weekly link-check issue ("Broken links detected in data/collections.json") — separate real failures from bot blocking, find each holder's current page, correct URLs and citations, flag what can't be resolved, and log every check. Use for "fix broken links", "link rot", or when the link-check issue updates.
+context: fork
+agent: general-purpose
+---
+
+# Fix broken links
+
+1. Read the open GitHub issue labelled `broken-links` in adab-tech/mapping
+ (GitHub MCP `issue_read`). Save its table to a file for the worksheet.
+2. Triage by result: **404 / 410 / DNS failure** = real, fix first.
+ **403 / 429 / 503 / timeout** = usually bot blocking or transient; only
+ investigate if it persists across two weekly runs.
+3. For each real failure, find the holder's **current own page** (WebFetch;
+ if blocked, WebSearch results on the holder's domain). Never substitute a
+ third-party or Wikipedia page. If a collection has truly gone, set
+ `verification_status: "unavailable"` with a dated `verification_note` —
+ never delete the record (METHODOLOGY §10).
+4. Apply with a migration script (pattern:
+ `scripts/migrations/2026-09-27-source-review.mjs`): update `url`, replace
+ the old URL inside `citation`, and append one `data/review-log.json`
+ entry per record (check `link`, evidence URLs, changes, outcome).
+5. Also move `http://` URLs to https where the same page is served there
+ (the validator lists them as warnings).
+6. `node scripts/validate-data.mjs && node --test tests/*.test.mjs && node scripts/build-data.mjs`;
+ CHANGELOG `[Unreleased]` under "Broken-source changes"; commit.
+7. Regenerate the worksheet with `--links` so unresolved records are priority 1.
diff --git a/.claude/skills/release-dataset/SKILL.md b/.claude/skills/release-dataset/SKILL.md
new file mode 100644
index 0000000..757dedf
--- /dev/null
+++ b/.claude/skills/release-dataset/SKILL.md
@@ -0,0 +1,30 @@
+---
+name: release-dataset
+description: Cut a new Mapping Voices dataset version — bump the version everywhere it appears, finalize the CHANGELOG, refresh counts, rebuild exports, and run every check. Use for "release", "bump the version", "cut v0.x".
+---
+
+# Release the dataset
+
+1. Pick the version (semantic: new records/fields → minor; corrections only → patch).
+2. Update **every** place the version or counts appear — find them with:
+ ```bash
+ grep -rn "0\.[0-9]\.[0-9]" --include=*.md --include=*.html --include=*.cff --include=*.json . \
+ | grep -v "data/\(collections\|languages\|themes\|countries\|stats\|review-log\)"
+ ```
+ Always: `data/dataset-meta.json` (version, released), `CITATION.cff`
+ (version, date-released), `index.html` JSON-LD (version, dateModified),
+ `about.html` fallback figures and citation, `README.md` headline counts
+ and citation, `DATA_DICTIONARY.md` schema line, `PROJECT.md` status.
+ Counts come from `node scripts/build-data.mjs && node -e 'console.log(require("./data/stats.json").totals)'`.
+3. CHANGELOG: rename `[Unreleased]` to `[x.y.z] — YYYY-MM-DD — `,
+ grouped as Dataset (additions, removals, corrections, broken-source
+ changes, taxonomy), Atlas, Tooling.
+4. Rebuild and run everything:
+ ```bash
+ node scripts/build-data.mjs
+ node scripts/validate-data.mjs && node --test tests/*.test.mjs \
+ && node scripts/build-data.mjs --check && node scripts/validate-locales.mjs
+ ```
+5. Run the `site-audit` skill if UI changed. Commit "Release dataset vx.y.z".
+6. After merge: suggest a GitHub release/tag (Zenodo mints a DOI per release
+ if the integration is enabled).
diff --git a/.claude/skills/site-audit/SKILL.md b/.claude/skills/site-audit/SKILL.md
new file mode 100644
index 0000000..fdcc4ae
--- /dev/null
+++ b/.claude/skills/site-audit/SKILL.md
@@ -0,0 +1,29 @@
+---
+name: site-audit
+description: Run the browser and accessibility audit (axe-core WCAG 2.1 AA, device emulation, reflow, keyboard, reduced motion, RTL) against a local server and fix what fails. Use after UI changes, before a release, or when asked to "test the site", "check accessibility", "check mobile".
+context: fork
+agent: general-purpose
+---
+
+# Site audit
+
+1. Serve the repo: `python3 -m http.server 8765` (background).
+2. Get dependencies without polluting the repo, e.g. in a scratch dir:
+ `npm pack axe-core leaflet@1.9.4` and extract (both unpack to `package/`
+ — extract one at a time into separate folders). Playwright: use a global
+ install if present (`npm root -g`), Chromium at `/opt/pw-browsers`; do
+ not run `playwright install`.
+3. Run:
+ ```bash
+ PW=$(npm root -g)/playwright AXE=/axe.min.js LEAFLET_DIR=/dist \
+ node tests/browser/audit.mjs http://localhost:8765
+ ```
+ (`LEAFLET_DIR` only if the unpkg CDN is unreachable.)
+4. For each FAIL: reproduce, fix in the page's CSS/JS, re-run until green.
+ Contrast failures on animated elements may be mid-transition — the audit
+ waits for animations; confirm at rest before changing colours.
+5. If you add a page, add it to the page lists in `tests/browser/audit.mjs`.
+6. Update the "Last run" line and results table in
+ docs/testing/BROWSER-TESTING.md. Only Chromium can run here — keep the
+ manual Safari/Firefox/device/screen-reader checklist honest.
+7. Stop the server.
diff --git a/.claude/skills/triage-leads/SKILL.md b/.claude/skills/triage-leads/SKILL.md
new file mode 100644
index 0000000..3746a16
--- /dev/null
+++ b/.claude/skills/triage-leads/SKILL.md
@@ -0,0 +1,45 @@
+---
+name: triage-leads
+description: Research open leads in docs/review-queue.csv (or a list the user gives) and resolve each one — add as a record, mark duplicate, out of scope, curator decision, or needs URL. Fans out to parallel subagents for batches. Use for "work the review queue", "triage leads", "process these suggestions".
+context: fork
+agent: general-purpose
+---
+
+# Triage leads
+
+**Input:** open rows in `docs/review-queue.csv` (column `triage_2026_09_27`
+starts with `open:`), or a list from the user. Prefer leads that improve
+coverage outside the US/UK — the dataset is skewed (see docs/AUDIT.md §5).
+
+## Delegate in batches
+
+For more than ~5 leads, split them into batches of 4–6 and spawn one
+general-purpose subagent per batch **in parallel**. Give each the lead rows
+and these instructions, and ask for JSON back — do not let subagents edit
+files (avoids conflicts):
+
+```json
+{"collection": "...", "outcome": "add|duplicate|out of scope|curator decision|needs URL",
+ "reason": "...", "evidence": ["https://..."],
+ "record": { /* full draft record per DATA_DICTIONARY §2, only when outcome=add */ }}
+```
+
+Each subagent: check duplicates against data/collections.json; apply the
+inclusion rules (METHODOLOGY §1–2); research with WebSearch/WebFetch; draft
+records only from what the holder's pages support; mark inferred languages
+or unknown years in `verification_note`; never guess a URL.
+
+## Merge (main context)
+
+1. Review every draft adversarially: is the URL the holder's own page? Is
+ any field unsupported? Downgrade to `curator decision` when unsure.
+2. Apply with a migration script in `scripts/migrations/-queue-*.mjs`
+ (pattern: `2026-09-27-queue-additions.mjs`): assign sequential `mv_id`s,
+ add vocab terms, reciprocal `related_ids`, review-log entries (method
+ `search-based` unless pages were opened), and write outcomes into a new
+ `triage_` column of docs/review-queue.csv.
+3. `node scripts/validate-data.mjs && node --test tests/*.test.mjs && node scripts/build-data.mjs`
+4. CHANGELOG `[Unreleased]`; commit. Report counts per outcome.
+
+Sensitive collections (trauma, legal proceedings, community-governed
+material) go to `curator decision` with the reason — METHODOLOGY §11.
diff --git a/.claude/skills/verify-records/SKILL.md b/.claude/skills/verify-records/SKILL.md
new file mode 100644
index 0000000..b979bb0
--- /dev/null
+++ b/.claude/skills/verify-records/SKILL.md
@@ -0,0 +1,29 @@
+---
+name: verify-records
+description: Run the field-by-field verification workflow — generate the prioritized review worksheet, or import a completed one so records become verified. Use for "verification pass", "review worksheet", "import my review", "promote records to verified".
+---
+
+# Verify records
+
+Only a **person who opened the source page** can make a record `verified`
+(METHODOLOGY §4). Claude can prepare and import, not verify. Full guide:
+docs/review/README.md.
+
+**Prepare** a worksheet (optionally raising link-check failures):
+```bash
+# save the body of the latest "Broken links detected" issue to /tmp/links.md if relevant
+node scripts/review-worksheet.mjs [--links /tmp/links.md]
+```
+Hand the user `docs/review/worksheet.csv` and the column guide from
+docs/review/README.md §2.
+
+**Import** a completed worksheet the user provides:
+```bash
+node scripts/apply-review.mjs --dry-run # show what would change
+node scripts/apply-review.mjs
+node scripts/validate-data.mjs && node scripts/build-data.mjs
+```
+Rows with fields marked `no` stay `partially_verified` and are listed for
+hand correction — make those corrections in data/collections.json only from
+the reviewer's notes, log them, and ask the reviewer to confirm. Report how
+many records are now verified (`data/stats.json` → `by_verification_status`).
diff --git a/.gitignore b/.gitignore
new file mode 100644
index 0000000..b751c44
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,2 @@
+# Local agent worktrees (Claude Code)
+.claude/worktrees/
diff --git a/CLAUDE.md b/CLAUDE.md
new file mode 100644
index 0000000..eff3b08
--- /dev/null
+++ b/CLAUDE.md
@@ -0,0 +1,40 @@
+# Mapping Voices — notes for Claude sessions
+
+Static, no-build atlas (plain HTML/CSS/JS + Leaflet) over an open dataset of
+oral-history collections. The research contribution is the metadata, so
+data integrity outranks everything else.
+
+## Ground rules
+- **Honesty rule:** never invent metadata. Omit fields the source doesn't
+ support; flag inferences in `verification_note`. Details: METHODOLOGY.md.
+- Only a person who opened the source page can make a record `verified`.
+ Search-based checks may correct links but never promote.
+- Every data change is logged in `data/review-log.json` and made through a
+ reproducible script in `scripts/migrations/` when it touches many records.
+- `mv_id`s are permanent; never renumber or reuse. Never delete a record
+ whose source disappears — mark it `unavailable`.
+- Counts describe the indexed dataset, not oral history in the world.
+
+## Checks (run before every commit)
+```bash
+node scripts/validate-data.mjs && node --test tests/*.test.mjs
+node scripts/build-data.mjs --check # run without --check to regenerate exports
+node scripts/validate-locales.mjs # UI strings exist in en, ha, fr, ar
+```
+
+## Quick actions (project skills in .claude/skills/)
+| Skill | Use for |
+|---|---|
+| `add-collection` | Add one collection correctly |
+| `triage-leads` | Work the review queue; fans out to parallel subagents |
+| `verify-records` | Generate / import the field-by-field review worksheet |
+| `fix-broken-links` | Follow up the weekly link-check issue |
+| `release-dataset` | Bump version everywhere, finalize CHANGELOG, rebuild |
+| `site-audit` | Browser + accessibility audit (tests/browser/audit.mjs) |
+
+## Environment notes
+- The Leaflet CDN, map tiles, and many archive sites may be unreachable from
+ cloud sessions; WebSearch usually works. `npm pack` works for fetching
+ packages into a scratch dir. Chromium is at /opt/pw-browsers.
+- UI strings: add every new key to all four `locales/*.json` files with real
+ translations.
From 7d0b902539e3c96981a3f39438e28093048e8be2 Mon Sep 17 00:00:00 2001
From: Claude
Date: Sun, 27 Sep 2026 11:24:45 +0000
Subject: [PATCH 2/4] Add Theme Explorer and Coverage Gaps pages
themes.html + js/themes.js: the 15 theme groups, each theme with its scope
note, collections, countries and languages; search, sort, URL state and
#theme- anchors, following the Language Explorer.
gaps.html + js/gaps.js: UN member states with and without an indexed
collection per UN M49 region and subregion, with propose-a-collection
links. Adds static data/reference/un-m49-countries.json (193 member
states) and tests/un-m49-countries.test.mjs.
Links both pages from the site nav, about page and atlas footer (new
locale keys footer.themes and footer.gaps); README and CHANGELOG updated.
Co-Authored-By: Claude Opus 5.5
Claude-Session: https://claude.ai/code/session_0134nrWUakNL58qJMzCVDcn6
---
CHANGELOG.md | 20 +
README.md | 17 +-
about.html | 6 +-
css/gaps.css | 189 +++++
css/themes.css | 42 +
data/reference/un-m49-countries.json | 1168 ++++++++++++++++++++++++++
gaps.html | 114 +++
index.html | 2 +
js/gaps.js | 301 +++++++
js/themes.js | 388 +++++++++
languages.html | 2 +
locales/ar.json | 4 +-
locales/en.json | 4 +-
locales/fr.json | 4 +-
locales/ha.json | 4 +-
tests/un-m49-countries.test.mjs | 68 ++
themes.html | 101 +++
17 files changed, 2426 insertions(+), 8 deletions(-)
create mode 100644 css/gaps.css
create mode 100644 css/themes.css
create mode 100644 data/reference/un-m49-countries.json
create mode 100644 gaps.html
create mode 100644 js/gaps.js
create mode 100644 js/themes.js
create mode 100644 tests/un-m49-countries.test.mjs
create mode 100644 themes.html
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 937d307..80d74b8 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -4,6 +4,26 @@ All notable changes to the Mapping Voices dataset and atlas. Dataset
changes are recorded as additions, removals, corrections, metadata changes,
broken-source changes, and taxonomy changes (METHODOLOGY §10, §12).
+## [Unreleased]
+
+### Atlas
+- New **Theme Explorer** (`themes.html`): the 15 top-level theme groups,
+ each theme with its scope note and count, the collections it is assigned
+ to, and the countries and languages they span, linking back into filtered
+ atlas views. Search, sort, and the open theme live in the URL
+ (`?q=…&sort=…#theme-…`).
+- New **Coverage Gaps** page (`gaps.html`): per UN M49 region and subregion,
+ which UN member states have an indexed collection and which have none
+ yet, each gap with a link to propose a collection. States plainly that it
+ shows gaps in the indexed dataset, not in oral history.
+- New static reference file `data/reference/un-m49-countries.json` (the 193
+ UN member states, ISO 3166-1 alpha-2, M49 region/subregion), with a test
+ (`tests/un-m49-countries.test.mjs`) that it agrees with
+ `data/vocab/countries.json`.
+- "Themes" and "Coverage gaps" added to the site navigation and the atlas
+ footer (new locale keys `footer.themes`, `footer.gaps` in en, ha, fr, ar).
+- No dataset changes.
+
## [0.3.0] — 2026-09-27 — Language Explorer, review tooling, source review
### Dataset
diff --git a/README.md b/README.md
index 072143d..43a98e8 100644
--- a/README.md
+++ b/README.md
@@ -63,7 +63,9 @@ Tanzania, Myanmar, Burkina Faso, and many other places have no entry yet,
usually because no verifiable public page for a real archive has been
confirmed, not because the region lacks oral history. The absence of a pin
is not evidence of absence. See [`docs/AUDIT.md`](docs/AUDIT.md) for the
-gap analysis.
+gap analysis, and the [Coverage Gaps](https://adamu.tech/mapping/gaps.html)
+page (`gaps.html`) for every UN member state with no indexed collection yet,
+by region, each with a link to propose one.
## How data is verified
@@ -95,7 +97,11 @@ The dataset is licensed **CC BY 4.0** and published as:
Atlas views are addressable: filters, search, and the selected collection
live in the URL, e.g.
[`?country=Niger&language=Hausa`](https://adamu.tech/mapping/?country=Niger&language=Hausa)
-or [`?c=MV-000023`](https://adamu.tech/mapping/?c=MV-000023).
+or [`?c=MV-000023`](https://adamu.tech/mapping/?c=MV-000023). The
+[Language Explorer](https://adamu.tech/mapping/languages.html) and the
+[Theme Explorer](https://adamu.tech/mapping/themes.html) (the 15 theme groups,
+scope notes, and the collections, countries, and languages behind each theme)
+browse the same data as lists and link back into these atlas views.
**Mapping an archive does not grant permission to reproduce its
contents.** Access to and reuse of the collections themselves is governed
@@ -208,14 +214,19 @@ repository" button on the repo page).
```
index.html the atlas
about.html project page: question, method, data, ethics, limitations
-css/style.css, css/about.css styles
+languages.html Language Explorer: languages, ISO codes, countries, collections
+themes.html Theme Explorer: theme groups, scope notes, collections, countries, languages
+gaps.html Coverage Gaps: UN member states with / without an indexed collection
+css/*.css styles (style.css tokens; about, languages, themes, gaps pages)
js/app.js map, search, filters, URL state, record panel
+js/{languages,themes,gaps}.js the explorer and coverage pages
locales/*.json UI translations (en, ha, fr, ar)
data/collections.json canonical records (edit this)
data/vocab/*.json controlled vocabularies: countries, languages, themes
data/dataset-meta.json dataset version, release date, license
data/collections.{csv,geojson} generated exports (don't edit by hand)
data/{languages,themes,countries,stats,datapackage}.json generated indexes
+data/reference/un-m49-countries.json static list of the 193 UN member states (UN M49 regions)
scripts/validate-data.mjs data validator
scripts/build-data.mjs export generator
scripts/check-links.mjs weekly link-health check
diff --git a/about.html b/about.html
index dd831df..4ee81d6 100644
--- a/about.html
+++ b/about.html
@@ -36,6 +36,8 @@
LimitationsCiteLanguages
+ Themes
+ Coverage gapsOpen the atlas
@@ -116,7 +118,9 @@
02 Method
language and theme classification, duplicate handling, and corrections — is
public: METHODOLOGY.md.
Browse languages, their ISO codes, and the collections recorded in them in
- the Language Explorer. Every field is defined in the
+ the Language Explorer, the thematic taxonomy in the
+ Theme Explorer, and where the index does not yet reach
+ in Coverage Gaps. Every field is defined in the
data dictionary.
+ Where does the index not yet reach? This page lists, region by region, the
+ UN member states with at least one indexed oral-history or voice-testimony
+ collection — and those with none yet.
+
+
+ This shows gaps in the indexed dataset, not in oral history. The absence of
+ a pin is not evidence of absence: every country listed here as “not yet
+ indexed” has its own oral traditions, and very likely archives that record
+ them, which have not yet been found or verified.
+
+
+
UN member states
–
+
With an indexed collection
–
+
None indexed yet
–
+
Territories & other entities indexed
–
+
+
+ Know a collection in one of these countries?
+ Propose it
+ — no Git or JSON needed. It must be a real, publicly documented collection
+ with its own page (inclusion criteria,
+ contributing guide).
+
+
+ Counts describe the collections indexed in Mapping Voices,
+ placed by the country each collection's pin belongs to (Methodology
+ §5).
+ The list of 193 member states and their UN M49 regions is a static reference file,
+ un-m49-countries.json.
+ Territories and other non-member entities are shown where indexed but not
+ counted as gaps; grouping follows UN M49 and records no political claim.
+
+
+
+
+
By region
+
Loading coverage…
+
+
+
UN member states with and without an indexed collection, by UN M49 region