- HumanStudy-Bench: Community Edition
-Open community-driven expansion of the HumanStudy-Bench benchmark
- -
- 
- 
- 
+ HumanStudy-Bench
+A benchmark for evaluating AI agents through runnable human studies
+
+
+ 
- Open community-driven expansion of the HumanStudy-Bench benchmark
- -
-
-
-
+ A benchmark for evaluating AI agents through runnable human studies
+
+
+
Overview
+## Add a study
-HumanStudy-Bench treats participant simulation as an *agent design problem* and provides a **standardized testbed** — combining an **Execution Engine** that reconstructs full experimental protocols from published studies and a **Benchmark** with standardized evaluation metrics — for *replaying human-subject experiments end-to-end* with alignment evaluation at the level of scientific inference.
+There are two contribution paths.
-## 
How to Contribute a Study
+| Path | Best for | Current status |
+|---|---|---|
+| **Build Study** | Researchers starting from a paper PDF and optional open materials | **Private beta.** HumanStudy-Hub runs this repository's pipeline, persists review decisions, and produces a ZIP or GitHub pull request. |
+| **Direct pull request** | Contributors who want full control over study files | **Available now.** Build the folder locally, run validation, and open a PR. |
+
+The intended Build Study workflow is:
+
+```text
+paper PDF + optional OSF/open materials
+ -> evidence and study extraction
+ -> researcher review at uncertain decisions
+ -> runnable HumanStudy-Bench folder
+ -> download ZIP and/or save to GitHub
+```
-### 1. Fork and clone
+Build Study is currently operated through the private HumanStudy-Hub web
+application. The direct contribution documentation remains supported for
+researchers who prefer to author or inspect every file themselves. The manual
+entry point is the [direct pull request guide](docs/submit_study.md).
+
+### Direct pull request
+
+1. Fork this repository and clone your fork.
```bash
git clone https://github.com/
Existing Studies
+Run a stage with:
-The 12 foundational studies (cognition, strategic interaction, social psychology) serve as reference examples. Browse them on the [website](https://www.hs-bench.clawder.ai/contribute#studies) or locally under `studies/`.
+```bash
+python generation_pipeline/run.py \
+ --settings config/settings.yaml \
+ --stage 1 \
+ --pdf /absolute/path/to/paper.pdf \
+ --provider openai \
+ --model gpt-5-mini
+```
-## 
Citation
+Generated evidence and review artifacts are written under
+`generation_pipeline/outputs/
License
+## License
-MIT License. See [LICENSE](LICENSE) for details.
+MIT License. See [LICENSE](LICENSE).
diff --git a/co_website/.gitignore b/co_website/.gitignore
deleted file mode 100644
index 5ef6a52..0000000
--- a/co_website/.gitignore
+++ /dev/null
@@ -1,41 +0,0 @@
-# See https://help.github.com/articles/ignoring-files/ for more about ignoring files.
-
-# dependencies
-/node_modules
-/.pnp
-.pnp.*
-.yarn/*
-!.yarn/patches
-!.yarn/plugins
-!.yarn/releases
-!.yarn/versions
-
-# testing
-/coverage
-
-# next.js
-/.next/
-/out/
-
-# production
-/build
-
-# misc
-.DS_Store
-*.pem
-
-# debug
-npm-debug.log*
-yarn-debug.log*
-yarn-error.log*
-.pnpm-debug.log*
-
-# env files (can opt-in for committing if needed)
-.env*
-
-# vercel
-.vercel
-
-# typescript
-*.tsbuildinfo
-next-env.d.ts
diff --git a/co_website/README.md b/co_website/README.md
deleted file mode 100644
index 9b95270..0000000
--- a/co_website/README.md
+++ /dev/null
@@ -1,41 +0,0 @@
-## co_website (HumanStudy-Bench web app)
-
-Next.js app for:
-- Homepage sections (`Overview`, `Leaderboard`, `Dataset`)
-- Study catalog and study detail pages
-- Contribute flow (upload + PR creation)
-- Docs tab (`/docs`)
-
-## Local preview
-
-```bash
-cd co_website
-npm install
-npm run dev
-```
-
-Open `http://localhost:3000`.
-
-## Build checks
-
-```bash
-cd co_website
-npm run lint
-npm run build
-```
-
-## Study ZIP download
-
-ZIP downloads are served by:
-- `GET /api/studies/:studyId/zip`
-
-This endpoint packages `studies/:studyId` directly from the repository.
-
-## Refreshing effect-size plot data
-
-Homepage effect-size scatter reads:
-- `public/data/effects/gemini_flash_v4_effect_data.json`
-
-To refresh:
-1. From repo root: `python scripts/plot_single_fig3_effects.py --export-data /absolute/path/to/co_website/public/data/effects/gemini_flash_v4_effect_data.json`
-2. Rebuild/redeploy the site.
diff --git a/co_website/adm-zip.d.ts b/co_website/adm-zip.d.ts
deleted file mode 100644
index 1caa077..0000000
--- a/co_website/adm-zip.d.ts
+++ /dev/null
@@ -1,14 +0,0 @@
-declare module "adm-zip" {
- interface ZipEntry {
- isDirectory: boolean;
- entryName: string;
- getData(): Buffer;
- }
-
- class AdmZip {
- constructor(buffer?: Buffer);
- getEntries(): ZipEntry[];
- }
-
- export default AdmZip;
-}
diff --git a/co_website/app/api/contribute/upload/route.ts b/co_website/app/api/contribute/upload/route.ts
deleted file mode 100644
index 3aa42a0..0000000
--- a/co_website/app/api/contribute/upload/route.ts
+++ /dev/null
@@ -1,288 +0,0 @@
-import { NextRequest, NextResponse } from "next/server";
-
-const MAX_ZIP_BYTES = 50 * 1024 * 1024; // 50MB
-
-function getStudyIdFromPath(entryName: string): string | null {
- const m = entryName.match(/^(?:data\/studies\/)?(study_\d{3})\//) || entryName.match(/^(study_\d{3})\//);
- return m ? m[1] : null;
-}
-
-function parseJson(buf: Buffer): unknown {
- return JSON.parse(buf.toString("utf8"));
-}
-
-function normalizeGithub(input: string): string {
- const raw = input.trim();
- if (!raw) return "";
- if (raw.startsWith("http://") || raw.startsWith("https://")) return raw;
- return `https://github.com/${raw.replace(/^@/, "")}`;
-}
-
-type ValidationResult = { ok: true; studyId: string; files: MapCitation content will appear here.
-- Add a new study to HumanStudy-Bench. -
-HumanStudy-Bench is a non-profit and open-source platform.
-We welcome collaborators from all backgrounds!
-All accepted contributions will be acknowledged and credited.
-- There are two ways to submit a study. We recommend contributing via GitHub. -
- -contrib-yourgithubid-013)
- studies/ with index.json, source/, scripts/, and README.md
- bash scripts/verify_study.sh study_XXX and open a PR
-
- Maintainers assign final study numbering by merge order. Include your GitHub ID in index.json contributors.
-
- Upload a .zip of your study and we create a GitHub PR for you. -
-- Questions? Open an issue on{" "} - - GitHub - - . For other inquiries, contact{" "} - - xul049@ucsd.edu - - . -
-- - Contribute - - {" / "} - Docs -
-- {meta.description} -
-Previous
-- {prev[1].title} -
-Next
-- {next[1].title} -
-- {authorsStr} - {yearStr ? ` (${yearStr})` : ""} -
-{study.study_id}
- - -{study.description}
-
- Path:{" "}
- studies/{study.study_id}/
-
- Standardized human study replays. -
-- An initial curated collection of 12 foundational human-subject studies spanning individual cognition, strategic interaction, and social psychology, all with complete experimental materials and clearly specified statistical tests. -
- - Become a contributor → - -Loading effect size data…
-- A non-profit and open-source platform converting human study research papers into standardized testbed for AI agents to replay human-subject experiments end-to-end, evaluating agent alignment with human participants at the level of scientific inference. -
-| Rank | -Model | -Variant | -PAS (Alignment) | -ECS | -Study Scores | -Cost | -Tokens | -Details | -
|---|---|---|---|---|---|---|---|---|
| - {index + 1} - | -- {modelShortName(row.model)} - | -- {row.variant} - | -- 75 ? 'text-green-700 bg-green-50' : - row.overall_alignment > 50 ? 'text-yellow-700 bg-yellow-50' : 'text-red-700 bg-red-50' - }`}> - {formatScore(row.overall_alignment)} - - | -- {row.overall_ecs} - | -
-
- {/* Only showing first 12 studies to save space if dataset grows */}
- {Array.from({length: 12}).map((_, i) => {
- const studyId = `study_${String(i+1).padStart(3, '0')}`;
- const score = row.study_scores[studyId] ? row.study_scores[studyId] * 100 : 0;
- const hasData = !!row.study_scores[studyId];
-
- return (
-
- )
- })}
-
- |
- - {formatCost(row.total_cost)} - | -- {formatInteger(row.total_tokens)} - | -- - {expandedRow === row.id ? 'Hide' : 'Show'} - - | -
|
-
- {row.raw_data.sort((a,b) => a.study_id.localeCompare(b.study_id)).map(studyData => {
- const findingsStr = studyData.findings_breakdown
- ? Object.entries(studyData.findings_breakdown)
- .map(([k, v]) => `${k} (${v})`)
- .join(', ')
- : '';
- return (
-
-
-
- );
- })}
-
-
- {studyData.study_id}
- {studyData.title}
- {findingsStr && (
-
- Findings: {findingsStr}
-
- )}
-
-
-
- |
-
- Evaluating agent design alignment with human behavior using Probability Alignment Score (PAS) and Effect Consistency Score (ECS) across 12 foundational human-subject studies. -
-- HumanStudy-Bench treats participant simulation as an agent design problem and provides a standardized testbed — combining an Execution Engine that reconstructs full experimental protocols from published studies and a Benchmark with standardized evaluation metrics — for replaying human-subject experiments end-to-end with alignment evaluation at the level of scientific inference. -
-Test different agent designs on the same experiments, run agents through real studies covering 6,000+ trials, and compare results rigorously using inferential-level metrics.
-- From published human studies to reusable simulation environment in four stages. -
-Curates human studies that are scientifically important and practically reproducible, ensuring full experimental details, quantifiable outcomes, and simulation feasibility.
-Extracts participants' profiles, experimental designs, statistical tests, and human ground-truth outcomes from unstructured papers into machine-executable representations.
-Runs agent designs through reconstructed experimental protocols, generating trial-level data via a shared execution engine that handles agent sampling, instruction dispatch, and response collection.
-Compares agent responses against human ground-truth using Probability Alignment Score (PAS) for inferential agreement and Effect Consistency Score (ECS) for effect-size alignment.
-Measures whether agents reach the same scientific conclusions as humans at the phenomenon level. It quantifies the probability that agent and human populations exhibit behavior consistent with the same hypothesis.
-Measures how closely agents reproduce the magnitude and pattern of human behavioral effects at the data level. It assesses both the precision and accuracy of agent responses compared to human ground truth.
-- Looking for a starting point or more instructions? Head over to{" "} - - Templates - {" "} - or the{" "} - - Learning - {" "} - center. -
-