diff --git a/app/public/paper/policybench.pdf b/app/public/paper/policybench.pdf index bc7f7329..bd26b5cc 100644 Binary files a/app/public/paper/policybench.pdf and b/app/public/paper/policybench.pdf differ diff --git a/app/public/paper/web/index.html b/app/public/paper/web/index.html index 1de6a8b4..ffecd0a0 100644 --- a/app/public/paper/web/index.html +++ b/app/public/paper/web/index.html @@ -299,7 +299,7 @@

Frozen snapshot an -
+
@@ -818,7 +818,7 @@

Reference-output credibility

PolicyBench rebuilt the references on 2026-09-29 with policyengine-us 2.15.17, the newest release when it began sweeping the references that day (uploaded 00:23 UTC), in place of 1.755.4. It computes each scored reference with policyengine_us.Simulation. The snapshot also records policyengine.py 6.1.2 for provenance; its certified US bundle carries an older policyengine-us, and policyengine.py does not load beside 2.15.17. PolicyBench ported the publication conventions to the new version. The SNAP convention holds each FY2026 figure directly for October to December 2026, because the new version carries USDA’s FY2027 figures as published values. policyengine-us 2.15.17 counts Maryland county income tax in its state income tax. PolicyBench’s state income tax output leaves local tax out, so an adapter restores that scope, and the county tax stays in the federal state and local tax deduction. policyengine-us #9261 changed from 40 hours to 0 the default of the weekly-hours input that SNAP’s work rules read, so the scenario builder also passes each prompt’s stated usual weekly hours under that input’s name. After the port, PolicyBench grouped every output that still moved by root cause. An investigator settled each root cause from the upstream commits, the engine code of both versions and primary legal sources with their publication dates, and an independent reviewer tried to refute each settlement. The move changes 4 scored references (Table 4). Two take law published before the freeze that policyengine-us 1.755.4 did not encode: New Jersey’s child tax credit schedule for 2026 to 2028 (P.L.2026, c.26, approved June 30, 2026), and Arizona’s broad-based categorical eligibility limit for SNAP, which Arizona raised from 185 to 200 percent of the poverty guideline starting with benefit month March 2026. Two follow engine corrections: policyengine-us added child support received to school-meal income (7 CFR 245.6(a)(5)(ii)), and policyengine-us #9425 corrected the rounding in New York’s Empire State child credit phase-out. The investigation found, and PolicyBench excluded, 3 federal income tax outputs that depend on an unlisted input: the new version counts a listed state and local tax refund as income (policyengine-us #9422), and whether any of it is income depends on whether the refunded tax reduced federal tax in the year the household paid it (26 U.S.C. 111(a)), which the prompt does not say. The re-review of a California household’s excluded SNAP output also flagged a scored output the move did not change: the head’s Medicaid eligibility. The head’s income is above the 138 percent limit for the adult expansion group, so only a disability pathway leads to Medi-Cal, and California’s Working Disabled Program requires SSI’s definition of disability, the unlisted input that already excludes the household’s SNAP. policyengine-us 2.15.17 tests the general disability flag there and gives 1 under either reading; with the program’s test reading SSI’s definition, the head qualifies only by meeting it, and PolicyBench excluded the output. Another 2 references move by less than the dollar tolerance. PolicyBench re-reviewed the 19 excluded outputs whose values move on 2.15.17; all stay excluded, each with the value it was decided on. policyengine-us 2.17.0, the newest release when PolicyBench checked PyPI on 2026-09-29 at 14:58 UTC, gives the same value as 2.15.17 for all 1,984 outputs under the same conventions and adapter. The reference sidecar lists every change, and reference_audit/2026-09-28 holds the recomputation of every output, the investigations and the reviews.

A review of release dashboard-data-20260930 on 2026-10-05 found three more inputs that no prompt states behind scored references; no judge had flagged the scored outputs they move (reference_audit/2026-10-05, reference_audit/2026-10-05-medicare-part-b and reference_audit/2026-10-05-payroll). The first is the state income tax in the federal state and local tax deduction, which takes the state income tax a household paid during the year, or general sales tax instead (26 U.S.C. 164(a)(3) and (b)(5)). No prompt states state income tax withheld or paid. policyengine-us fills that input with state_withheld_income_tax, a formula estimate of withholding on the person’s whole federal adjusted gross income rather than the household’s state liability, and the references take the estimate. Read as the household paying its 2026 state income tax during the year, or as paying none, the federal income tax of five households moves, three of them scored. The second is Medicare enrollment and a Part B premium. policyengine-us treats every Medicare-eligible person as enrolled and adds the modeled Part B premium, $2,434.80 for 2026 where the Medicare Savings Program does not pay it, to medical expenses; no prompt states either. The prompt’s instruction to assume program take-up supports the reference, and its rules that unlisted numbers are zero and that unlisted expenses and health coverage are not to be inferred support the other reading. Without the premium, the outputs of one household move: it itemizes, so its federal income tax moves, and Virginia’s itemized deductions start from the federal ones, so its Virginia income tax moves too. The third is whether the employer deducts the employee share of a state paid-leave or disability premium. The payroll output counts mandatory employee state payroll taxes. In Minnesota, Colorado, Massachusetts and New York the law lets the employer deduct an employee share of such a premium from wages, up to a cap, but does not require it, so the employee pays from $0 to the cap at the employer’s choice. The four payroll references from those states count the cap, and the prompt does not state the employer’s choice; read as the amount the law requires of the employee, each is employee federal payroll tax alone. California’s, Connecticut’s, New Jersey’s and Pennsylvania’s employee contributions are owed by the worker and withheld by the employer, and the review left those references as they were. Each audit recomputed every output on policyengine-us 2.15.17 with the publication conventions, under each reading, and each sweep’s baseline reproduces all 1,928 references that release scored. Beyond the one-dollar tolerance, the sweeps move eight scored outputs, which PolicyBench excluded, and two federal income tax outputs already excluded for other reasons; they move no output that is still scored by any amount. The eight records are computed on policyengine-us 2.15.17, whose values are the references.

On 2026-10-06 PolicyBench ruled to exclude ten more outputs. Eight came from a reference adversary (reference_audit/2026-10-05-reference-adversary). It flags the scored outputs where many models, or several of the strongest, agree on an answer the reference does not give. A judge that cannot yet see how the engine derived the reference works each flagged output from primary law, and only then reconciles its answer with the derivation. The adversary flagged 61 outputs and ran 52, the other nine belonging to other audits, and it held 45 references. An independent check of its seven other verdicts confirmed four as engine defects: Arizona’s 2026 standard deduction left unindexed (A.R.S. 43-1041(H)), Ohio’s medical deduction counting health insurance premiums only for a Medicare-eligible filer with no employer-paid plan (R.C. 5747.01(A)(10)), Colorado’s 2026 sales tax refund paid without the excess state revenues that C.R.S. 39-22-2003(2) requires, and New York’s 2026 child and dependent care credit, which Tax Law 606(c-2) replaced. The check refuted two, the Medicaid eligibility of two children in a North Carolina household, whose references stand. The seventh is a question of definition. A Pennsylvania household’s dependent earns $45,000 of wages and must file their own return, and the output definitions name no return, so the household’s federal and state income tax can be read with or without that return; a scan for the same question found a Missouri household, and PolicyBench excluded the four outputs. The other two outputs are two Louisiana households’ 2026 state income tax (reference_audit/2026-10-05-louisiana). Their references rest on a standard deduction that policyengine-us computed from published price indexes under Louisiana’s formula; Louisiana published the return amount itself on 2026-09-28, after the freeze. All ten records are computed on policyengine-us 2.15.17.

-

On 2026-10-10 PolicyBench moved the references again, from policyengine-us 2.15.17 to 2.38.6, a release that fixes engine defects behind excluded outputs. With the same publication conventions and adapter, it recomputed every output (reference_audit/2026-10-09-engine-upgrade). policyengine-us 2.38.6 was the newest release when PolicyBench began sweeping the references on 2026-10-10 (uploaded 05:39 UTC). It was still the newest release when PolicyBench checked PyPI on 2026-10-10 at 06:53 UTC. The move fixes the engine defects behind 18 outputs. Earlier releases excluded 14 of them, which return to scoring at the new version’s values; the other four, ruled on 2026-10-06 while still scored, stay scored at those values. Of the 18, 16 land within $1 of the corrected values their exclusion records carry. The other two are held instead to the values the audit’s fix modules give on policyengine-us 2.37.2, an engine that still has the defects; each lands within $1 of its own. The reference sidecar records each target with the upstream fix it names (regenerated_exclusions). The restored outputs take the upstream fixes for the IRA deduction’s compensation limit and a dependent’s contributions (policyengine-us #10032, #9801; three outputs), the IRA deduction’s active-participant phase-out and California’s itemized deduction conformity (policyengine-us #10032, #10034), New Jersey’s worker unemployment and workforce contributions (policyengine-us #10031), Arizona’s standard deduction indexing (policyengine-us #9928), California’s itemized deduction conformity (policyengine-us #10034), Ohio’s medical deduction for health insurance premiums (policyengine-us #10020), estate income in gross income (policyengine-us #10027, #9633; two outputs), Colorado’s 2026 sales tax refund (policyengine-us #9946), the IRA deduction’s active-participant phase-out (policyengine-us #10032; four outputs), New York’s 2026 child and dependent care credit (policyengine-us #9948) and the IRA deduction’s active-participant phase-out and estate income in gross income (policyengine-us #10032, #10027; two outputs). The move changes one scored references beyond the dollar tolerance and 2 within it, and removes two outputs from scoring. The changed scored reference follows Idaho’s $10 permanent building fund tax (Idaho Code 63-3082), which policyengine-us now counts in state income tax (policyengine-us #9810). The two outputs that leave scoring depend on an input no prompt states: county of residence (Indiana county income tax). PolicyBench re-reviewed the 17 excluded outputs whose values move on 2.38.6; all stay excluded, each with the value it was decided on. The 58 excluded outputs now keep the values they were decided on (38 computed with policyengine-us 1.755.4, 18 with 2.15.17, 2 with 2.38.6).

+

On 2026-10-10 PolicyBench moved the references again, from policyengine-us 2.15.17 to 2.38.6, a release that fixes engine defects behind excluded outputs. With the same publication conventions and adapter, it recomputed every output (reference_audit/2026-10-09-engine-upgrade). policyengine-us 2.38.6 was the newest release when PolicyBench began sweeping the references on 2026-10-10 (uploaded 05:39 UTC). It was still the newest release when PolicyBench checked PyPI on 2026-10-10 at 06:53 UTC. The move fixes the engine defects behind 18 outputs. Earlier releases excluded 14 of them, which return to scoring at the new version’s values; the other four, ruled on 2026-10-06 while still scored, stay scored at those values. Of the 18, 16 land within $1 of the corrected values their exclusion records carry. The other two are held instead to the values the audit’s fix modules give on policyengine-us 2.37.2, an engine that still has the defects; each lands within $1 of its own. The reference sidecar records each target with the upstream fix it names (regenerated_exclusions). The fixed outputs take the upstream fixes for the IRA deduction’s compensation limit and a dependent’s contributions (policyengine-us #10032, #9801; three outputs), the IRA deduction’s active-participant phase-out and California’s itemized deduction conformity (policyengine-us #10032, #10034), New Jersey’s worker unemployment and workforce contributions (policyengine-us #10031), Arizona’s standard deduction indexing (policyengine-us #9928), California’s itemized deduction conformity (policyengine-us #10034), Ohio’s medical deduction for health insurance premiums (policyengine-us #10020), estate income in gross income (policyengine-us #10027, #9633; two outputs), Colorado’s 2026 sales tax refund (policyengine-us #9946), the IRA deduction’s active-participant phase-out (policyengine-us #10032; four outputs), New York’s 2026 child and dependent care credit (policyengine-us #9948) and the IRA deduction’s active-participant phase-out and estate income in gross income (policyengine-us #10032, #10027; two outputs). The move changes one scored references beyond the dollar tolerance and 2 within it, and removes two outputs from scoring. The changed scored reference follows Idaho’s $10 permanent building fund tax (Idaho Code 63-3082), which policyengine-us now counts in state income tax (policyengine-us #9810). The two outputs that leave scoring depend on an input no prompt states: county of residence (Indiana county income tax). PolicyBench re-reviewed the 17 excluded outputs whose values move on 2.38.6; all stay excluded, each with the value it was decided on. The 58 excluded outputs now keep the values they were decided on (38 computed with policyengine-us 1.755.4, 18 with 2.15.17, 2 with 2.38.6).

The audit tooling itself required hardening, which we report because it changes how much the annotations can be trusted. The derivation narratives displayed alongside reference values are generated from engine computation traces. After earlier model-written narratives were found to misattribute eligibility pathways, mechanical validators now enforce that each narrative states the engine’s actual mechanism — for example, that a child ineligible for CHIP because they qualify for Medicaid is never described as failing a CHIP income test. The failure-audit judge reads these grounded narratives together with engine facts and must return a per-model diagnosis naming the specific rule or computation the model missed. It is barred from re-adjudicating the reference in prose: hedged or reference-adjudicating verdicts are mechanically rejected and re-judged, and genuine reference doubt is routed to a structured flag that triggers the manual adjudication described above.

After freezing the snapshot and completing response-contract repairs, we annotated the 8,001 rows whose legacy threshold score is below 1 and their scenario-output cases. Table 5 reports the final classifications for that legacy-threshold audit universe. It contains 7,997 of the snapshot’s 7,997 exact-match misses and 4 exact hits. The 2,138 other rows with bounded score below 100 fall outside the selection and have no audit annotation. Each annotated row on a scored output resolves to one of two final classes: llm_error (a substantive miscalculation, with a per-model diagnosis naming the rule or computation the model missed) or parse_contract_failure (a missing or unparsable answer, classified deterministically). Three further classes describe outputs the snapshot does not score, and each such output is removed from scoring for every model, so no model gains or loses from it. prompt_ambiguity marks 42 outputs whose reference depends on an input the prompt never lists, so a careful reader could take the stated facts either way; for each, the reference was recomputed under the other reading with the engine version that produced it, and it moved (the California Medicaid output once the engine’s Working Disabled Program test read SSI’s definition, as the program does). reference_engine_defect marks the 14 outputs where that engine misapplies the law on stated facts. policyengine-us 2.38.6 computes four of them at their corrected values; they stay excluded because each also moves under an input the prompt does not state. policyengine-us 2.38.6 computes one more at its recorded corrected value, but its reference rests on a further engine defect, found on 2.38.6 and not fixed there, so it stays excluded. reference_later_law marks the two Louisiana outputs whose reference rests on an amount the state published after the freeze. Together they remove 58 outputs in 42 households, and of the 2,212 annotated rows on those outputs, 2,164 carry one of these classes as description rather than as a scored class and the rest are answers that never parsed. A scored reference also follows from law published before the references were frozen on 2026-07-03. USDA published the FY2027 SNAP cost-of-living figures on 2026-08-21, after the freeze and after most rows were answered, so grading any model on them would score it against law it could not have seen; SNAP’s October to December months therefore hold the FY2026 figures, and the publication conventions described above apply the same rule to every projected amount. Every model is therefore scored on 1,926 of its 1,984 requested outputs. The exclusion record (reference_exclusions.json) travels with the frozen run and the manifest pins it. No scored row remains classified as a prompt ambiguity, unresolved reference issue, or needs-review item, and zero annotated rows carry a standing reference-suspect flag. The audit pipeline is model-assisted, with developer adjudication of each reference-suspect or prompt-ambiguity flag rather than independent external validation; this snapshot records 70 cases of developer adjudication, kept beside the annotations with the judge’s verdict and the reasoning. Three judge models produced the verdicts, and all are board rows: GPT-5.6 Sol, called through the Codex CLI, judged the cases whose verdicts predate the September 2026 additions; Claude Opus 5, called through the Claude Code CLI, judged the 94 cases the September 5 additions joined that no later judge re-judged; and Claude Opus 5.5 judged the cases the September 22, September 29, September 30 and October 9 additions joined or a reference revision changed (314 of 677 cases), its own row’s cases included. Each verdict is recorded in a per-case provenance sidecar in the audit tree. Judge verdicts classify misses after scoring and change no score, and every reference flag a judge raised is settled by the developer adjudication above. The unlisted inputs behind the unlisted-input exclusions are the SSI disability criterion (meets_ssi_disability_criteria), false for every person in the certified June 2026 microdata build; months of SSDI receipt (months_receiving_social_security_disability), never carried by the data and integer-typed, which the prompt builder does not expose; weekly hours worked (weekly_hours_worked_before_lsr), which policyengine-us 1.755.4 defaulted to 40 where the prompt treats unlisted numbers as zero; whether survivor benefits other than Social Security are a pension or annuity, which the prompt does not say and which decides both their taxability and whether they count toward Wisconsin’s homestead credit income; who paid for the coverage behind listed employment disability benefits, which decides whether they are taxable; whether listed home mortgage interest is on the home the household occupies, which decides whether SNAP counts it as a shelter cost; whether an adult tax dependent is the claiming filers’ child, which decides whose income counts in that dependent’s Medicaid household; whether the tax behind a listed state and local tax refund reduced federal tax in the year the household paid it, which decides how much of the refund is income; the state income tax a household withheld or paid during 2026, which the federal state and local tax deduction takes as its state income tax part and which policyengine-us estimates from federal adjusted gross income; Medicare enrollment and a Part B premium, which policyengine-us assumes for every Medicare-eligible person and counts as a medical expense; whether the employer deducts the employee share of a state paid-leave or disability premium that the law lets it deduct but does not require, which decides whether that share is a mandatory employee state payroll tax; and whether listed interest comes from a Massachusetts bank deposit, which moves only an output already recorded as an engine-defect exclusion. One definition was also open: the output definition of federal income tax before refundable credits does not say whether it includes the net investment income tax, which the reference adds and Form 1040 reports after its line for tax after nonrefundable credits, so the outputs that tax moves are excluded as well. The prompt lists a general is disabled fact and never a program-specific determination (Section 4.2). Every exclusion but the California Medicaid output and the 2026-10-06 records was derived mechanically rather than from which models missed: each is an output that a sweep of every reference, under an alternative reading or an engine defect’s sandbox fix, moved on policyengine-us 1.755.4 or, for the state and local tax refund reading and the readings of the 2026-10-05 review, on 2.15.17; the outputs the 2026-10-10 upgrade excluded are ones policyengine-us 2.38.6 itself moved, onto an input no prompt states. The 2026-10-06 records are the exception by design. The reference adversary selects the outputs where models agree against the reference, and an independent check against primary law then decided each; the two Louisiana records came from a sweep of that state’s 2026 standard deduction. On 1.755.4, PolicyBench excluded every output an alternative reading of the September 22 audit moved and every output a defect not fixed upstream moved by more than a dollar, which is why the set includes three SNAP outputs that move through the elderly-or-disabled SNAP rules (in two of those households SSI itself stays at zero; in the third an aged member already receives SSI and the disabled member would add a second benefit). On 2.15.17, PolicyBench re-ran four of those sweeps (the IRA deduction limit fix, the net investment income tax definition, and the readings for mortgage residence and 40 unlisted weekly hours) and ran a new one for the state and local tax refund reading. Beyond the one-dollar exact-match tolerance, the new sweep moves three previously scored federal income tax outputs, and PolicyBench excluded them. Set against the same 2.15.17 calculation without its fix or reading, no sweep moves a scored output by more than the tolerance, and three scored outputs move by less. PolicyBench did not re-run the other sweeps on 2.15.17. On 2.15.17, the California Medicaid output stays at 1 under either reading of SSI’s disability criterion, because the engine’s Working Disabled Program test reads the general flag in its place. PolicyBench found that output when it re-reviewed the household’s excluded SNAP output, and excluded it because the program itself requires SSI’s definition of disability.

@@ -891,7 +891,7 @@

Reference-output credibility US references on policyengine-us 2.38.6 (October 2026 upgrade) -The move from policyengine-us 2.15.17 fixes the engine defects behind 18 outputs (14 that earlier releases excluded return to scoring at the new version’s values; the 4 ruled on 2026-10-06 while still scored stay scored) and changes 1 scored reference beyond the dollar tolerance; 2 outputs leave scoring. PolicyBench re-reviewed the 17 excluded outputs whose values move; all stay excluded. The restored outputs take the upstream fixes for the IRA deduction’s compensation limit and a dependent’s contributions (policyengine-us #10032, #9801; three outputs), the IRA deduction’s active-participant phase-out and California’s itemized deduction conformity (policyengine-us #10032, #10034), New Jersey’s worker unemployment and workforce contributions (policyengine-us #10031), Arizona’s standard deduction indexing (policyengine-us #9928), California’s itemized deduction conformity (policyengine-us #10034), Ohio’s medical deduction for health insurance premiums (policyengine-us #10020), estate income in gross income (policyengine-us #10027, #9633; two outputs), Colorado’s 2026 sales tax refund (policyengine-us #9946), the IRA deduction’s active-participant phase-out (policyengine-us #10032; four outputs), New York’s 2026 child and dependent care credit (policyengine-us #9948) and the IRA deduction’s active-participant phase-out and estate income in gross income (policyengine-us #10032, #10027; two outputs). The changed scored reference follows Idaho’s $10 permanent building fund tax (Idaho Code 63-3082), which policyengine-us now counts in state income tax (policyengine-us #9810). The two outputs that leave scoring depend on an input no prompt states: county of residence (Indiana county income tax). +The move from policyengine-us 2.15.17 fixes the engine defects behind 18 outputs (14 that earlier releases excluded return to scoring at the new version’s values; the 4 ruled on 2026-10-06 while still scored stay scored) and changes 1 scored reference beyond the dollar tolerance; 2 outputs leave scoring. PolicyBench re-reviewed the 17 excluded outputs whose values move; all stay excluded. The fixed outputs take the upstream fixes for the IRA deduction’s compensation limit and a dependent’s contributions (policyengine-us #10032, #9801; three outputs), the IRA deduction’s active-participant phase-out and California’s itemized deduction conformity (policyengine-us #10032, #10034), New Jersey’s worker unemployment and workforce contributions (policyengine-us #10031), Arizona’s standard deduction indexing (policyengine-us #9928), California’s itemized deduction conformity (policyengine-us #10034), Ohio’s medical deduction for health insurance premiums (policyengine-us #10020), estate income in gross income (policyengine-us #10027, #9633; two outputs), Colorado’s 2026 sales tax refund (policyengine-us #9946), the IRA deduction’s active-participant phase-out (policyengine-us #10032; four outputs), New York’s 2026 child and dependent care credit (policyengine-us #9948) and the IRA deduction’s active-participant phase-out and estate income in gross income (policyengine-us #10032, #10027; two outputs). The changed scored reference follows Idaho’s $10 permanent building fund tax (Idaho Code 63-3082), which policyengine-us now counts in state income tax (policyengine-us #9810). The two outputs that leave scoring depend on an input no prompt states: county of residence (Indiana county income tax). US federal and Virginia income tax (October 2026 review) diff --git a/app/src/paperSnapshot.json b/app/src/paperSnapshot.json index f8af53e9..fc3f5300 100644 --- a/app/src/paperSnapshot.json +++ b/app/src/paperSnapshot.json @@ -1,6 +1,6 @@ { - "pdfVersion": "1607c71abaf3", + "pdfVersion": "f15264138ec2", "responseWindow": "June 12 and October 8, 2026", "snapshotDate": "2026-10-10", - "webVersion": "31cd4141373a" + "webVersion": "33be2303d996" } diff --git a/paper/index.qmd b/paper/index.qmd index 447d714e..f8932d45 100644 --- a/paper/index.qmd +++ b/paper/index.qmd @@ -892,7 +892,7 @@ reference_audit = pd.DataFrame( ["US SNAP, SSI and CHIP (September 2026 wave)", "Whether listed mortgage interest is on the occupied home, unlisted hours worked, survivor-benefit taxability, and whether an adult tax dependent is the filers' child (three Medicaid outputs) excluded as unlisted inputs. SNAP October-December references regenerated with the FY2026 figures, the last published before the freeze. SNAP references regenerated for every month of 2026 with the engine's SNAP rounding defects fixed as policyengine-us fixed them after the freeze (cents in the allotment, an unrounded minimum benefit, floored net income, unrounded income standards). One Michigan SNAP reference regenerated with the fix for the SNAP child support option, whose engine values carried the opposite meaning (Michigan deducts child support from net income, and the engine excluded it from gross income; fixed upstream in policyengine-us #9586 on 2026-09-24). The heat-and-eat utility allowance granted without an elderly or disabled member, which P.L. 119-21 ended, excluded as an engine defect. The CHIP flag dissolved: employer coverage bars CHIP."], ["US projected 2026 parameters (September 2026 wave)", "No judge flag: a recomputation with every projected 2026 value held found references resting on price-index projections. Wisconsin, Minnesota, Michigan and Missouri amounts and Maryland's withholding allowance had been published before the freeze; California's indexed amounts, the IRS sales tax tables, Idaho's threshold and Maryland's return deduction had not. Scored references regenerated with the published or last-published amounts; Wisconsin's reach only the corrected values of Wisconsin outputs excluded for engine defects."], [f"US references on policyengine-us {r.september_upgrade.engine_version} (September 2026 upgrade)", f"Once PolicyBench had ported the conventions, an investigator traced every output that still moved between policyengine-us {r.september_upgrade.previous_engine_version} and {r.september_upgrade.engine_version} to its upstream change and settled it against primary sources and their publication dates. An independent reviewer tried to refute each settlement. The upgrade changes {r.september_upgrade.scored_change_count} scored references, for law published before the freeze that {r.september_upgrade.previous_engine_version} did not encode or misapplied: New Jersey's 2026-2028 child tax credit schedule (P.L.2026, c.26), Arizona's 200 percent SNAP categorical eligibility limit from March 2026, child support received counted as income for school meals (7 CFR 245.6(a)(5)(ii)), and New York's Empire State child credit phase-out rounding. The investigation found, and PolicyBench excluded, {r.september_upgrade.new_exclusion_count} federal income tax outputs that depend on an unlisted input: {r.september_upgrade.engine_version} counts a listed state and local tax refund as income, which is right only if the refunded tax reduced federal tax in the prior year (26 U.S.C. 111(a)). PolicyBench re-reviewed the {r.september_upgrade.rechecked_count} excluded outputs that move; all stay excluded. The re-review of a California household's excluded SNAP output flagged its head's Medicaid eligibility, which turns on the same unlisted input, SSI's disability criterion, through the Working Disabled Program; PolicyBench excluded it."], - *([[f"US references on policyengine-us {r.policyengine_us_version} (October 2026 upgrade)", f"The move from policyengine-us {r.previous_policyengine_us_version} fixes the engine defects behind {r.engine_upgrade_restored_count} outputs ({r.engine_upgrade_restored_count - r.engine_upgrade_restored_ruled_count} that earlier releases excluded return to scoring at the new version's values; the {r.engine_upgrade_restored_ruled_count} ruled on {r.ruling_date} while still scored stay scored) and changes {r.engine_upgrade_scored_change_count} scored {'reference' if r.engine_upgrade_scored_change_count == 1 else 'references'} beyond the dollar tolerance; {r.engine_upgrade_new_exclusion_count} {'output leaves' if r.engine_upgrade_new_exclusion_count == 1 else 'outputs leave'} scoring. PolicyBench re-reviewed the {r.engine_upgrade_rechecked_count} excluded outputs whose values move; all stay excluded. {r.engine_upgrade_restored_sentence} {r.engine_upgrade_scored_change_sentence} {r.engine_upgrade_new_exclusion_sentence}"]] if r.engine_upgrade_count > 1 else []), + *([[f"US references on policyengine-us {r.policyengine_us_version} (October 2026 upgrade)", f"The move from policyengine-us {r.previous_policyengine_us_version} {r.engine_upgrade_restored_split_clause} and changes {r.engine_upgrade_scored_change_count} scored {'reference' if r.engine_upgrade_scored_change_count == 1 else 'references'} beyond the dollar tolerance; {r.engine_upgrade_new_exclusion_count} {'output leaves' if r.engine_upgrade_new_exclusion_count == 1 else 'outputs leave'} scoring. PolicyBench re-reviewed the {r.engine_upgrade_rechecked_count} excluded outputs whose values move; all stay excluded. {r.engine_upgrade_restored_sentence} {r.engine_upgrade_scored_change_sentence} {r.engine_upgrade_new_exclusion_sentence}"]] if r.engine_upgrade_count > 1 else []), ["US federal and Virginia income tax (October 2026 review)", f"No judge flag: a review of release dashboard-data-20260930 swept every reference on policyengine-us {r.review_exclusion_engine_version} under readings of two inputs no prompt states. The state income tax part of the federal state and local tax deduction is policyengine-us's formula estimate of state income tax withheld on federal adjusted gross income; read as the household paying its 2026 liability during the year, or paying none, {r.review_sweep_newly_excluded_count_word('salt_withholding')} scored federal income tax outputs move, and PolicyBench excluded them; the readings also move {r.review_sweep_already_excluded_count_word('salt_withholding')} federal outputs already excluded. policyengine-us treats every Medicare-eligible person as enrolled and counts a modeled Part B premium as a medical expense; without it, {r.review_sweep_moved_household_count_word('medicare_part_b')} household's federal and Virginia income tax move, and both are excluded."], ], columns=["Discrepancy class", "Review outcome"], diff --git a/paper/snapshot/20260501/manifest.json b/paper/snapshot/20260501/manifest.json index 35abfa10..dd479b7a 100644 --- a/paper/snapshot/20260501/manifest.json +++ b/paper/snapshot/20260501/manifest.json @@ -169,12 +169,12 @@ "rendered_paper_artifacts": { "pdf": { "path": "app/public/paper/policybench.pdf", - "sha256": "1607c71abaf35fae6f6d51576450c91747f8fb158485fc6ea64431e4c4c683c3" + "sha256": "f15264138ec20def17884219075533bd2604a7ba879f3bd11c8973b800f97c23" }, "web": { "files": { "figures/positive_zero_scatter.png": "f13ca00d6a6a2232ee658733507d6e0d39dfc8f639b74ec4b3f016c85acc7fb6", - "index.html": "31cd4141373acbcd2d474841f9e4dbf488df8dd33e62b70699a5ffe47bd42551", + "index.html": "33be2303d996e4cbd9271c528a3836c76560d6613db42f2193ed98a125717f9e", "pe-tokens.css": "8f24d8da26f583c8ffddffcdcd172b6d52cbecfec20eda55bd39d7aa829f41d8", "policybench-theme.css": "0e12c5fd615558259e5bce0167a38424e54f9ceb280666c4afd660d759cd1cb9", "site_libs/clipboard/clipboard.min.js": "e17a1d816e13c0826e0ed7febfabc3277f45571234bde0bf9120829a7169edc9", diff --git a/policybench/paper_results.py b/policybench/paper_results.py index 7375450c..59b4338f 100644 --- a/policybench/paper_results.py +++ b/policybench/paper_results.py @@ -288,8 +288,10 @@ def partition_engine_upgrade_changes( """Split an engine upgrade's changed outputs into four disjoint groups. ``excluded`` is the exclusion record once the upgrade was decided, and - ``restored`` the excluded outputs the upgrade returned to scoring at the - new engine's values. A change to an excluded output is a new exclusion; a + ``restored`` the outputs whose engine defects the upgrade fixes, scored at + the new engine's values (those a published board excluded return to + scoring; any a ruling decided while still scored stay scored). A change to + an excluded output is a new exclusion; a change to a restored output is a restoration. Any other change is scored, and moves beyond the exact-match tolerance ($1 for an amount, any change for a 0/1 flag) or within it. Every change lands in exactly one group, so @@ -389,6 +391,49 @@ def restored_split_sentence(restored: int, ruled: int, ruling_date: str) -> str: ) +def restored_split_clause(restored: int, ruled: int, ruling_date: str) -> str: + """The upgrades table's form of ``restored_split_sentence``: a verb phrase + finishing "The move from policyengine-us X ...", with numerals as the rest + of that cell writes counts. It names the ``restored`` outputs whose engine + defects the upgrade fixes and, in parentheses, splits them as the sentence + does: ``restored - ruled`` that earlier releases excluded return to + scoring, and ``ruled`` that rulings on ``ruling_date`` decided while they + were still scored stay scored. A lone output in either group is "one", not + "1".""" + if not 0 <= ruled <= restored: + raise ValueError(f"{ruled} ruled of {restored} fixed outputs") + if not restored: + return "fixes none of the engine defects behind excluded outputs" + returned = restored - ruled + defects = "defect" if restored == 1 else "defects" + noun = "output" if restored == 1 else "outputs" + lead = f"fixes the engine {defects} behind {restored} {noun}" + if not ruled: + excluded = ( + "an earlier release excluded it, and it returns" + if restored == 1 + else "earlier releases excluded them, and they return" + ) + return f"{lead} ({excluded} to scoring at the new version's values)" + if not returned: + decided = ( + "it while it was still scored, and it stays" + if ruled == 1 + else "them while they were still scored, and they stay" + ) + return ( + f"{lead} (the {ruling_date} rulings decided {decided} scored at the new " + "version's values)" + ) + back = "one that earlier releases excluded returns" + if returned > 1: + back = f"{returned} that earlier releases excluded return" + kept = f"the one ruled on {ruling_date} while still scored stays" + if ruled > 1: + kept = f"the {ruled} ruled on {ruling_date} while still scored stay" + return f"{lead} ({back} to scoring at the new version's values; {kept} scored)" + + def _key(entry: dict) -> tuple[str, str]: return entry["scenario_id"], entry.get("variable", "snap") @@ -430,8 +475,9 @@ class EngineUpgrade: holds those excluded once it was decided: records decided by the end of its date and not removed by then. A record decided after the upgrade never counts here, and a record a later revision removed still does. - ``restored`` holds the outputs the upgrade returned to scoring (its - ``regenerated_exclusions``). + ``restored`` holds the outputs whose engine defects the upgrade fixes (its + ``regenerated_exclusions``): those a published board excluded return to + scoring, and any a ruling decided while still scored stay scored. """ def __init__( @@ -1851,7 +1897,10 @@ def engine_upgrade_new_exclusion_count(self) -> int: @property def engine_upgrade_restored_count(self) -> int: - """Excluded outputs the last upgrade returned to scoring.""" + """Outputs whose engine defects the last upgrade fixes (its + ``regenerated_exclusions``): those a published board excluded return + to scoring, and those the 2026-10-06 rulings decided while still + scored (``engine_upgrade_restored_ruled_count``) stay scored.""" last = self.last_engine_upgrade return 0 if last is None else last.restored_count @@ -1861,7 +1910,7 @@ def engine_upgrade_restored_count_word(self) -> str: @property def engine_upgrade_restored_ruled_count(self) -> int: - """Restored outputs the 2026-10-06 rulings had decided to exclude. No + """Fixed outputs the 2026-10-06 rulings had decided to exclude. No published board excluded them: they were still scored when the rulings came, so they stay scored rather than return to scoring.""" last = self.last_engine_upgrade @@ -1883,6 +1932,16 @@ def engine_upgrade_restored_split_sentence(self) -> str: RULING_DATE, ) + @property + def engine_upgrade_restored_split_clause(self) -> str: + """The same split as a clause for the upgrades table, after "The move + from policyengine-us X".""" + return restored_split_clause( + self.engine_upgrade_restored_count, + self.engine_upgrade_restored_ruled_count, + RULING_DATE, + ) + @cached_property def engine_upgrade_timing(self) -> dict | None: """The last upgrade's sweep timing record, when that upgrade is the @@ -1949,11 +2008,11 @@ def engine_upgrade_timing_sentence(self) -> str: @property def engine_upgrade_restored_fixes(self) -> list[tuple[str, int, list[str]]]: - """The defects behind the outputs the last upgrade returned to scoring: - each one's name, how many outputs it restores and the upstream pull - requests its regenerated_exclusions entries name (not the related - ones), in the sidecar's order. A record with several root causes names - each.""" + """The defects the last upgrade fixes behind its fixed outputs (those + that return to scoring and those still scored alike): each one's name, + how many outputs it fixes and the upstream pull requests its + regenerated_exclusions entries name (not the related ones), in the + sidecar's order. A record with several root causes names each.""" last = self.last_engine_upgrade fixes: dict[str, tuple[int, list[str]]] = {} for entry in [] if last is None else last.revision["regenerated_exclusions"]: @@ -1971,7 +2030,7 @@ def engine_upgrade_restored_fixes(self) -> list[tuple[str, int, list[str]]]: @property def engine_upgrade_restored_fix_phrase(self) -> str: - """The fixes behind the restored outputs as one series, each with the + """The fixes behind the fixed outputs as one series, each with the upstream pull requests it names and, past one, how many outputs.""" parts = [] for label, count, prs in self.engine_upgrade_restored_fixes: @@ -1982,15 +2041,17 @@ def engine_upgrade_restored_fix_phrase(self) -> str: @property def engine_upgrade_restored_sentence(self) -> str: - """One sentence naming the fixes behind the restored outputs.""" + """One sentence naming the fixes behind all the fixed outputs. They + are "fixed", not "restored": the ones the 2026-10-06 rulings decided + while still scored never left scoring.""" phrase = self.engine_upgrade_restored_fix_phrase if not phrase: return "" - return f"The restored outputs take the upstream fixes for {phrase}." + return f"The fixed outputs take the upstream fixes for {phrase}." @property def engine_upgrade_restored_targets(self) -> dict[str, list[dict]]: - """The last upgrade's restored outputs by how their corrected value is + """The last upgrade's fixed outputs by how their corrected value is known (the sidecar's ``target.kind``): ``record`` for the value the exclusion record carries, ``fix_modules`` for the value the audit's fix modules give on a later engine that still has the defect. Refuses, @@ -2007,14 +2068,14 @@ def engine_upgrade_restored_targets(self) -> dict[str, list[dict]]: raise ValueError(f"{key} has an unknown target kind {target['kind']!r}") tolerance = entry["tolerance"] if not 0 <= tolerance <= 1: - raise ValueError(f"{key} is restored on a tolerance of {tolerance}") + raise ValueError(f"{key} is fixed on a tolerance of {tolerance}") variable = entry["variable"] if abs( entry["regenerated"] - target["value"] ) > tolerance or moves_beyond_tolerance( variable, target["value"], entry["regenerated"] ): - raise ValueError(f"{key} is restored off its target") + raise ValueError(f"{key} is fixed off its target") if target["kind"] == "record": if target["value"] != entry["record"]["alternative_value"]: raise ValueError(f"{key}'s target is not its record's value") @@ -2027,7 +2088,7 @@ def engine_upgrade_restored_targets(self) -> dict[str, list[dict]]: @property def engine_upgrade_restored_target_sentence(self) -> str: - """How the restored outputs' corrected values are known: by the + """How the fixed outputs' corrected values are known: by the exclusion record, or by the audit's fix modules on a later engine.""" targets = self.engine_upgrade_restored_targets by_record, by_modules = targets["record"], targets["fix_modules"] diff --git a/tests/test_disclosures.py b/tests/test_disclosures.py index 0ad9047d..5728f34e 100644 --- a/tests/test_disclosures.py +++ b/tests/test_disclosures.py @@ -688,6 +688,25 @@ def test_rendered_pdf_takes_its_engine_times_from_the_timing_record(): assert sentence in text, sentence +def test_rendered_html_calls_the_upgrade_fixes_fixed_not_restored(): + """Round-2 final review nits: the upgrades table renders its split from + restored_split_clause, and the sentence naming the upstream fixes, in the + table and in the October paragraph, calls all the fixed outputs "fixed", + since the ones ruled on 2026-10-06 while still scored never left scoring.""" + text = _html_text() + assert r.engine_upgrade_restored_split_clause in text + sentence = r.engine_upgrade_restored_sentence + assert sentence.startswith("The fixed outputs take the upstream fixes for ") + assert text.count(sentence) == 2 + assert "restored outputs take" not in text + + +def test_rendered_pdf_calls_the_upgrade_fixes_fixed_not_restored(): + text = _straight_quotes(re.sub(r"\s+", " ", _pdf_running_text())) + assert "restored outputs take" not in text + assert "fixed outputs take the upstream fixes for" in text + + def _engine_groups(exclusions) -> list[tuple[str, int]]: """Excluded outputs by the engine version of their value, oldest first.""" by_engine = Counter( @@ -961,11 +980,11 @@ def test_card_counts_follow_the_frozen_records(): r = PaperResults() card = re.sub(r"\s+", " ", BENCHMARK_CARD.read_text()) - # The October move: what it restores, from where, and why. - restored, ruled = r.engine_upgrade_restored_count, r.ruled_regenerated_count + # The October move: what it fixes, from where, and why. + fixed, ruled = r.engine_upgrade_restored_count, r.ruled_regenerated_count assert ( - f"The new version fixes the defects behind {count_word(restored)} outputs. " - f"Release 20261006 excluded {count_word(restored - ruled)} of them, which " + f"The new version fixes the defects behind {count_word(fixed)} outputs. " + f"Release 20261006 excluded {count_word(fixed - ruled)} of them, which " f"return to scoring; the other {count_word(ruled)} are the 2026-10-06 " "defects below, which stay scored." ) in card diff --git a/tests/test_paper_results.py b/tests/test_paper_results.py index e8468798..2b7426e8 100644 --- a/tests/test_paper_results.py +++ b/tests/test_paper_results.py @@ -1566,7 +1566,7 @@ def test_a_count_reads_as_a_word_up_to_ten(n, word): def test_the_upgrade_sentences_name_what_the_sidecar_records(): """The paper names the last upgrade's fixes, change and new exclusions - from the sidecar: every restored output is counted once under its root + from the sidecar: every fixed output is counted once under its root cause's name with the upstream pull request its entry names, every scored change beyond the tolerance under its cause's, and every new exclusion under its record's unlisted input.""" @@ -1578,6 +1578,9 @@ def test_the_upgrade_sentences_name_what_the_sidecar_records(): assert sum(count for _, count, _ in fixes) == len(entries) assert len(entries) == r.engine_upgrade_restored_count sentence = r.engine_upgrade_restored_sentence + # The sentence covers every fixed output, the ruled ones that never left + # scoring included, so it calls them fixed (round-2 final review nit 2). + assert sentence.startswith("The fixed outputs take the upstream fixes for ") for entry in entries: for cause in entry["record"]["root_cause"].split("+"): assert ROOT_CAUSE_LABELS[cause] in sentence @@ -1611,10 +1614,11 @@ def test_the_upgrade_sentences_name_what_the_sidecar_records(): assert f"{{r.{accessor}}}" in paper assert "`{python} r.engine_upgrade_restored_target_sentence`" in paper assert "RELEASE AUTHOR" not in paper + assert "restored outputs take" not in paper def _MOCK_fix_modules_target(entry: dict, engine: str = "policyengine-us 9.9.9"): - """MOCK: turn a restored output's target into the fix-module kind, with a + """MOCK: turn a fixed output's target into the fix-module kind, with a record value and a defective engine value both beyond the tolerance.""" value = entry["regenerated"] entry["record"]["alternative_value"] = value + 50.0 @@ -1627,7 +1631,7 @@ def _MOCK_fix_modules_target(entry: dict, engine: str = "policyengine-us 9.9.9") def _native_fix_modules() -> tuple[set[int], str]: - """The frozen sidecar's restored entries already held to fix modules, and + """The frozen sidecar's fixed-output entries already held to fix modules, and the engine a MOCK target must name to sit beside them (the sentence names one engine).""" entries = r.last_engine_upgrade.revision["regenerated_exclusions"] @@ -1642,7 +1646,7 @@ def _native_fix_modules() -> tuple[set[int], str]: def _with_restored(edit) -> PaperResults: - """MOCK edit of the frozen sidecar's restored entries.""" + """MOCK edit of the frozen sidecar's fixed-output entries.""" results = PaperResults() results.reference_revisions = deepcopy(r.reference_revisions) edit(results.reference_revisions[-1]["regenerated_exclusions"]) @@ -1671,10 +1675,24 @@ def test_the_fixed_outputs_split_into_returned_and_still_scored(): "values; the other four, ruled on 2026-10-06 while still scored, stay " "scored at those values." ) + # The upgrades table's clause gives the same split in the cell's numerals + # (round-2 final review nit 1). + assert r.engine_upgrade_restored_split_clause == ( + "fixes the engine defects behind 18 outputs (14 that earlier releases " + "excluded return to scoring at the new version's values; the 4 ruled on " + "2026-10-06 while still scored stay scored)" + ) paper = (ROOT / "paper/index.qmd").read_text() assert "`{python} r.engine_upgrade_restored_split_sentence`" in paper assert "excluded outputs to scoring" not in paper assert "PolicyBench excluded `{python} r.ruled_exclusion_count_word`" not in paper + # The table builds its split from the helper, not inline arithmetic. + assert ( + "{r.previous_policyengine_us_version} {r.engine_upgrade_restored_split_clause}" + " and changes" in paper + ) + assert "r.engine_upgrade_restored_count -" not in paper + assert "{r.engine_upgrade_restored_ruled_count}" not in paper @given(st.integers(0, 40), st.data()) @@ -1702,8 +1720,61 @@ def test_the_split_sentence_counts_both_kinds(restored, data): restored_split_sentence(restored, restored + 1, "2026-10-06") +@given(st.integers(0, 40), st.data()) +def test_the_split_clause_counts_both_kinds(restored, data): + """For any split, the upgrades table's clause names the total in numerals + with a matching noun, says "return" only when some outputs return and + "stay" only when some stay (as the split sentence does), agrees each verb + with its count, and never prints "the 0", "the 1" or a lone "1" in the + split (round-2 final review nit 1).""" + from policybench.paper_results import restored_split_clause, restored_split_sentence + + ruled = data.draw(st.integers(0, restored)) + clause = restored_split_clause(restored, ruled, "2026-10-06") + sentence = restored_split_sentence(restored, ruled, "2026-10-06") + assert clause.startswith("fixes ") + with pytest.raises(ValueError): + restored_split_clause(restored, restored + 1, "2026-10-06") + with pytest.raises(ValueError): + restored_split_clause(restored, -1, "2026-10-06") + if not restored: + assert clause == "fixes none of the engine defects behind excluded outputs" + return + returned = restored - ruled + noun = "output" if restored == 1 else "outputs" + defects = "defect" if restored == 1 else "defects" + assert clause.startswith(f"fixes the engine {defects} behind {restored} {noun} (") + assert clause.endswith(")") and clause.count("(") == clause.count(")") == 1 + split = clause[clause.index("(") + 1 : -1] + for kind in ("return", "stay"): + assert (kind in clause) == (kind in sentence) + assert ("return" in split) == (returned > 0) + assert ("stay" in split) == (ruled > 0) + assert ("2026-10-06" in split) == (ruled > 0) + if returned and ruled: + back = ( + "one that earlier releases excluded returns to scoring" + if returned == 1 + else f"{returned} that earlier releases excluded return to scoring" + ) + kept = ( + "the one ruled on 2026-10-06 while still scored stays scored" + if ruled == 1 + else f"the {ruled} ruled on 2026-10-06 while still scored stay scored" + ) + assert split.startswith(back) and split.endswith(kept) + elif returned: + verb = "it returns" if restored == 1 else "they return" + assert f"{verb} to scoring" in split + else: + verb = "it stays" if ruled == 1 else "they stay" + assert f"{verb} scored" in split + assert re.search(r"\b(the [01]|[01] that)\b", split) is None, split + assert re.search(r"(?