Reproduce

Setting up, checking, and verifying

Two kinds of command live in this repository. Fast checks take minutes and run on every commit. Scientific reproduction re-derives recorded results from the original artifacts or retrains models, and takes hours. Nothing below starts training unless it says so.

1. Environment

Python 3.13 with the exact versions used for the research (numpy, scipy, PyTorch CPU, scikit-learn, pytest).

minutes

git clone https://github.com/smarwat/persistent-physical-evidence.git
cd persistent-physical-evidence
uv venv .venv --python 3.13          # or: python3.13 -m venv .venv
uv pip install --python .venv/bin/python -r requirements-research.txt

2. Fast checks

What continuous integration runs. The test script deselects the tests that need the local research artifacts, which are not in Git; each verifier re-derives the Gate 0 results from scratch.

about 8 minutes in total

# the site's figure data equals a fresh export from the locked records (run first)
.venv/bin/python scripts/export_site_data.py --check

# unit and regression tests that need no artifacts (about 4 min)
ci/fast-tests.sh

# Gate 0: exact benchmark, ideal observers, structure and adjudication (about 1 min each)
.venv/bin/python scripts/verify_gate0_core.py
.venv/bin/python scripts/verify_gate0_structure.py
.venv/bin/python scripts/verify_gate0_final.py

# Gate 1 state-token serialisation audit (about 10 s)
.venv/bin/python scripts/verify_gate1_serialization.py

3. Verify the recorded Gate 1 results

Re-derives every Gate 1 number from the registered checkpoints and data, and checks every lock and pre-registration hash. It runs each earlier verifier live. It needs the local artifacts/ directory (datasets and checkpoints, about 2 GB), which is not distributed with the repository; without it the verifiers report the missing files.

about 10 minutes, needs artifacts

.venv/bin/python scripts/verify_gate1d.py   # Gate 1D, then 1C, 1B, 1A-v2 and Gate 0

4. Training

The training scripts are the ones that produced the registered models. Each refuses to run unless its family's pre-registration and lock are intact, and refuses to retrain after the family has been unlocked, so they cannot silently overwrite a recorded result. A fresh reproduction of a trained gate is a new, separately recorded run.

hours of CPU; do not run casually

# Gate 1A-v2: 5 registered runs
scripts/train_gate1a_v2.py
# Gate 1B-Expected: 220 registered runs, about 13 h on 8 cores
scripts/train_gate1b_expected.py
# Gate 1C: 20 registered runs, about 35 min on 8 cores
scripts/train_gate1c.py
# Gate 1D trains nothing

Details: docs/reproduce.md and the artifact policy.