Physical reasoning under uncertain object correspondence

Binding Physical Evidence to Persistent Object Identity

A controlled study of physical reasoning when useful evidence is separated in time from uncertain object correspondence.

Results GitHub Research status

Status: Active research
Completed: Gate 0, Gate 1A, Gate 1B, Gate 1C, Gate 1D
Next: Gate 1E: decision pending
Latest result: Gate 1D, evaluated 29 September 2026
Scope so far: State-token benchmark; no video yet

The core idea

Useful evidence arrives before the objects become ambiguous

A system sees an earlier interaction that reveals a hidden physical property. Later the objects look identical, and it must predict the effect of an intervention on one of them. A correct prediction combines two things: the historical evidence, and an uncertain belief about which current object that evidence belongs to.

Early interaction

A puck is struck. How far it slides depends on its hidden mass.
Physical evidence

The launch speed makes one puck more likely to be the heavy one.
Occlusion

A curtain hides both pucks. A carrier may swap them.
Uncertain correspondence

Two identical pucks reappear. A noisy cue says whether they were swapped.
Intervention query

Strike the puck now on the left. Will it slide past the marker?
Counterfactual prediction

The answer needs the old evidence bound to the right current object.

Remembering the evidence

The property is revealed once, before the ambiguity. A model has to keep it; the current frame no longer shows it.

Binding it to an object

Evidence about “the puck that was struck” is useless unless it is attached to the right puck after the curtain.

Keeping the uncertainty

When the correspondence is uncertain, the evidence must be weighted by it, not assigned to one object as if certain.

The benchmark

A minimal world with an exact answer

Two identical pucks slide on a one-dimensional track. The generator's ideal observer is known in closed form, so every learned model is scored against the exact Bayes prediction, not only against sampled outcomes.

Two pucks. Two identical pucks, A on the left and B on the right. Exactly one is heavy; each is heavy with probability ½.
Evidence. A launcher strikes puck A. A launch speed of 1.0 is more likely for the heavy puck: P(A heavy) 0.82.
Curtain and cue. A curtain hides both pucks and a carrier may swap them. A noisy cue reads −1: P(swap) 0.14.
Reappearance. Two identical pucks. The frame alone cannot say which is which: the left one is A with probability 0.86.
Query. Strike the left puck. It is heavy with probability 0.73, so it slides past the marker with probability 0.32.

Figure A. One episode of the state-level benchmark (setting C[d0=1,k=2], partial ambiguity). The drawing is schematic; the probabilities are the exact ideal observer’s for this history. No learned model is shown.

Observed
Hidden
Correspondence
Prediction

Six cells

Each cell switches one source of information on or off, so its correct behaviour is known in advance.

Cell	Condition	What it tests
A	Current observation suffices. Gauges show which puck is heavy	History must add nothing
B	Memory, identity kept. The carrier is parked, so no swap is possible	Evidence alone answers the query
C	Memory and ambiguity. A shuffle with a partly visible cue	Both sources are needed, and must be weighted
D	Full ambiguity. The cue carries no information	The evidence must be ignored
E	Property irrelevant. The query is too strong for mass to matter	Null: nothing should change the answer
F	No evidence. The launch response is screened	Null: there is nothing to bind

Three identity regimes

Within cell C, each episode falls into one regime according to how much the cue reveals about the swap.

Regime I, resolved. The cue settles the correspondence: |β_C| = 1. Hard assignment is exact here.
Regime II, partial. The cue is informative but noisy: 0 < |β_C| < 1. The evidence must be down-weighted.
Regime III, full. The cue is uninformative: β_C = 0. The ideal prediction ignores the evidence.

The learned-model experiments use a pooled environment of cells A, B, C, E and F: 460 legal histories, evaluated exactly. Full ambiguity appears there as C's regime III, the D-post stratum. Definitions: benchmark specification.

The central equation

Evidence and correspondence enter as a product

In the ambiguous cell, the ideal observer's prediction for the queried puck has a closed form. Neither source of information is useful alone: the evidence matters only to the extent that the correspondence is known.

p^{*} = \bar{θ} - \frac{Δ θ}{2} β_{E} β_{C}

$p^{*}$: Probability that the queried puck slides past the marker, given everything observed.
$\bar{θ}, Δ θ$: Mean of, and gap between, the outcome probabilities of a light and a heavy puck.
$β_{E}$: Confidence from physical evidence: 2 P(struck puck heavy | evidence) − 1.
$β_{C}$: Confidence in correspondence: 2 P(queried puck is the struck puck | cue) − 1.

In the episode above, the evidence gives β_E = 0.647 and the cue gives β_C = 0.714. With outcome probabilities θ_L = 0.9 for a light puck and θ_H = 0.1 for a heavy one, the ideal observer predicts p* = 0.5 − 0.4 × 0.647 × 0.714 = 0.315.

A system that commits to one identity treats β_C as ±1. Here it would predict 0.241: too confident, because it ignores the 0.14 chance that the pucks were swapped. Under full ambiguity (β_C = 0) the correct prediction ignores the evidence entirely.

Results

What the experiments show so far

All results are for state-token inputs: the models read an exact symbolic record of each episode, not pixels. Every experiment was pre-registered, and every number below is re-derived by a verifier from locked records.

A full-history transformer matches the ideal observer on the training distribution

Trained on 10⁶ episodes, a small transformer that reads the whole history reaches the exact Bayes prediction in every stratum, including partial ambiguity, where both sources of information have to be combined.

Gate 1B-Expected (report)

Regret, C partial: 0.00073 bits [0.00055, 0.00092]
Margin: 0.00295 bits, pre-registered
Decision: EQ in all 7 strata; all 6 null twin families pass and all 3 response ratios are consistent
Seeds: 20 independent training runs

Committing to one identity is overconfident exactly when identity is uncertain

A baseline with exact evidence and exact correspondence scoring, which then commits to the most probable identity, is exact when identity is resolved. Under partial ambiguity it loses 0.0295 bits per prediction; under full ambiguity, 0.3018. It keeps the confidence of the resolved case while its accuracy stays the same as the ideal observer's, so the failure is one of calibration.

The implicit transformer shows none of this: it keeps the uncertainty.

Gate 1D (report), outcome D1

(a) Expected-NLL regret over the ideal observer P3 · bits, linear scale

Resolved identity · B and C-res: correspondence is certain

Transformer

0.00019 [0.00016, 0.00023]

Hard assignment

0 exact

Partial ambiguity · C partial: the cue is informative but noisy

Transformer

0.00073 [0.00055, 0.00092]

Hard assignment

0.0295 exact

Full ambiguity · D-post: the cue carries no information

Transformer

0.00020 [0.000065, 0.00035]

Hard assignment

0.3018 exact

(b) Mean predictive entropy · bits, linear scale; higher means less certain

Resolved identity

P3 0.748 · T 0.750
hard 0.748

Partial ambiguity

P3 0.877 · T 0.873
hard 0.748

Full ambiguity

P3 1.000 · T 1.000
hard 0.748

Ideal observer P3
Transformer (T)
Hard assignment

Figure 1. Hard identity assignment against the full-history transformer, by identity regime (Gate 1D; exact population values). Hard assignment is Gate 0's P2, reproduced to 1.1e-16; the transformer is the locked Gate 1B model, 20 seeds, mean and two-sided 95% t-interval. Hard assignment is exact when identity is resolved, and its predictions stay as confident as in the resolved case when identity is uncertain. Top-1 accuracy is identical to P3's in every regime (0.78, 0.70, 0.50): the failure is one of calibration, not of the decision.

Data

Regime	Hard regret	Transformer regret	Entropy P3	Entropy hard	ECE hard
Resolved identity	0	0.00019 [0.00016, 0.00023]	0.7478	0.7478	0.000
Partial ambiguity	0.0295	0.00073 [0.00055, 0.00092]	0.8766	0.7478	0.080
Full ambiguity	0.3018	0.00020 [0.000065, 0.00035]	1.0000	0.7478	0.280

The model uses both evidence and correspondence, and loses exactly the information removed

Masking the physical evidence, the correspondence cue, or the history removes a precisely known amount of information. Each restricted model reaches the Bayes limit of its own input, so its loss relative to the full model is the theoretical one.

Gate 1C (report), outcome C1

Partial ambiguity (C partial) · both sources are needed; excess expected NLL over P3, bits, linear scale

C0 Full history

loss 0
learning 0.00073

C1 Current observation only

loss 0.1234
learning 0.000069

C2 No correspondence cue

loss 0.1234
learning 0.000094

C3 No physical evidence

loss 0.1234
learning 0.000072

C4 Fixed window

loss 0.1234
learning 0.000070

Identity kept (B) · the evidence alone suffices; excess expected NLL over P3, bits, linear scale

C0 Full history

loss 0
learning 0.00012

C1 Current observation only

loss 0.2522
learning 0.000069

C2 No correspondence cue

loss 0
learning 0.00018

C3 No physical evidence

loss 0.2522
learning 0.000071

C4 Fixed window

loss 0.2522
learning 0.000070

Information loss of the restricted input (exact)
Trained transformer's excess over P3 (95% interval)

Figure 2. Removing a source of information costs what theory says it should (Gate 1C, C1). Each restricted transformer is trained exactly like the full-history model, with one input field masked. The bar is the exact information loss of the restricted input, L(restricted Bayes predictor) − L(P3); the diamond is the trained model's measured excess over P3 (5 seeds; C0: the 20 Gate 1B seeds). Their difference, the learning regret, is printed on the right and never exceeds 0.00073 bits. Where identity is kept (B), the correspondence cue carries no information and removing it (C2) costs nothing.

Data

Input	Removes	Loss, C partial	Learning regret, C partial	Loss, B	Learning regret, B
C0 Full history	nothing	0	0.00073 [0.00055, 0.00092]	0	0.00012 [0.000094, 0.00015]
C1 Current observation only	the strike, the launch speed and the cue	0.1234	0.000069 [0.000057, 0.000081]	0.2522	0.000069 [0.000057, 0.000081]
C2 No correspondence cue	the cue	0.1234	0.000094 [0.000019, 0.00017]	0	0.00018 [7.1 × 10⁻⁶, 0.00035]
C3 No physical evidence	the launch speed	0.1234	0.000072 [0.000067, 0.000078]	0.2522	0.000071 [0.000066, 0.000076]
C4 Fixed window	the strike and the launch	0.1234	0.000070 [0.000060, 0.000080]	0.2522	0.000070 [0.000059, 0.000080]

Held-out combinations of evidence and correspondence remain difficult

When one combination of evidence strength and cue reliability never appears in training, the same transformer composes only partially. The result is recorded as a failure against a pre-registered margin. No structured model has been evaluated on this condition yet.

Gate 1B-Expected, compositional test (report, §N)

Regret over P3 · bits, linear scale; 20 seeds each

IID: partial ambiguity
combinations seen in training

0.00073 [0.00055, 0.00092]
EQ against margin 0.00295

Held out: certain evidence × partial correspondence
never seen together in training

0.0626 [0.0387, 0.0864]
FAIL against margin 0.00817

Transformer, mean and 95% interval
One seed
Pre-registered margin
Count table (cannot compose)

Figure 3. Held-out composition remains difficult (Gate 1B-Expected, outcome B: IID success, compositional failure). The held-out histories combine certain evidence (launch speed ½ or 4) with partial correspondence (cue ±1). Both factors appear in training, never together. On them the transformer's regret is 75% lower than that of a count table, which cannot compose at all, but it stays above the pre-registered margin: the frozen decision is FAIL. On the IID stratum the same model is within its margin (EQ). Intervals are two-sided 95% t-intervals over seeds; the frozen decisions use one-sided bounds with Holm correction.

Method

Validate the benchmark first, then test learned models against it

The work proceeds in gates. Each gate is pre-registered before any model is trained, adjudicated by frozen rules, and re-verified from locked records. A failed gate is kept on record, not rerun until it passes.

Gate 0

An exact benchmark and its ideal observer

The generator's Bayes-optimal predictor is derived in closed form and checked against exhaustive enumeration and a Monte Carlo cross-check. Reference predictors bracket it: current observation only (P0), evidence without the cue (P1), hard identity assignment (P2), and the ideal observer (P3). Twin tests and a leakage audit check that nothing but the intended information predicts the outcome.

Status: state level validated (GO). Report

Gate 1

Learned models on state tokens

Transformers read an exact symbolic record of each episode. A positive control trained directly on the ideal observer's answers establishes that the stack can learn the Bayes mapping (largest excess bound 2.4 × 10⁻⁷ bits against 0.00147). The implicit model, the ablations and the hard-assignment baseline are then scored exactly on all 460 legal histories.

Status: Gates 1A–1D complete. Records

Pixels, then real video

The same questions will be asked of models that see rendered video, where perception and correspondence must be learned from pixels, and later of real physical video. Neither has been run: every result on this page is state-level.

Status: pixel-level Gate 0 pending; real-world video planned.

Status

Active research

Completed gates are adjudicated and verified. Everything after Gate 1D is open, and nothing here is a claim about video or real-world data.

Gate 0, complete Benchmark validation Exact benchmark and ideal observer, state level
Gate 1A, complete Learnability Positive control: the Bayes mapping is learnable
Gate 1B, complete Implicit model Full-history transformer: IID success, compositional failure
Gate 1C, complete Information ablations Clean dependence on evidence and correspondence
Gate 1D, complete Hard association Hard identity assignment fails under ambiguity
Gate 1E, not complete Structured model Decision pending
Video, not complete Pixel benchmark Not started
Real world, not complete Real-world video Planned

Every gate, its question and its frozen outcome, with the final report.
Experiment	Question	Result	Record
Gate 0	Is the benchmark correct, and free of leaks?	State level validated (GO); pixel level pending	GATE0-FINAL-REPORT
Gate 1A-v1	Can the stack learn the Bayes mapping under direct supervision?	FAIL: early stopping ended runs on a plateau (kept on record)	GATE1A-FINAL-REPORT
Gate 1A-v2	The same, with the full training budget	PASS: 26/26 criteria; largest excess bound 2.4 × 10⁻⁷ bits	GATE1A-V2-FINAL-REPORT
Gate 1B-Expected	Does a full-history transformer reach the ideal observer?	Outcome B: IID success, compositional failure	GATE1B-EXPECTED-V1-FINAL-REPORT
Gate 1B-Realized	The same, trained on sampled outcomes	Deferred	SPEC-RESOLUTIONS
Gate 1C	Does it use both evidence and correspondence?	PASS, C1: clean information dependence	GATE1C-V1-FINAL-REPORT
Gate 1D	What does committing to one identity cost?	D1: hard assignment fails under ambiguity	GATE1D-V1-FINAL-REPORT
Gate 1E	Does an explicit probabilistic model help?	Decision pending; a minimal structured baseline is recommended	GATE1E-DECISION

Binding Physical Evidence to Persistent Object Identity

Useful evidence arrives before the objects become ambiguous

Early interaction

Physical evidence

Occlusion

Uncertain correspondence

Intervention query

Counterfactual prediction

Remembering the evidence

Binding it to an object

Keeping the uncertainty

A minimal world with an exact answer

Six cells

Three identity regimes

Evidence and correspondence enter as a product

What the experiments show so far

A full-history transformer matches the ideal observer on the training distribution

Committing to one identity is overconfident exactly when identity is uncertain

The model uses both evidence and correspondence, and loses exactly the information removed

Held-out combinations of evidence and correspondence remain difficult

Validate the benchmark first, then test learned models against it

An exact benchmark and its ideal observer

Learned models on state tokens

Pixels, then real video

Active research