Isomorphic Perturbation Testing (IPT)

Diagnose whether a model output is a genuine rule or a reward shortcut. A shortcut passes the standard verifier (extensional) but fails when object constants are renamed (isomorphic) — exposing that it memorised training instances rather than learning a generalizable rule.

Candidate Hypothesis (model output)

Validation Program

Positive predicate

Negative predicate

Result

Verdict

Isomorphic accuracy (genuine correctness)

Extensional accuracy (naive verifier)

Errors / warnings

This interface evaluates one hypothesis at a time. Use the Python API for batch processing.

Select example

Genuine rule Blatant shortcut Negation shortcut

A genuine relational rule — passes both verifications.

Rule

Validation program

eastbound / westbound