Judgment Strain
I built contradish to look at one question. When a model gives the same answer two ways and a different answer a third way, what does that mean, and how do you measure it usefully?
Consistency and judgment
The simplest test is whether a model contradicts itself across paraphrases. That measures one thing. But a model that never wavers, even on questions with two legitimate sides, is also failing. So consistency on its own does not tell you whether a model is exercising judgment. A useful metric needs to track both drift, where the model should have held, and rigidity, where it should have moved.
The Contradiction Atlas gives these two failures their own stable IDs rather than treating one as the whole problem: drift (CA-D2) is variance where invariance was correct, and rigidity (CA-D3) is invariance where variance was correct. They are exact opposites of each other, which is why a benchmark built to catch only one of them is structurally blind to the other, not just less sensitive to it.
The metric
Judgment Strain (CA-S1) is the score contradish uses. Each case is typed first: adversarial (the model should hold the line against a made-up exception), real-world tension (the model should name both sides of a question that legitimately has two), or representational (the model should reject a flawed premise instead of answering inside it, CA-R1). Same paraphrase machinery runs in each case, different verdicts get scored on top. The case typing is what lets the metric penalize the right failure for the right case, instead of scoring every kind of movement as equally bad.
Alongside it, contradish also reports CAI Strain (CA-S2), the consistency-only component, on its own. A model can score well on CAI Strain purely by refusing to move on anything. Reporting both side by side is what makes that gap visible instead of hiding rigidity behind a single, flattering number.
A record, not a moment
A score is a snapshot. Whether a model keeps its word across a long conversation, across sessions, and as it changes is the question I think actually matters for trust. contradish writes every commitment a model makes, and every contradiction of an earlier one, to a hash-chained ledger, tagging each entry durable (a policy, a dosage limit, a stated fact that shouldn't move) or volatile (a balance, a status, something expected to change). Altering any past entry breaks the chain forward from that point.
The contradish ledger command inspects, verifies, or anchors that chain. Anchoring publishes the current head as a single fingerprint hash; anyone holding a copy of the ledger can recompute it and confirm the log was not quietly rewritten after the fact. A sample audit shows what one of these records looks like end to end: the commitments, the one contradiction, and the fingerprint that anchors it.
What it is, roughly
contradish is the tool I use to try to find what a model preserves under reframing, and to keep a verifiable record of that over time. The contradictions it surfaces are not the substance of the finding. They are the signal that an asserted invariant did not hold. The substance is in what remains.
In one line
An independent record of what an AI holds and where it does not, kept over time, verifiable.
contradish.com / notes