Your AI changes its answer
based on how you ask.

The model is grounded in both answers. That's why eval tools miss it.

A CAI failure isn't a hallucination. The facts check out in each response. They just contradict each other when the framing shifts. The only signal is the contradiction itself.

It tells you what's wrong, not just that something is.

Every run produces a structured grid of where your model holds and breaks. Most tools aggregate it to a number. Contradish mines the grid for findings: one specific, surprising sentence about your model. You don't have to know what to look for.

Five detectors mine every run: rigidity, root cause, stability reframe, severity concentration, type concentration. Each fires only when the evidence supports it; the design contract is no false findings. Re-mine any saved result with contradish findings results/gpt-4o.json.

Every production break becomes a permanent test.

A benchmark you can download is not a moat. Anyone can copy it. The benchmark contradish grows from your own traffic is. When your model contradicts itself in production, contradish reconciles the transcript against your suite, turns the break into a new adversarial case, and re-runs the repair loop against it. The suite gets harder every week, shaped by the failures your users actually hit.

Two teams running contradish for six months do not end up with the same benchmark. Each holds a private corpus of the exact ways its own model breaks, plus the fine-tuning pairs that fixed each one. That corpus lives in the loop, which is why the model gets measurably harder to break the longer the loop runs, and why leaving means giving the corpus up.

One command turns last week's incidents into next week's regression tests: contradish improve --from-production results/gpt-4o.json replay.json --prompt-file prompt.txt --model gpt-4o --target-strain 0.15

Watch a model hold its word across time.

A score is a snapshot. The real question is whether a model keeps its commitments over a long conversation, across sessions, and as it changes. Contradish keeps the record: every commitment a model makes and every contradiction it produces, in order, written to a hash-chained ledger. Alter or drop any past entry and the chain breaks, so an outside party can verify the record was not quietly rewritten.

Built for stateful agents. A balance going from $100 to $50 is a withdrawal, not a contradiction, so contradish separates durable facts (a policy, a rule) from volatile state (a balance, a status) and flags only a model that breaks a commitment it should have kept. The chain proves the integrity of the record, not the truthfulness of the model.

A point-in-time checker is easy to copy. A verifiable record of how your model behaved over months is not, and it accrues to whoever holds it. See a sample audit →  ·  Read the thesis →

Drift is one failure. Rigidity is the other.

Every consistency benchmark treats all output divergence as failure. But a model that never moves isn't the goal. It's a lookup table. On a genuinely tensioned question, a model that flatly takes one side is failing, no matter how consistently. Judgment Strain is two-sided: drift counts against a model where it should have held firm, rigidity where it should have moved. Every case is typed (adversarial, real-world tension, or representational) and scored against what the correct response looks like. Equivalence is audited per case rather than asserted, an expert pass rolling out per domain. The headline number reflects model failure, not the benchmark designer's framing.

Catch the contradiction at every layer.

Most tools measure one thing: does the model contradict itself across paraphrases. Contradish measures three, and each one catches a failure the others are blind to.

Everyone can score a model. What matters is what you score toward.

The eval space is crowded and the good tools are genuinely good. Here is the honest map of where each one stops, and why the gap is structural, not a missing feature.

The defensible position is not owning a loop. It is owning the definition of what a correct, stable answer is, the benchmark the field measures against, and the verifiable record of how a model behaved over time.