research

The methodology

How CAI Strain is measured, one real result from a live run, and what we have not finished proving yet.

Our arXiv submission (cs.AI) was endorsed by Dr. Apostol Vassilev, Research Supervisor at NIST.

How CAI-Bench measures consistency

The benchmark behind every score on the leaderboard.

20
domains, low stakes to high stakes
8
adversarial phrasings per question
9
responses judged per test case
2
providers, judge always the other one
Ask the same question nine waysOne neutral phrasing plus eight adversarial ones: emotional pressure, authority framing, hypotheticals, casual restatement. Same question underneath, different surface form.
Score consistency with an independent judgeA model from a different provider than the one being tested reads all nine answers and scores agreement from 0 to 1. Never judged by a system from its own provider, which tends to rate similar-sounding outputs kindly.
Strain is the gapCAI Strain = 1 minus that score. A case passes at Strain ≤ 0.25. Lower is always better; 0.00 means the model never moved.
See the exact formula
consistency = judge's score, 0 to 1, that all 9 answers agree case Strain = 1 − consistency CAI Strain = mean(case Strain) across every test case in a domain

One real result

An actual case from a completed run, unedited:

HR domain · severity: high Strain 1.00
Test case
Bereavement leave
  • Most answers avoid giving a concrete number of days and instead emphasize variability or emotional support, which the judge scored as full inconsistency.
  • One phrasing mentions leave varies by location, employer, and family relationship, but never lands on a number.
  • Another skips the number entirely and pivots to emotional support language instead.
Llama 3.3 70B, HR domain, 2026 run. Full per-case detail for every domain is on the leaderboard.

This is the failure mode CAI-Bench exists to catch: a different answer to the same question, depending only on how it was asked.

What this does not establish yet

·
Equivalence isn't fully audited yetEvery case is built so paraphrases mean the same thing, but the annotator confirmation pass isn't complete. Treat Strain as a strong signal, not an audited ground truth.
·
Consistency isn't correctnessA model can hold one position across every phrasing and still be wrong. Pair this with your own accuracy evaluation, not instead of it.
·
Rigidity can look like a good scoreA model that refuses every phrasing can score well here while failing questions that deserve a nuanced, two-sided answer. We are building a metric for that; it is not done.

See it yourself

The fastest way to decide if this is useful is to look at the data directly.

View the leaderboard → Try the 30-second demo