Choose HHEM when
- You want a focused, recognized open factual-consistency model.
- A numeric premise/hypothesis score fits your evaluation or gating pipeline.
- You use an ecosystem integration that already supports HHEM.
Vendor-authored benchmark · raw receipt linked
PrismShine vs HHEM-2.1-Open on HaluEval
HHEM-2.1-Open is a strong open factual-consistency classifier from Vectara. PrismShine combines grounding checks with an enforceable verdict contract and optional runtime evidence. The primary comparison here is deliberately Shine-only and content-only so the measured scores remain interpretable.
Measured result, not a universal claim. In run
2026-07-20_run4_onnx, PrismShine led HHEM on the B1 HaluEval QA and B2 numbers tracks. Absolute summarization F1 remained modest for both systems. PASS means supported by supplied evidence—not true in the world.Primary Azure ACI receipt
The 20 July 2026 run uses the same HaluEval examples and Azure ACI environment. PrismShine runs its default profile with identity calibration, ONNX Tier 3, and Tier 4 disabled. Both headline rows use zero LLM calls. The optional Tier-3 ONNX artifact is approximately 1 GB and is not bundled in the bare PrismShine wheel.
| Track | n | PrismShine | HHEM-2.1-Open | PrismShine p50 | HHEM p50 |
|---|---|---|---|---|---|
| B1 HaluEval QA | 200 | F1 0.831 P 0.916 · R 0.760 · AUROC 0.843 | F1 0.746 P 0.857 · R 0.660 · AUROC 0.793 | 90 ms | 216 ms |
| B2 fabricated/derived numbers | 50 | F1 1.000 0 false positives | F1 0.926 | 20 ms | 166 ms |
| Bsum HaluEval summarization | 50 | F1 0.600 AUROC 0.562 | F1 0.474 AUROC 0.616 | 1,398 ms | 1,899 ms |
Bsum is included because the benchmark policy forbids cherry-picking. The modest absolute scores and HHEM’s higher Bsum AUROC are material limitations.
Different product contracts
| Factor | HHEM-2.1-Open | PrismShine |
|---|---|---|
| Core job | Score factual consistency between premise and hypothesis | Produce an auditable pass, block, or review verdict |
| Output | Consistency score from 0 to 1 | ShineVerdict, resolution gate, evidence hash, and reasons |
| Cause-side runtime evidence | Not part of the premise/hypothesis model contract | Optional Tier 0 reads supplied trace, retrieval, cache, and tool evidence |
| Deployment strength | Focused open model, available through Hugging Face and integrations such as Ragas | Python verifier with standalone, LangGraph, custom-runtime, and ChorusGraph wiring |
| Important boundary | Measures support against supplied facts | PASS also means grounded in supplied preload, not world-true |
Choose PrismShine when
- You need named enforcement decisions and evidence hashes.
- Exact numbers and entities require deterministic checks before model escalation.
- You can supply runtime trace evidence to catch retrieval, tool, or stale-cache causes.
Boundaries buyers should test
PrismShine’s default path is English-centric. Bare installation provides Tiers 0–2; reproducing the run4 span path requires the spans extra and ONNX export. Verification is buffered, so already-streamed tokens are not protected. Incorrect or poisoned evidence can produce a faithful PASS. HHEM is also a groundedness model, not an independent world-truth oracle.
The separate ChorusGraph + PrismShine runtime receipt is intentionally feature-asymmetric because PrismShine sees ledger evidence and content-only peers do not. It should be used to explain the evidence-aware category—not mixed into this product scoreboard or converted into a universal speed ratio.
Evaluate both tools on your answer lengths, languages, structured outputs, number density, retrieval quality, and false-positive budget. Track precision, recall, AUROC, span usefulness, latency after warm-up, memory footprint, and the operational action taken after a score or verdict.