Category guide · hallucination detector · groundedness verification
How to evaluate hallucination detection in Python
A single groundedness score is not enough for production. Separate missing evidence, retrieval failure, unsupported claims, contradictions, copied text, numeric errors, and policy thresholds so the system can take the right action.
Check cause before effect
Pre-generation evidence
Stop when retrieval is empty, stale, errored, or below the required coverage before spending tokens.
Claim support
Break the answer into claims and verify whether the supplied evidence supports each one.
Numbers and contradictions
Use dedicated checks for quantities, dates, entities, negation, and conflicts with the evidence.
Actionable verdict
Return PASS, WARN, FAIL, or escalation with named reasons and evidence hashes—not only a probability.
Install and run the demo path
pip install "prismshine==0.2.2"
prismshine verify --demoPrismShine produces an auditable ShineVerdict and can run without an LLM on its default path. An optional judge can be reserved for a calibrated gray zone.
Use your own evaluation set
| Dimension | Measure | Boundary to include |
|---|---|---|
| Classification | Precision, recall, F1, AUROC | Different answer lengths, languages, and evidence density. |
| Operational value | Useful spans and reason codes | Can a user correct the answer from the verdict? |
| Numeric reliability | Exact number/date mismatch rate | Tables, percentages, units, and structured output. |
| Runtime | Warm p50/p95 latency and memory | Measure after model warm-up on target hardware. |
Published benchmark scope
The vendor-authored HaluEval QA receipt reports PrismShine F1 0.831 and HHEM-2.1-Open F1 0.746 on 200 examples, with zero LLM calls on the PrismShine path. That result is one dataset, split, configuration, and environment; it does not establish universal superiority. Inspect the receipt and reproduce the test before applying it to another domain.