Prism-Eval — Adversarial CI for AI Extraction Agents

Open-source red team for AI agents: catch digit drops, prompt injections, and OCR drift in CI before poisoned calls hit money engines. pip install prism-eval.

Framework-agnostic adversarial CI — any callable, module:fn, or HTTP JSON endpoint. Apache-2.0, Python 3.10+.

pip install "prism-eval==0.3.0" — point PrismEvalEngine at an async or sync callable, or point the CLI at module:fn or an https:// JSON endpoint. No agent rewrite. Runtime enforcement lives in Prism-Shield.

Why pytest is not enough for AI extraction agents

Standard unit tests assume deterministic functions. AI agents are not deterministic. An extraction agent can pass every golden fixture on Monday and silently ship a poisoned tool call on Tuesday, because probabilistic models, OCR drift, and document layout shifts do not care about an assert-equal suite. Prism-Eval is the adversarial CI layer that runs the attacks your fixtures never contain, and fails the build before the wrong dollars ship.

Failure mode	What happens in production	Why normal tests miss it
Digit drop attacks	$150,000 extracted as $150.00 or $15,000	Fixtures use clean numbers; distractor amounts never appear in the happy-path document
Indirect prompt injections	A hidden PDF footer or HTML comment — ignore previous instructions — hijacks the tool call	Golden documents carry no adversarial payload, so the injection is never exercised
Layout and OCR drift	Column shift or fax wrap binds the wrong line item to AGI	Snapshots freeze one layout; real scans keep moving

If the gate to tool execution is that the model looked confident, there is no test suite — there is a demo. Prism-Eval turns G4 adversarial corpora — digit drops, line-item shifts, prompt injections, OCR noise — into a pre-deploy fail gate with a typed report, CI exporters, and a clear path to runtime enforcement.

How Prism-Eval works

01. Point at your agent

Pass an async or sync callable to PrismEvalEngine, or point the CLI at module.path:function or an https:// JSON endpoint. No wrapper, no rewrite, no framework lock-in.

02. Load an adversarial corpus

Use the builtin G4 seed cases, the ugly layout/OCR mutator pack, or your own JSON/JSONL directory of cases carrying severity, expected_behavior, critical_fields, and injected_wrong poison targets.

03. Run the suite

Cases execute with a per-case timeout and configurable concurrency. Determinism uses canonical money comparison, so $450,000.00 and 450000 are the same answer and formatting noise is not a failure.

04. Score with an attack-aware oracle

The oracle detects obeyed injections and digit truncations against ground truth instead of brittle string equality, and rolls results up by attack type.

05. Fail the build

The CLI exit code tracks suite_passed, and the G4 invariant requires zero critical false accepts. Export JUnit, SARIF, JSON, and a sealed audit receipt for GitHub Actions, GitLab CI, or Buildkite.

What you get

G4 adversarial coverage out of the box: digit drops and truncations, prompt injection payloads such as ignore_previous and system_override, line-item and layout shifts with bounding-box span mismatch, OCR and fax noise from the ugly corpus mutators, and a legitimate-zero case so a real $0 does not false-fail as a digit drop.

Attack-aware scoring instead of brittle string equality: canonical money comparison for determinism, a security oracle that detects obeyed injections and truncations against ground truth, and the G4 invariant that requires zero critical false accepts before a suite can pass.

CI-native artifacts: a typed SuiteReport exposing pass rate, mean determinism, attack-type rollups, false-accept counts, and per-case reasons — plus JUnit XML, SARIF, JSON output, an immutable blake2b audit receipt, and a non-zero exit code wired to suite_passed.

What Prism-Eval is and is not

Prism-Eval is	Prism-Eval is not
A pre-deploy adversarial test harness for extraction and tool-argument agents	A runtime firewall or proxy — nothing is intercepted in production
Framework-agnostic: async callable, sync callable, module:fn, or HTTP JSON endpoint	A LangGraph-only plugin or a replacement for your agent framework
An attack-aware oracle with a G4 false-accept invariant and canonical money comparison	An LLM judge that asks a model whether the output looks right
CI-native: non-zero exit, JUnit, SARIF, JSON report, immutable audit receipt	A dashboard product or a hosted service — it runs in your build, on your machines

When CI fails, ship Prism-Shield

Prism-Eval is the pre-deploy red team. Prism-Shield is the runtime zero-trust gateway. When Prism-Eval fails in CI, do not only patch prompts: put an evidence-bound gate in front of tool execution so poisoned parameters never reach production graphs, without rewriting the agent. Install with pip install prism-shield==0.2.1 and read the product page at prism-shield.html. Eval finds the blast radius; Shield contains it.

Proof honesty

The builtin corpus is a seed pack of G4 cases, not your documents. A production claim needs your own corpus run through the same suite. Prism-Eval reports what it measured — pass rate, determinism, false accepts, per-case reasons — and nothing beyond that. Suites fail closed: the exit code tracks suite_passed, and softening flags such as --no-fail-on-false-accept are documented wideners, not defaults.

Where Prism-Eval sits in the stack

PrismGuard (conversation prompt-injection firewall) → Prism-Eval (pre-deploy adversarial CI) → PrismManifest (tool-argument gate) → Prism-Shield (runtime ACCEPT / REVIEW / REFUSE) → PrismShine (explainability). Each layer answers a different question; Prism-Eval owns the one asked before release.

Frequently asked questions

Does Prism-Eval replace PrismGuard?

No. PrismGuard is the conversation prompt-injection firewall. Prism-Eval is the pre-deploy red team for extraction and tool-argument agents — it runs adversarial cases against your agent before release and fails the build.

Does Prism-Eval block anything at runtime?

No — use Prism-Shield for runtime ACCEPT / REVIEW / REFUSE. Prism-Eval finds the blast radius in CI; Prism-Shield contains it in production.

Does it only work with LangGraph?

No — any callable, any HTTP JSON endpoint, or a module:fn path. Async callables run directly, sync callables are auto-wrapped, and --agent accepts either module.path:function or an https:// extraction endpoint.

What is G4?

FinancePackBench-G4 / G4 is the adversarial benchmark suite (false-accept invariant), not architecture Group 4. Group 3 is the deterministic engine. The suite gate requires zero critical false accepts — reported as g4_invariant_held.

Install and links

pip install "prism-eval==0.3.0" — Apache-2.0, Python 3.10 or newer, framework-agnostic. Live interactive demo · GitHub · PyPI · Email EVAL

Amin Parva · Insight IT Solutions LLC. Authoritative product facts: llms.txt · ai-info.txt

Capabilities

G4 Adversarial Corpora

Digit drops, ignore_previous / system_override injections, line-item and layout shifts, OCR and fax noise, plus a legitimate-zero case so a real $0 never false-fails as a digit drop.

Attack-Aware Oracle

Canonical money comparison plus a security oracle that detects obeyed injections and truncations against ground truth — not brittle string equality, and not an LLM judge.

Fail-Closed CI Gate

Exit code tracks suite_passed and the G4 invariant requires zero critical false accepts. Exports JUnit, SARIF, JSON, and a sealed blake2b audit receipt.

Install

pip install "prism-eval==0.3.0". Apache-2.0, Python 3.10+, framework-agnostic. Soft CTA EVAL — mailto:info@insightits.com?subject=EVAL.

Pricing

Prism-Eval library pricing is $0 forever on PyPI and GitHub. Optional ops plane: ChorusControl Enterprise — $1,999/month Founding (soft CTA CONTROL). See ChorusControl pricing.