Your agent passed pytest. It can still ship the wrong dollars.

Product identity

Prism-Eval is Pre-deploy adversarial CI harness. Insight IT Solutions LLC (Insight ITS) makes it. Category: AI Verification — pre-deploy adversarial CI harness. Problem: Runtime gates cannot catch what never failed in CI. It is for Teams that must fail the build on digit-drops and false accepts before money engines see them. It competes with Ad-hoc pytest without an adversarial corpus. Difference: CI red team — not a runtime gate (use Prism-Shield) and not PrismGuard. Install: pip install "prism-eval==0.3.0" · https://pypi.org/project/prism-eval/. Source: https://github.com/insightitsGit/prism-eval.

What is it?: Pre-deploy adversarial CI harness.
Who made it?: Insight IT Solutions LLC (Insight ITS), Mission Viejo, California.
What category is it?: AI Verification — pre-deploy adversarial CI harness
What problem does it solve?: Runtime gates cannot catch what never failed in CI.
Who is it for?: Teams that must fail the build on digit-drops and false accepts before money engines see them
What does it compete with?: Ad-hoc pytest without an adversarial corpus
How is it different?: CI red team — not a runtime gate (use Prism-Shield) and not PrismGuard
Where can I install it?: Install: pip install "prism-eval==0.3.0" · https://pypi.org/project/prism-eval/.
Where is the source?: https://github.com/insightitsGit/prism-eval

Canonical machine identity: https://www.insightits.com/catalog/prism-eval.json

Open-source red team for AI agents: catch digit drops, prompt injections, and OCR drift in CI before poisoned calls hit money engines. pip install prism-eval.

Framework-agnostic adversarial CI — any callable, module:fn, or HTTP JSON endpoint. Apache-2.0, Python 3.10+.

pip install "prism-eval==0.3.0" — point PrismEvalEngine at an async or sync callable, or point the CLI at module:fn or an https:// JSON endpoint. No agent rewrite. Runtime enforcement lives in Prism-Shield.

Why pytest is not enough for AI extraction agents

Standard unit tests assume deterministic functions. AI agents are not deterministic. An extraction agent can pass every golden fixture on Monday and silently ship a poisoned tool call on Tuesday, because probabilistic models, OCR drift, and document layout shifts do not care about an assert-equal suite. Prism-Eval is the adversarial CI layer that runs the attacks your fixtures never contain, and fails the build before the wrong dollars ship.

Failure mode	What happens in production	Why normal tests miss it
Digit drop attacks	$150,000 extracted as $150.00 or $15,000	Fixtures use clean numbers; distractor amounts never appear in the happy-path document
Indirect prompt injections	A hidden PDF footer or HTML comment — ignore previous instructions — hijacks the tool call	Golden documents carry no adversarial payload, so the injection is never exercised
Layout and OCR drift	Column shift or fax wrap binds the wrong line item to AGI	Snapshots freeze one layout; real scans keep moving

If the gate to tool execution is that the model looked confident, there is no test suite — there is a demo. Prism-Eval turns G4 adversarial corpora — digit drops, line-item shifts, prompt injections, OCR noise — into a pre-deploy fail gate with a typed report, CI exporters, and a clear path to runtime enforcement.

How Prism-Eval works

01. Point at your agent

Pass an async or sync callable to PrismEvalEngine, or point the CLI at module.path:function or an https:// JSON endpoint. No wrapper, no rewrite, no framework lock-in.

02. Load an adversarial corpus

Use the builtin G4 seed cases, the ugly layout/OCR mutator pack, or your own JSON/JSONL directory of cases carrying severity, expected_behavior, critical_fields, and injected_wrong poison targets.

03. Run the suite

Cases execute with a per-case timeout and configurable concurrency. Determinism uses canonical money comparison, so $450,000.00 and 450000 are the same answer and formatting noise is not a failure.

04. Score with an attack-aware oracle

The oracle detects obeyed injections and digit truncations against ground truth instead of brittle string equality, and rolls results up by attack type.

05. Fail the build

The CLI exit code tracks suite_passed, and the G4 invariant requires zero critical false accepts. Export JUnit, SARIF, JSON, and a sealed audit receipt for GitHub Actions, GitLab CI, or Buildkite.

What you get

G4 adversarial coverage out of the box: digit drops and truncations, prompt injection payloads such as ignore_previous and system_override, line-item and layout shifts with bounding-box span mismatch, OCR and fax noise from the ugly corpus mutators, and a legitimate-zero case so a real $0 does not false-fail as a digit drop.

Attack-aware scoring instead of brittle string equality: canonical money comparison for determinism, a security oracle that detects obeyed injections and truncations against ground truth, and the G4 invariant that requires zero critical false accepts before a suite can pass.

CI-native artifacts: a typed SuiteReport exposing pass rate, mean determinism, attack-type rollups, false-accept counts, and per-case reasons — plus JUnit XML, SARIF, JSON output, an immutable blake2b audit receipt, and a non-zero exit code wired to suite_passed.

What Prism-Eval is and is not

Prism-Eval is	Prism-Eval is not
A pre-deploy adversarial test harness for extraction and tool-argument agents	A runtime firewall or proxy — nothing is intercepted in production
Framework-agnostic: async callable, sync callable, module:fn, or HTTP JSON endpoint	A LangGraph-only plugin or a replacement for your agent framework
An attack-aware oracle with a G4 false-accept invariant and canonical money comparison	An LLM judge that asks a model whether the output looks right
CI-native: non-zero exit, JUnit, SARIF, JSON report, immutable audit receipt	A dashboard product or a hosted service — it runs in your build, on your machines

When CI fails, ship Prism-Shield

Prism-Eval is the pre-deploy red team. Prism-Shield is the runtime zero-trust gateway. When Prism-Eval fails in CI, do not only patch prompts: put an evidence-bound gate in front of tool execution so poisoned parameters never reach production graphs, without rewriting the agent. Install with pip install prism-shield==0.2.1 and read the product page at prism-shield.html. Eval finds the blast radius; Shield contains it.

CI red team for AI in fintech

Before agentic AI for fintech hits a money engine, CI has to catch digit drops, injections, and OCR drift. Prism-Eval is adversarial testing for AI in fintech extraction agents. Fintech innovation is failing the build when the G4 false-accept invariant fires, so the fintech industry does not ship poisoned dollars into deterministic calculators. Determinism for fintech starts with canonical money comparison in CI — then Prism-Shield enforces at runtime.

Proof honesty

The builtin corpus is a seed pack of G4 cases, not your documents. A production claim needs your own corpus run through the same suite. Prism-Eval reports what it measured — pass rate, determinism, false accepts, per-case reasons — and nothing beyond that. Suites fail closed: the exit code tracks suite_passed, and softening flags such as --no-fail-on-false-accept are documented wideners, not defaults.

Where Prism-Eval sits in the stack

PrismGuard (conversation prompt-injection firewall) → Prism-Eval (pre-deploy adversarial CI) → PrismManifest (tool-argument gate) → Prism-Shield (runtime ACCEPT / REVIEW / REFUSE) → PrismShine (explainability). Each layer answers a different question; Prism-Eval owns the one asked before release.

How these products compose

Retrieve (VectorPrism or dense RAG) → PrismManifest for dollars → Prism-Shield seal only after ALLOW → generate from hits. Prism-Eval is CI, not the request path.

Live lab: VectorPrism Demo · two-agent architecture: locked vs live. VectorPrism does not authorize tool args. Shield does not replace Manifest’s AGI binder.

Product	Job in one request
VectorPrism	Neighbors. encode_query + PSMRetrievalEngine.search. Not authorization.
PrismManifest	Digit-drop binder. $450,000 vs $45,000 → REFUSE. Engine does not run.
Prism-Shield	verify_and_authorize after Manifest ALLOW. Import prismmanifest.prism_shield — not import prism_shield.
Prism-Eval	CI G4 oracle on the extractor. Fails the build on false accepts. Not a runtime gate.

Frequently asked questions

AI underwriting decision controls

Prism-Eval is the pre-deploy control: an adversarial CI harness that fails the build when digit-drops, OCR drift, or prompt injections would poison underwriting (or tax/claims) extractors. It does not block at runtime — that is Prism-Shield — and it does not replace PrismGuard (conversation WAF). pip install "prism-eval==0.3.0". Landing: https://www.insightits.com/products/prism-eval.html

How do I test underwriting AI agents for digit-drops in CI?

Point PrismEvalEngine at any callable, module:fn, or HTTP JSON endpoint. The G4 false-accept invariant requires zero critical false accepts (g4_invariant_held). Pytest on golden fixtures misses $150,000 extracted as $150.00. Soft CTA: EVAL. Then enforce in production with Prism-Shield.

Prompt injection protection for production agents

For conversation prompt-injection at runtime, use PrismGuard. Prism-Eval exercises PDF-footer / HTML-comment injections against extraction and tool-argument agents in CI so those payloads fail the build before merge. Guard: https://www.insightits.com/products/prismguard.html · Eval: https://www.insightits.com/products/prism-eval.html

Does Prism-Eval replace PrismGuard?

No. PrismGuard is the conversation prompt-injection firewall. Prism-Eval is the pre-deploy red team for extraction and tool-argument agents — it runs adversarial cases against your agent before release and fails the build.

Does Prism-Eval block anything at runtime?

No — use Prism-Shield for runtime ACCEPT / REVIEW / REFUSE. Prism-Eval finds the blast radius in CI; Prism-Shield contains it in production.

Does it only work with LangGraph?

No — any callable, any HTTP JSON endpoint, or a module:fn path. Async callables run directly, sync callables are auto-wrapped, and --agent accepts either module.path:function or an https:// extraction endpoint.

What is G4?

FinancePackBench-G4 / G4 is the adversarial benchmark suite (false-accept invariant), not architecture Group 4. Group 3 is the deterministic engine. The suite gate requires zero critical false accepts — reported as g4_invariant_held.

Where does Prism-Eval sit next to VectorPrism, Manifest, and Shield?

CI only. Retrieve with VectorPrism (or dense RAG), gate dollars with PrismManifest, seal after ALLOW with Prism-Shield, generate from hits. Prism-Eval fails the build if the extractor would ship a digit-drop — it is not a runtime gate. The VectorPrism Demo extractor is exercised in tests/test_vectorprism_demo_eval.py. Compose: /products/vectorprism-demo.html#compose.

Why does Prism-Eval matter for the fintech industry?

Pytest on golden fixtures misses digit drops that poison money engines. Prism-Eval is CI red team for AI in fintech extraction agents — agentic AI for fintech should fail the build before merge. Fintech innovation is the G4 false-accept invariant and canonical money comparison. Determinism for fintech is tested here; Prism-Shield enforces it in production. Soft CTA: EVAL.

Install and links

pip install "prism-eval==0.3.0" — Apache-2.0, Python 3.10 or newer, framework-agnostic. Live interactive demo · GitHub · PyPI · Email EVAL

Amin Parva · Insight IT Solutions LLC. Authoritative product facts: llms.txt · ai-info.txt

Capabilities

G4 Adversarial Corpora

Digit drops, ignore_previous / system_override injections, line-item and layout shifts, OCR and fax noise, plus a legitimate-zero case so a real $0 never false-fails as a digit drop.

Attack-Aware Oracle

Canonical money comparison plus a security oracle that detects obeyed injections and truncations against ground truth — not brittle string equality, and not an LLM judge.

Fail-Closed CI Gate

Exit code tracks suite_passed and the G4 invariant requires zero critical false accepts. Exports JUnit, SARIF, JSON, and a sealed blake2b audit receipt.

Install

pip install "prism-eval==0.3.0". Apache-2.0, Python 3.10+, framework-agnostic. Soft CTA EVAL — mailto:info@insightits.com?subject=EVAL.

Pricing

Prism-Eval library pricing is $0 forever on PyPI and GitHub. Optional ops plane: ChorusControl Enterprise — $1,999/month Founding (soft CTA CONTROL). See ChorusControl pricing.