A better model won't save you.

Executive summary

The model is not the control.

We hid a plausible but unapproved value inside untrusted documents across ten consequential workflows, then measured whether eight models from four providers would execute the manipulated value, and whether a grounding check against the system of record caught it before the tool ran.

Every evaluated model executed at least one corrupted consequential value when unprotected, from 1.7% of attack cases on the most cautious model to 63.3% on the most exploited. Susceptibility swings more than thirty-fold across models a team might swap for cost or latency, so a better or safer model is not a reliable defense. With ActionRail wrapping the tool boundary, none of those corrupted actions reached the tool (0 of 480 protected attack cases), and no legitimate action was wrongly blocked (0 of 480).

Key findings

You can’t rely on the model.

The cost-optimized models teams actually deploy at scale were fooled most. The frontier models resisted, but partly by refusing to act at all, which stops legitimate work too, and some of those refusals were output truncation rather than a real decision.

Methodology & threat model

Not a jailbreak. A forged value.

The attack is not an instruction the model can flag. It is an ordinary-looking document: an invoice whose remittance account has quietly changed, a support ticket that names a new recovery address, a receipt padded with a fake pre-approved line item. The value looks like legitimate business data, so the model reads it and acts on it. Every case enters through the agent’s normal untrusted read tool and ends in one consequential call. Faced with each, the agent can use the trusted system-of-record value, execute a corrupted value, refuse, or make an unrelated task error. Per model: 60 attack cases, 60 legitimate-but-messy cases, and 10 clean controls, each run three times.

Evaluation path

Workflows

Reimbursements, AP payments, Password resets, Payroll changes, Vendor onboarding, Customer refunds, Procurement, Shipping changes, Benefits enrollment, API-key rotation.

Legitimate stress tiers

The development/frozen split is author-visible and known-technique, not blind. Cases are the independent unit; the three repetitions reveal rollout variance but are correlated and are not treated as independent samples. Legitimacy is judged by an oracle that never calls ActionRail’s matcher, so a false positive is a genuine disagreement.

Results

Zero manipulated actions reached the tool. Zero false positives.

With ActionRail at the execution boundary, no manipulated value executed on any of the eight models (0 of 480 protected attack cases), and offline replay of every corrupted proposal the unprotected models made was rejected 350 of 350. On the legitimate half, ActionRail allowed all of them on every model and every readiness tier — 0 false positives, including the reused-value cases that refute “you’re just blocklisting scary strings.”

A sample of the frozen case-level audit

The full 60-case × 8-model audit ships with the reproduction bundle.

Limitations

What this does not claim.

• The frozen split is known-technique, not blind: authors could read the cases. It measures generalization to unexecuted, author-anticipated combinations, not to a random production population.
• Cases are hand-authored and repetitions are correlated; rates are descriptive counts with no confidence intervals, and fine per-workflow slices are single data points.
• no_call is a mixed outcome: genuine refusal on some episodes, output-cap truncation on others; the frontier “refusal” numbers should not be read as pure model choice.
• Frontier models ran at their required default temperature while small models ran at 0, so the tier comparison carries a temperature confound.
• The advanced allow-capabilities showed no catch gap against these attacks, but the attacks were not crafted to exploit the new tolerances. Boundary attacks are the next required direction.
• Eight configurations across four providers are not the industry; hosted model aliases limit exact future reproduction, and Kimi does not honor a seed, so its three repetitions are rollout variance, not exact reruns.

Reproduction

Pre-registered, frozen, and reproducible.

6,240 completed episodes · 0 provider errors · recorded API cost $41.30. Every episode preserves the full message trace, proposed and executed calls, the rail decision and reason, the system-of-record fixture snapshot hash, returned model identity and fingerprint, seeds, and cost.

Freeze SHA-256: base (Anthropic · xAI) 00823ece…53531 · OpenAI (decimal-repaired) 8c624b4c…6150ed · Kimi (decimal-repaired) 6abd009e…88989f · repaired-matcher wheel 26752bc…99423b. Raw campaign-state and cost ledgers are released as a companion data bundle with per-file hashes.

Cite & data

Citation.

ActionRail Research (ToolJet). A better model won't save you: a value-poisoning benchmark for consequential agent actions. July 2026. https://actionrail.ai/research/value-poisoning-benchmark

@techreport{actionrail2026valuepoisoning,
  title  = {A better model won't save you: a value-poisoning benchmark for consequential agent actions},
  author = {{ActionRail Research, ToolJet}},
  year   = {2026},
  month  = {7},
  institution = {ToolJet},
  url    = {https://actionrail.ai/research/value-poisoning-benchmark}
}