Test action behavior before an agent runs

Action Tests are executable contracts for consequential tool calls. Each case supplies a proposed action, trusted invocation context, deterministic Source records, and the outcome ActionRail must return:

proposed action + trusted context + Source fixtures → allow | hold | block

The runner calls the same rule loader, policy evaluator, grounding matcher, and decision pipeline used during runtime enforcement. It does not maintain a simplified test-only interpretation of your rules.

That makes Action Tests useful for reviewing rule changes in the same way you review application code. A pull request can prove that a legitimate refund is still allowed, a high-value refund is still held, and a cross-customer refund is still blocked.

Create an action contract

Keep runtime rules in actionrail.yaml and test cases in actionrail.cases.yaml. The test file has its own versioned schema:

actionrail.cases.yaml

schema_version: 1

cases:
  - name: cross-customer refund is blocked
    tool: issue_refund
    args:
      order_id: order-100
      amount: 75
    context:
      customer_id: customer-2
    expect: block
    fixtures:
      billing-production:
        by_value:
          order-100:
            customer_id: customer-1
            status: delivered

This case exercises a rule that grounds order_id through the billing-production Source and compares the returned customer_id with trusted context. The proposed order is real, but it belongs to another customer, so the required contract is block.

Run the contract

From the directory containing both files:

actionrail test

Or pass explicit paths:

actionrail test config/actionrail.yaml \
  --cases tests/actionrail.cases.yaml

The command prints each expected and actual outcome plus the local decision reasons:

ActionRail Action Tests

PASS  owned delivered order is refundable
      issue_refund: expected allow, got allow
PASS  cross-customer refund is blocked
      issue_refund: expected block, got block
PASS  refund above the automatic limit is held
      issue_refund: expected hold, got hold

3 passed · 0 failed

Exit code 0 means every contract passed. A mismatch returns 1; an invalid rule or Action Tests document returns 2. See Continuous integration for a ready-to-use CI pattern.

Fixtures model Sources, not adapters

A fixture is keyed by the Source name referenced in the runtime rule. It provides the records that Source would return for an argument value:

fixtures:
  billing-production:
    by_value:
      order-100:
        customer_id: customer-1
        status: delivered
      order-404: null

• an object represents one returned record;
• a list represents multiple returned records;
• null represents zero returned records;
• default can define rows for values not listed under by_value.

The shared matcher still evaluates row-count conditions, field operators, trusted-context comparisons, cross-argument comparisons, and time conditions. Only the adapter I/O is replaced.

Fixtures are intentionally Source-oriented rather than Postgres-, MySQL-, HTTP-, or MCP-specific. The same contract remains valid if a team changes the adapter behind billing-production without changing the meaning of its rules.

Production Sources are off by default

Normal Action Tests never construct or call the Sources declared in actionrail.yaml. Every Source referenced by the selected action must have a fixture. This gives pull-request CI three useful properties:

• no production credentials are required;
• tests are repeatable and fast;
• a test command cannot unexpectedly query a customer system.

Use --live-sources only for an explicit integration-test stage:

actionrail test --live-sources

When enabled, a case may omit a fixture and use the configured Source instead. Fixtures still take precedence, so one suite can combine deterministic and live checks. Treat this option as real network and data-plane access.

Test hostile and incomplete proposals

args and context describe the exact proposal being tested. They may omit a field intentionally. This is useful for proving fail-closed behavior when an agent omits a required value or authenticated context is unavailable:

- name: missing trusted customer is blocked
  tool: issue_refund
  args:
    order_id: order-100
    amount: 75
  context: {}
  expect: block
  fixtures:
    billing-production:
      by_value:
        order-100:
          customer_id: customer-1
          status: delivered

Recommended minimum coverage for each consequential action:

1. one ordinary action that must be allowed;
2. every policy boundary that must hold;
3. wrong-owner, wrong-state, missing-record, and stale-record blocks;
4. missing or malformed proposed arguments;
5. missing trusted-context fields;
6. Source outage behavior in a separate live or lower-level integration test.

Run the repository example

An editable checkout includes a complete four-case contract:

actionrail test examples/action_tests/actionrail.yaml \
  --cases examples/action_tests/actionrail.cases.yaml

It declares a production-style Postgres Source but uses fixtures, so the example requires neither Postgres nor credentials.

For every accepted field, continue to the Action Tests schema reference.