Action Tests schema

Action Tests use a separate versioned YAML document. The current schema version is 1.

schema_version: 1
cases: []

Unknown fields are rejected at every schema level. Case names must be unique and the document must contain at least one case.

Document fields

Field	Type	Required	Description
schema_version	integer	Yes	Must be 1.
cases	non-empty list	Yes	Ordered Action Test cases.

Case fields

- name: owned delivered order is refundable
  tool: issue_refund
  args: {}
  context: {}
  expect: allow
  fixtures: {}

Field	Type	Required	Description
name	non-empty string	Yes	Unique human-readable contract name.
tool	non-empty string	Yes	Action name configured under tools in the rule file.
args	object	No	Exact proposed tool arguments; defaults to {}.
context	object	No	Exact trusted invocation context; defaults to {}.
expect	enum	Yes	allow, hold, or block.
fixtures	object	No	Source name to fixture. Required for referenced Sources unless --live-sources is enabled.

Missing argument and context fields are allowed because a case may intentionally exercise an incomplete or hostile proposal. The runtime pipeline determines the outcome.

Source fixture

fixtures:
  billing-production:
    by_value:
      order-100:
        customer_id: customer-1
        status: delivered
      order-200:
        - customer_id: customer-1
          status: delivered
        - customer_id: customer-1
          status: refunded
      order-404: null
    default: null

Field	Type	Required	Description
by_value	object	No	Grounded argument value to simulated returned rows. Defaults to {}.
default	object, list, or null	No	Rows returned for an unlisted value. Defaults to no rows.

Each by_value entry and default accepts:

• one object for one row;
• a list of objects for multiple rows;
• null for zero rows.

Fixture keys are compared with the grounded argument’s textual form. The first row supplies fields for column conditions; the complete list length supplies the count for row conditions. This matches built-in Source evaluation semantics.

A fixture Source name must be referenced by the selected action. Unused fixtures and missing required fixtures are rejected to catch naming mistakes.

CLI

actionrail test [CONFIG] [--cases PATH] [--live-sources] [--format text|json]

Option	Default	Description
CONFIG	actionrail.yaml	Runtime rule document.
--cases	actionrail.cases.yaml	Action Tests document.
--live-sources	off	Construct configured Sources for cases without fixtures.
--format	text	Human-readable or stable JSON output.

JSON report

{
  "schema_version": 1,
  "config": "actionrail.yaml",
  "cases": "actionrail.cases.yaml",
  "summary": {"total": 1, "passed": 1, "failed": 0},
  "results": [
    {
      "name": "safe refund",
      "tool": "issue_refund",
      "expected": "allow",
      "actual": "allow",
      "passed": true,
      "reasons": [],
      "preview": "Refund order-100 for 75",
      "source_health": [],
      "error": null
    }
  ]
}

actual is error when a case raises during evaluation. The runner continues with later cases and returns exit code 1.

Python API

from actionrail import ActionTestError, ActionTestReport, run_action_tests

report = run_action_tests(
    "actionrail.yaml",
    "actionrail.cases.yaml",
    live_sources=False,
)

if not report.successful:
    for result in report.results:
        if not result.passed:
            print(result.name, result.expected, result.actual)

run_action_tests() raises ActionTestError when either document is invalid or unreadable. Evaluation failures are captured on their individual result instead.