Run Action Tests in CI

Action Tests are designed to be a fast required check. Deterministic fixtures do not need a model, Console, database service, Source credentials, or network access.

GitHub Actions

.github/workflows/action-tests.yml

name: Action contracts

on:
  pull_request:
  push:
    branches: [master]

jobs:
  action-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: pip
      - run: python -m pip install actionrail
      - run: actionrail test

The job fails when an expected allow, hold, or block changes. Invalid rules and invalid test documents also fail the job before cases execute.

Stable JSON output

Use JSON when another CI step needs to publish or inspect results:

actionrail test --format json > actionrail-test-report.json

The JSON document has summary and results fields. Result objects include the case name, action name, expected and actual outcomes, pass status, local decision reasons, preview, Source health, and any runtime error.

Action Tests can contain representative business identifiers in fixtures and local reasons. Treat the JSON report as test output: do not publish it to an untrusted artifact store unless the fixture data is safe to disclose.

Exit codes

Code	Meaning	CI interpretation
0	Every case matched its expected outcome.	Pass.
1	One or more outcomes mismatched, or a case raised at runtime.	Contract regression.
2	The rules or Action Tests document is invalid or unreadable.	Configuration error.

Separate deterministic and live checks

Use two stages when you also want adapter integration coverage:

1. Every pull request: actionrail test with fixtures. No secrets or services.
2. Protected or scheduled environment: actionrail test --live-sources with least-privilege test credentials and disposable data.

Never point live Action Tests at a writable production identity. Built-in SQL adapters require read-only transactions, but database privileges remain the primary security boundary.

Review rule changes as code

For a pull request that changes actionrail.yaml, require the author to change or add cases when intended behavior changes. A reviewer should be able to see the proposed action, trusted facts, simulated Source response, and expected decision together in one diff.

Avoid updating an expectation only to make CI green. Confirm why the decision contract changed and whether the change expands the set of actions that can reach a consequential tool.