Public beta — not for production use. Data may be wiped at any time. Questions? Contact us.
A
Axiom
Sign in
Documentation menu

Test a flow

Write tests/*.test.yaml cases that assert on a flow's real, mocked run — status, typed output, which nodes ran, and typed failures — with axiom flow test.

View as Markdown

Mocking a node gets a flow to run on fixtures instead of real side effects. It does not check that the run did what you meant — axiom flow run prints a result and stops there. axiom flow test is the assertion layer: tests/<name>.test.yaml files beside flow.yaml declare cases — an input, an optional per-case mock overlay, and typed expectations — and each case runs through the exact same real compile+invoke path every other flow command uses. There is no separate, simpler evaluator standing in for the worker: a case's green means the real engine produced what you expected.

Prerequisites: read Mock nodes while authoring a flow first — axiom flow test cases build directly on flow.yaml's mocks: section and the same fixture dialect.

Write a test file

Create tests/<name>.test.yaml next to your flow.yaml:

# tests/triage.test.yaml
cases:
  - name: billing category routes to notify
    input: {text: "card declined twice"}
    mocks:
      classify:
        output: {content: "category: billing", model: "gpt-4o-mini"}
    expect:
      status: success
      output:
        subset: {channel: "#support"}
      nodes:
        executed: [classify, notify]

  - name: upstream reset fails the run
    input: {text: "card declined twice"}
    mocks:
      classify: {error: {code: TRANSPORT}}
    expect:
      status: failure
      error: {node: classify, code: TRANSPORT}

Run every suite in tests/:

axiom flow test support-triage.flow.yaml
✓ PASS   triage/billing category routes to notify
✓ PASS   triage/upstream reset fails the run

2 passed, 0 failed (2 case(s))

A failing case prints a diff naming the case, the expectation, and want/got, and the command exits non-zero:

✗ FAIL   triage/billing category routes to notify
    - output.channel: want "#support", got "#general"

0 passed, 1 failed (1 case(s))

Run a single case while you're iterating:

axiom flow test support-triage.flow.yaml --case "billing category routes to notify"

Or get the whole report as one JSON object (--json) for CI:

axiom flow test support-triage.flow.yaml --json

Case schema

What the case sends

FieldMeaning
nameRequired. Identifies the case in reports and --case; must be unique within its file.
inputThe start node's (or input facade's) JSON input — the same dialect axiom flow run -d takes.
mocksOptional. A per-case overlay over flow.yaml's own mocks: section (same output/file/error shape as flow.yaml mocks). An alias here replaces flow.yaml's mock for that alias wholesale — never a field-by-field merge — and is never written back to the file.
resumeOptional. The human decision a config.hitl hold is waiting for, keyed by the pausing node's alias. Posted the moment that hold appears, and recorded against the pause as resumed_by: axiom flow test. The value must decode into the paused node's own input message — that is what a resume replaces. It is an assertion: an alias that never holds for a human fails the case, so a flow with no config.hitl cannot pass a signed-path case by running straight through.

What the case asserts

FieldMeaning
expect.statusRequired. Either a union — success (completed successfully) or failure (any of FAILED / CANCELLED / COMPENSATED) — or one terminal state by name: completed, failed, cancelled, compensated. Prefer the name whenever the case exists to prove a particular ending; see Name the ending, don't just assert "not success".
expect.outputOptional. equals: {...} (exact match) or subset: {...} (every key in subset must be present and equal in the actual decoded output; extra actual keys are ignored). Declare at most one of those two. Arrays inside either must match exactly — a partial-array match is ambiguous, so it isn't attempted. equals: {} is a real assertion: this branch published an empty output.
expect.output.fieldsOptional, and combinable with equals/subset or usable alone. A map of dotted output path → present, absent, or non_empty. This is how you assert that something happened without pinning what it said — error_message: non_empty passes a flow that surfaces the problem in its own words and fails one that swallows it, where an exact match would have graded the wording of somebody else's node. non_empty rejects null, "", [] and {}; a number or boolean is always non-empty, 0 and false included. absent is a statement about the output document, so remember a proto zero value may simply not be on the wire.
expect.errorOptional, only meaningful with a failing status (failure, failed, cancelled, compensated). node (the failed alias) and/or code (the classified NodeErrorCode — see the boundary below).
expect.nodesOptional. executed / not_executed — lists of aliases, checked as subsets: name only the nodes this case is actually about, not the whole graph. not_executed is how you pin a wrong-branch regression: the branch that should NOT have fired must show up here, not merely be absent from the output.

Unknown YAML keys anywhere in a test file are a hard parse error (same strict decoding as flow.yaml), and an alias in mocks:/resume:/expect.nodes/ expect.error.node that doesn't exist in the flow is caught before any network round trip.

Name the ending, don't just assert "not success"

status: failure is a union over the three ways a run can end badly: FAILED, CANCELLED and COMPENSATED. That is the right assertion when the case genuinely does not care which one it got — a declined card, say, where all you want is "this did not go through."

It is the wrong assertion when the case exists to prove a specific ending, because "the run did not succeed" is something a flow can reach by many routes, most of them nothing to do with what you are testing. A case written to prove a human-approval timeout cancels the run:

- name: the approval timed out and the run was cancelled
  input: {amount: 5000}
  expect:
    status: failure          # ← passes for ANY bad ending

also passes for a flow that never pauses at all and simply dead-ends an edge so the run fails. The suite goes green, and the mechanism it was written for was never exercised. Name the state instead:

    status: cancelled        # ← only a CANCELLED run satisfies this

completed is success with the state spelled out, and behaves identically (expect.output still compares against the run's output).

One honest limit, because it changes what you can assert: the synchronous invoke answer reports only success/not-success, so to check an exact name the CLI reads the execution record. If that read does not resolve, the case is reported as cannot evaluate — never guessed, and never quietly downgraded to the union. If you see that line, either the run is still settling or the record is unreachable; assert failure if the case truly does not care which ending it got.

A case that declares resume: runs asynchronously rather than on the synchronous wait: the ingress answers a HITL pause and a plain wait expiry with the same shape, and only after the full --timeout has elapsed, so waiting would cost the case its whole timeout before the CLI even learned there was a hold to answer. Two placements of the same registry node that both hold for a human are refused rather than guessed at — a pause record names the node, not the canvas alias, so the two holds are indistinguishable.

What actually runs

Each case: merge the mock overlay onto flow.yaml's mocks:, compile a MOCKED dev artifact from the merge (the same POST /graphs/compile axiom flow compile uses), then invoke it (the same POST /v1/flows/invoke axiom flow run uses, including its terminal-state polling). Compiling is cached per distinct effective mock set — cases that only vary input: compile once and invoke many times.

If a case's mock set fails to compile (a mock's declared shape doesn't match the node's real output descriptor, say), that case reports as an ERROR naming the registry's diagnostic — it does not abort the rest of the run. This is deliberate: it is what lets axiom flow test catch a flow that validates locally but dies at compile, as an ordinary, readable test result instead of a crash.

Boundary — read this before trusting a green suite

A green suite means "the graph behaves." It does not mean "the deployed app works." Every node in a case is either mocked or fed entirely by mocked upstream output, so axiom flow test cannot catch:

  • Routing — no ingress/gateway path beyond the invoke call itself.
  • Secret delivery — a mocked node's required_secrets is moot; a flow broken specifically because a secret is missing or wrong at the sidecar boundary looks identical to one that's fine.
  • Cold starts — a mocked node never scales a ksvc from zero.
  • Real-database constraints — a mocked node's output is authored, not queried; a unique-constraint violation or a real empty-result-set shape your mock doesn't reproduce is invisible.
  • Cross-app descriptor collisions.

Keep axiom flow run against a live invocation — and your publish-time smoke test — for all of the above. axiom flow test is the middle tier between a unit test and a live run, not a replacement for either end.

error.code is a best-effort classification

The platform does not expose the typed NodeErrorCode enum as a structured field on any endpoint axiom flow test reads (that's a server change, not in this command's scope). What it does instead: an error: mock declared without a custom message produces the worker's own fixed catalog string for that code (e.g. TIMEOUT → "node execution timed out"), and axiom flow test reverse-matches that string. This is reliable for the common, documented mock form:

mocks:
  charge_card: {error: {code: TIMEOUT}}      # classifies: TIMEOUT

A custom message defeats it — the command reports UNCLASSIFIABLE rather than guessing:

mocks:
  charge_card: {error: {code: TIMEOUT, message: "card_declined"}}  # UNCLASSIFIABLE

If a case only needs expect.error.node, this doesn't matter — the failed alias comes straight off the execution's event timeline, not off message text.

Not yet built

A resume: for axiom flow run (there is no --resume-with flag; only a suite can answer a hold today), cel: output assertions, --watch, --record (mint a case's mocks from a live invoke, mirroring axiom flow mock add --from-execution), and app-builder codegen emitting a suite per generated flow automatically are tracked as follow-ons — see ADR-200 in the knowledge base for the full list.

Related pages

Related
← Mock nodes while authoring a flowPause a flow for a human decision →