--- title: "Test a flow" description: "Write tests/*.test.yaml cases that assert on a flow's real, mocked run — status, typed output, which nodes ran, and typed failures — with axiom flow test." category: guide surfaces: [cli] related: [guides/mock-nodes-in-a-flow, guides/publish-a-flow, guides/debug-a-flow, reference/error-catalog, concepts/execution-model] last_reviewed: 2026-08-17 --- # Test a flow [Mocking a node](mock-nodes-in-a-flow.md) gets a flow to *run* on fixtures instead of real side effects. It does not check that the run did what you meant — `axiom flow run` prints a result and stops there. **`axiom flow test`** is the assertion layer: `tests/.test.yaml` files beside `flow.yaml` declare cases — an input, an optional per-case mock overlay, and typed expectations — and each case runs through the exact same real compile+invoke path every other flow command uses. There is no separate, simpler evaluator standing in for the worker: a case's green means the real engine produced what you expected. Prerequisites: read [Mock nodes while authoring a flow](mock-nodes-in-a-flow.md) first — `axiom flow test` cases build directly on flow.yaml's `mocks:` section and the same fixture dialect. ## Write a test file Create `tests/.test.yaml` next to your `flow.yaml`: ```yaml # tests/triage.test.yaml cases: - name: billing category routes to notify input: {text: "card declined twice"} mocks: classify: output: {content: "category: billing", model: "gpt-4o-mini"} expect: status: success output: subset: {channel: "#support"} nodes: executed: [classify, notify] - name: upstream reset fails the run input: {text: "card declined twice"} mocks: classify: {error: {code: TRANSPORT}} expect: status: failure error: {node: classify, code: TRANSPORT} ``` Run every suite in `tests/`: ```bash axiom flow test support-triage.flow.yaml ``` ```text ✓ PASS triage/billing category routes to notify ✓ PASS triage/upstream reset fails the run 2 passed, 0 failed (2 case(s)) ``` A failing case prints a diff naming the case, the expectation, and want/got, and the command exits non-zero: ```text ✗ FAIL triage/billing category routes to notify - output.channel: want "#support", got "#general" 0 passed, 1 failed (1 case(s)) ``` Run a single case while you're iterating: ```bash axiom flow test support-triage.flow.yaml --case "billing category routes to notify" ``` Or get the whole report as one JSON object (`--json`) for CI: ```bash axiom flow test support-triage.flow.yaml --json ``` ## Case schema ### What the case sends | Field | Meaning | |---|---| | `name` | Required. Identifies the case in reports and `--case`; must be unique within its file. | | `input` | The start node's (or [input facade's](type-a-flow-contract.md)) JSON input — the same dialect `axiom flow run -d` takes. | | `mocks` | Optional. A **per-case overlay** over flow.yaml's own `mocks:` section (same `output`/`file`/`error` shape as [flow.yaml mocks](mock-nodes-in-a-flow.md#declare-a-mock-in-flowyaml)). An alias here **replaces** flow.yaml's mock for that alias wholesale — never a field-by-field merge — and is never written back to the file. | | `resume` | Optional. The human decision a [`config.hitl`](pause-for-human-approval.md) hold is waiting for, keyed by the **pausing node's alias**. Posted the moment that hold appears, and recorded against the pause as `resumed_by: axiom flow test`. The value must decode into the **paused node's own input message** — that is what a resume replaces. **It is an assertion:** an alias that never holds for a human fails the case, so a flow with no `config.hitl` cannot pass a signed-path case by running straight through. | ### What the case asserts | Field | Meaning | |---|---| | `expect.status` | Required. Either a **union** — `success` (completed successfully) or `failure` (any of `FAILED` / `CANCELLED` / `COMPENSATED`) — or **one terminal state by name**: `completed`, `failed`, `cancelled`, `compensated`. Prefer the name whenever the case exists to prove a *particular* ending; see [Name the ending, don't just assert "not success"](#name-the-ending-dont-just-assert-not-success). | | `expect.output` | Optional. `equals: {...}` (exact match) or `subset: {...}` (every key in `subset` must be present and equal in the actual decoded output; extra actual keys are ignored). Declare at most one of those two. Arrays inside either must match exactly — a partial-array match is ambiguous, so it isn't attempted. `equals: {}` is a real assertion: *this branch published an empty output*. | | `expect.output.fields` | Optional, and combinable with `equals`/`subset` or usable alone. A map of dotted output path → `present`, `absent`, or `non_empty`. This is how you assert **that** something happened without pinning **what it said** — `error_message: non_empty` passes a flow that surfaces the problem in its own words and fails one that swallows it, where an exact match would have graded the wording of somebody else's node. `non_empty` rejects `null`, `""`, `[]` and `{}`; a number or boolean is always non-empty, `0` and `false` included. `absent` is a statement about the output document, so remember a proto zero value may simply not be on the wire. | | `expect.error` | Optional, only meaningful with a failing status (`failure`, `failed`, `cancelled`, `compensated`). `node` (the failed alias) and/or `code` (the classified NodeErrorCode — see the boundary below). | | `expect.nodes` | Optional. `executed` / `not_executed` — lists of aliases, checked as **subsets**: name only the nodes this case is actually about, not the whole graph. `not_executed` is how you pin a wrong-branch regression: the branch that should NOT have fired must show up here, not merely be absent from the output. | Unknown YAML keys anywhere in a test file are a hard parse error (same strict decoding as `flow.yaml`), and an alias in `mocks:`/`resume:`/`expect.nodes`/ `expect.error.node` that doesn't exist in the flow is caught before any network round trip. ## Name the ending, don't just assert "not success" `status: failure` is a **union** over the three ways a run can end badly: `FAILED`, `CANCELLED` and `COMPENSATED`. That is the right assertion when the case genuinely does not care which one it got — a declined card, say, where all you want is "this did not go through." It is the wrong assertion when the case exists to prove a **specific** ending, because "the run did not succeed" is something a flow can reach by many routes, most of them nothing to do with what you are testing. A case written to prove a human-approval timeout cancels the run: ```yaml - name: the approval timed out and the run was cancelled input: {amount: 5000} expect: status: failure # ← passes for ANY bad ending ``` also passes for a flow that never pauses at all and simply dead-ends an edge so the run fails. The suite goes green, and the mechanism it was written for was never exercised. Name the state instead: ```yaml status: cancelled # ← only a CANCELLED run satisfies this ``` `completed` is `success` with the state spelled out, and behaves identically (`expect.output` still compares against the run's output). One honest limit, because it changes what you can assert: the synchronous invoke answer reports only success/not-success, so to check an exact name the CLI reads the execution record. If that read does not resolve, the case is reported as **`cannot evaluate`** — never guessed, and never quietly downgraded to the union. If you see that line, either the run is still settling or the record is unreachable; assert `failure` if the case truly does not care which ending it got. A case that declares `resume:` runs **asynchronously** rather than on the synchronous wait: the ingress answers a HITL pause and a plain wait expiry with the same shape, and only after the full `--timeout` has elapsed, so waiting would cost the case its whole timeout before the CLI even learned there was a hold to answer. Two placements of the **same registry node** that both hold for a human are refused rather than guessed at — a pause record names the node, not the canvas alias, so the two holds are indistinguishable. ## What actually runs Each case: merge the mock overlay onto `flow.yaml`'s `mocks:`, compile a [MOCKED dev artifact](mock-nodes-in-a-flow.md#compile-a-mocked-flow) from the merge (the same `POST /graphs/compile` `axiom flow compile` uses), then invoke it (the same `POST /v1/flows/invoke` `axiom flow run` uses, including its terminal-state polling). Compiling is cached per distinct effective mock set — cases that only vary `input:` compile once and invoke many times. If a case's mock set fails to compile (a mock's declared shape doesn't match the node's real output descriptor, say), that case reports as an `ERROR` naming the registry's diagnostic — it does not abort the rest of the run. This is deliberate: it is what lets `axiom flow test` catch a flow that *validates* locally but *dies* at compile, as an ordinary, readable test result instead of a crash. ## Boundary — read this before trusting a green suite **A green suite means "the graph behaves." It does not mean "the deployed app works."** Every node in a case is either mocked or fed entirely by mocked upstream output, so `axiom flow test` cannot catch: - **Routing** — no ingress/gateway path beyond the invoke call itself. - **Secret delivery** — a mocked node's `required_secrets` is moot; a flow broken specifically because a secret is missing or wrong at the sidecar boundary looks identical to one that's fine. - **Cold starts** — a mocked node never scales a ksvc from zero. - **Real-database constraints** — a mocked node's output is authored, not queried; a unique-constraint violation or a real empty-result-set shape your mock doesn't reproduce is invisible. - **Cross-app descriptor collisions.** Keep `axiom flow run` against a live invocation — and your publish-time smoke test — for all of the above. `axiom flow test` is the middle tier between a unit test and a live run, not a replacement for either end. ### `error.code` is a best-effort classification The platform does not expose the typed `NodeErrorCode` enum as a structured field on any endpoint `axiom flow test` reads (that's a server change, not in this command's scope). What it does instead: an `error:` mock declared **without a custom message** produces the worker's own fixed catalog string for that code (e.g. `TIMEOUT` → "node execution timed out"), and `axiom flow test` reverse-matches that string. This is reliable for the common, documented mock form: ```yaml mocks: charge_card: {error: {code: TIMEOUT}} # classifies: TIMEOUT ``` A **custom** message defeats it — the command reports `UNCLASSIFIABLE` rather than guessing: ```yaml mocks: charge_card: {error: {code: TIMEOUT, message: "card_declined"}} # UNCLASSIFIABLE ``` If a case only needs `expect.error.node`, this doesn't matter — the failed alias comes straight off the execution's event timeline, not off message text. ## Not yet built A `resume:` for **`axiom flow run`** (there is no `--resume-with` flag; only a suite can answer a hold today), `cel:` output assertions, `--watch`, `--record` (mint a case's mocks from a live invoke, mirroring `axiom flow mock add --from-execution`), and app-builder codegen emitting a suite per generated flow automatically are tracked as follow-ons — see ADR-200 in the knowledge base for the full list. ## Related pages - [Mock nodes while authoring a flow](mock-nodes-in-a-flow.md) — the `mocks:` substrate this command asserts against. - [Publish a flow to the marketplace](publish-a-flow.md) — a suite never publishes anything; the compiled artifact under test stays a mocked dev artifact throughout. - [Debug a flow](debug-a-flow.md) — inspect a case's execution directly (`axiom executions get `) when a failure report needs more context than the diff line gives you. - [Error catalog](../reference/error-catalog.md) — compile-time mock rejections in full.