pgaftest

pgaftest is the pg_auto_failover integration test runner. It reads .pgaf spec files that describe a cluster topology and a set of test steps, spins up the cluster using Docker Compose, and drives the steps to completion.

Two modes of operation are supported:

CI mode (pgaftest run): headless TAP output, non-zero exit on failure.
Interactive mode (pgaftest setup): cluster stays up, shell or tmux session opened for hands-on exploration.

Synopsis 

pgaftest run    [options] <spec.pgaf>
pgaftest run    --schedule <file> [options]
pgaftest setup  [options] <spec.pgaf> [--tmux]
pgaftest step   <step-name>  [--work-dir <dir>]
pgaftest show   <spec.pgaf>
pgaftest prepare <spec.pgaf> [<output-dir>]
pgaftest down   [--work-dir <dir>]

Sub-commands 

`pgaftest run`

Run a spec in CI mode. Docker Compose is started, the setup {} block runs, each step in sequence runs in order, teardown {} always runs, and the stack is removed. Output is TAP: ok N - step_name / not ok.

pgaftest run tests/tap/specs/basic_operation.pgaf

Options:

--schedule <file>: Run every spec listed in the file (one name or path per line).
--expected <dir>: Compare TAP output against .out files in the directory.
--work-dir <dir>: Working directory for compose files and state (default: $TMPDIR/pgaftest/<spec-name>).
--no-cleanup: Leave the compose stack running after the run for post-mortem inspection. Use pgaftest down to clean up manually.

`pgaftest setup`

Start a cluster interactively. Docker Compose is started and the setup {} block runs. Control is then handed back to you.

pgaftest setup tests/tap/specs/basic_operation.pgaf
pgaftest setup docs/tutorial/interactive_tutorial.pgaf --tmux

Options:

--tmux

Open a three-pane tmux session immediately after the cluster is ready. The setup block runs in the bottom pane so you can watch the logs pane while the cluster initialises. The three panes are:

top — docker compose logs -f (live container output)
middle — pg_autoctl watch on the monitor
bottom — interactive bash in the first data node (once setup completes)

--work-dir <dir>

Working directory (default: $TMPDIR/pgaftest/<spec-name>).

`pgaftest step`

Run a single named step against a cluster that was started with pgaftest setup.

pgaftest step stop_primary --work-dir /tmp/pgaftest/basic_operation

The step name must match a step <name> {} block in the spec. The spec file is read from <work-dir>/spec.pgaf.

`pgaftest down`

Run the teardown {} block (if any) and then docker compose down --volumes --remove-orphans.

pgaftest down --work-dir /tmp/pgaftest/basic_operation
pgaftest down tests/tap/specs/basic_operation.pgaf

When called with only --work-dir, the project name is derived from the last path component of the working directory, matching the name that pgaftest setup used when creating the stack.

`pgaftest show`

Print the docker-compose.yml that would be generated from the spec, without running anything.

pgaftest show tests/tap/specs/basic_operation.pgaf

`pgaftest prepare`

Write docker-compose.yml, per-node .ini files, and a Makefile to an output directory for manual inspection or customisation.

pgaftest prepare tests/tap/specs/basic_operation.pgaf ./my-cluster

Spec file format 

A .pgaf spec file has four top-level sections. Only cluster {} is required; the others are optional.

# comments start with #

cluster {
    ...             # topology declaration
}

setup {
    ...             # commands to run after compose up
}

teardown {
    ...             # commands to run on cleanup
}

step <name> {
    ...             # named test step
}

sequence
    step1
    step2
    ...

The `cluster {}` block 

Declares the Docker Compose topology. Every cluster must have at least one formation with at least one node.

cluster {
    monitor                       # required for multi-node HA
    image "pg_auto_failover:pg17" # optional; default: build from source

    ssl   self-signed             # self-signed | cert | off (default: self-signed)
    auth  trust                   # trust | md5 | scram     (default: trust)

    formation {                   # optional name; default name: "default"
        node1
        node2
        node3  async  candidate-priority 0
    }

    # Citus: multiple formations
    formation workers num-sync 1 {
        w1  worker  group 1
        w2  worker  group 1
    }
}

Node modifiers:

`async`	Mark as async standby (no sync quorum)
`candidate-priority <N>`	Failover priority 0–100 (default: 50)
`launch deferred`	Container starts with `sleep infinity`; use `exec node pg_autoctl node start`
`coordinator` / `worker group <N>`	Citus role
`no-monitor`	Standalone node (no monitor)
`listen`	Bind all interfaces (`--listen 0.0.0.0`)
`auth <method>`	Per-node auth override
`ssl <mode>`	Per-node SSL override
`volume <name> <path>`	Mount a named Docker volume at `<path>`

Commands inside `setup`, `teardown`, and `step` blocks 

State waits

wait until <node> state is <state>  [timeout <N>s]
wait until primary, secondary       [timeout <N>s]
wait until <node> stopped           [timeout <N>s]

The multi-node form wait until primary, secondary waits until at least one node in each named state is present in the formation.

Assertions

assert <node> state is <state>
assert <node> assigned-state is <state>
assert <node> stays <state> while {
    # commands that must not trigger a transition
}

Execute commands in a container

exec        <service>  <command...>   # must exit 0
exec-fails  <service>  <command...>   # must exit non-zero

SQL

sql <service> { SELECT ... }
expect { <expected output> }
expect error [<SQLSTATE>]

expect matches the output of the immediately preceding sql command. Fields are separated by whitespace; rows are wrapped in { }.

Network

network disconnect <node>
network connect    <node>

Compose lifecycle

compose stop  <service>
compose start <service>
compose kill  <service>
compose down

PostgreSQL control

stop postgres  <node>
start postgres <node>

Failover

promote <node> [, <node2>, ...]

Monitor targeting (replace-monitor tests)

set monitor <service>

Log grep

logs <service> [not] <pattern>

Sleep

sleep <N>s

Environment variables 

PGAF_IMAGE

Override the Docker image used for all node containers. Useful in CI to avoid rebuilding the image on every run:

PGAF_IMAGE=pgaf:run pgaftest run tests/tap/specs/basic_operation.pgaf

PGAFTEST_COMPOSE_SERVICE

Name of the service inside which pgaftest itself runs when executing inside a container. Used by the CI workflow to locate the binary.

PGAFTEST_HOST_WORK_DIR

Host-side working directory when pgaftest runs inside a container. Bind-mounted so compose files written by the binary are visible to the Docker daemon on the host.

TMPDIR

Base directory for auto-derived working directories (default: /tmp). Working directories are placed under $TMPDIR/pgaftest/<spec-name>.

TAP output 

In run mode, pgaftest produces TAP version 13 output on standard output. Each named step in sequence becomes one test point:

TAP version 13
1..4
ok 1 - stop_primary
ok 2 - check_failover
ok 3 - restart_node1
ok 4 - verify_replication

On failure:

not ok 2 - check_failover
# DIAG: wait until node2 state is primary: timed out after 90s

Examples 

Run a single spec:

pgaftest run tests/tap/specs/basic_operation.pgaf

Run the full schedule:

pgaftest run --schedule tests/tap/schedule

Bring up an interactive cluster, watch it in tmux:

pgaftest setup docs/tutorial/interactive_tutorial.pgaf --tmux

Run a named step against a live cluster:

pgaftest step stop_primary --work-dir /tmp/pgaftest/basic_operation

Tear down a cluster started with --no-cleanup or setup:

pgaftest down --work-dir /tmp/pgaftest/basic_operation

Print the generated docker-compose.yml without running anything:

pgaftest show tests/tap/specs/basic_operation.pgaf