Frequently Asked Questions

Those questions have been asked in GitHub issues for the project by several people. If you have more questions, feel free to open a new issue, and your question and its answer might make it to this FAQ.

I stopped the primary and no failover is happening for 20s to 30s, why?

In order to avoid spurious failovers when the network connectivity is not stable, pg_auto_failover implements a timeout of 20s before acting on a node that is known unavailable. This needs to be added to the delay between health checks and the retry policy.

See the Configuring pg_auto_failover part for more information about how to setup the different delays and timeouts that are involved in the decision making.

See also pg_autoctl watch to have a dashboard that helps understanding the system and what’s going on in the moment.

The secondary is blocked in the CATCHING_UP state, what should I do?

In the pg_auto_failover design, the following two things are needed for the monitor to be able to orchestrate nodes integration completely:

1. Health Checks must be successful

   The monitor runs periodic health checks with all the nodes registered in the system. Those health checks are Postgres connections from the monitor to the registered Postgres nodes, and use the hostname and port as registered.

   The pg_autoctl show state commands column Reachable contains “yes” when the monitor could connect to a specific node, “no” when this connection failed, and “unknown” when no connection has been attempted yet, since the last startup time of the monitor.

   The Reachable column from pg_autoctl show state command output must show a “yes” entry before a new standby node can be orchestrated up to the “secondary” goal state.

2. pg_autoctl service must be running

   The pg_auto_failover monitor works by assigning goal states to individual Postgres nodes. The monitor will not assign a new goal state until the current one has been reached.

   To implement a transition from the current state to the goal state assigned by the monitor, the pg_autoctl service must be running on every node.

When your new standby node stays in the “catchingup” state for a long time, please check that the node is reachable from the monitor given its hostname and port known on the monitor, and check that the pg_autoctl run command is running for this node.

When things are not obvious, the next step is to go read the logs. Both the output of the pg_autoctl command and the Postgres logs are relevant. See the Should I read the logs? Where are the logs? question for details.

Should I read the logs? Where are the logs?

Yes. If anything seems strange to you, please do read the logs.

As maintainers of the pg_autoctl tool, we can’t foresee everything that may happen to your production environment. Still, a lot of efforts is spent on having a meaningful output. So when you’re in a situation that’s hard to understand, please make sure to read the pg_autoctl logs and the Postgres logs.

When using systemd integration, the pg_autoctl logs are then handled entirely by the journal facility of systemd. Please then refer to journalctl for viewing the logs.

The Postgres logs are to be found in the $PGDATA/log directory with the default configuration deployed by pg_autoctl create .... When a custom Postgres setup is used, please refer to your actual setup to find Postgres logs.

My failover is stuck: standbys are in report_lsn and nothing moves

This happens when the primary and one or more quorum standbys (nodes counted by number_sync_standbys) fail at the same time. The monitor drives all surviving standbys to the report_lsn state to determine the most advanced node, but it refuses to promote any candidate while a quorum member is missing. The missing node may have acknowledged a synchronous commit that no surviving standby has yet replicated, and promoting would silently discard those transactions.

First, try to bring the missing node back. If it recovers and reports its LSN, the election resumes automatically.

If the missing node is permanently lost and you accept the data-loss risk, unblock the election:

pg_autoctl perform failover --allow-data-loss

The command promotes the most advanced surviving standby. Transactions acknowledged by the missing node but not yet replicated to any survivor will be permanently lost once the new primary starts accepting writes.

See Recovering a failover stuck in report_lsn and the pgautofailover.guard_data_loss GUC in Configuring pg_auto_failover for a full explanation.

My standby won’t rejoin — Postgres logs a timeline mismatch, what should I do?

This happens when a standby’s local WAL has genuinely diverged from the rest of the group — for example, it was promoted or written to directly at the Postgres level, outside of pg_autoctl’s control, during an incident. Postgres itself refuses the reconnect and logs something like requested timeline N is not a child of this server's history.

This is usually automatic. pg_autoctl walks the upstream’s real timeline history to tell a standby that’s simply behind apart from one that’s genuinely diverged, and the monitor checks every currently-healthy secondary’s ancestry on each of its regular reports — not just at the next incidental transition. As soon as a genuine divergence is found, the node is pushed to catchingup and rewound with pg_rewind (or a fresh pg_basebackup if pg_rewind can’t connect), typically within about a second, with no health-check cycle or maintenance toggle to wait for. See Timeline Forks for the full scenario.

If it doesn’t resolve on its own, check what the monitor knows:

pg_autoctl show timeline

A node flagged FORK: diverges from the reference timeline, pg_rewind required needs an operator decision. This most often means auto-detection couldn’t tell which branch was real — for instance in a two-node formation, where there’s no sibling node to disagree with the diverged one. Confirm which timeline is actually ground truth from other evidence (which node was manually promoted, pg_controldata output, etc.), then pin it:

pg_autoctl accept timeline --tli <N> --reason "..."

The diverged node then rewinds and rejoins automatically. See pg_autoctl show timeline, pg_autoctl accept timeline, and Resolving a detected timeline fork for the full command reference and walkthrough.

The state of the system is blocked, what should I do?

This question is a general case situation that is similar in nature to the previous situation, reached when adding a new standby to a group of Postgres nodes. Please check the same two elements: the monitor health checks are successful, and the pg_autoctl run command is running.

When things are not obvious, the next step is to go read the logs. Both the output of the pg_autoctl command and the Postgres logs are relevant. See the Should I read the logs? Where are the logs? question for details.

Impossible / unresolveable state after crash - How to recover?

The pg_auto_failover Failover State Machine is great to simplify node management and orchestrate multi-nodes operations such as a switchover or a failover. That said, it might happen that the FSM is unable to proceed in some cases, usually after a hard crash of some components of the system, and mostly due to bugs.

Even if we have an extensive test suite to prevent such bugs from happening, you might have to deal with a situation that the monitor doesn’t know how to solve.

The FSM has been designed with a last resort operation mode. It is always possible to unregister a node from the monitor with the pg_autoctl drop node command. This helps the FSM getting back to a simpler situation, the simplest possible one being when only one node is left registered in a given formation and group (state is then SINGLE).

When the monitor is back on its feet again, then you may add your nodes again with the pg_autoctl create postgres command. The command understands that a Postgres service is running and will recover from where you left.

In some cases you might have to also delete the local pg_autoctl state file, error messages will instruct you about the situation.

The monitor is a SPOF in pg_auto_failover design, how should we handle that?

When using pg_auto_failover, the monitor is needed to make decisions and orchestrate changes in all the registered Postgres groups. Decisions are transmitted to the Postgres nodes by the monitor assigning nodes a goal state which is different from their current state.

Consequences of the monitor being unavailable

Nodes contact the monitor each second and call the node_active stored procedure, which returns a goal state that is possibly different from the current state.

The monitor only assigns Postgres nodes with a new goal state when a cluster wide operation is needed. In practice, only the following operations require the monitor to assign a new goal state to a Postgres node:

• a new node is registered

• a failover needs to happen, either triggered automatically or manually

• a node is being put to maintenance

• a node replication setting is being changed.

When the monitor node is not available, the pg_autoctl processes on the Postgres nodes will fail to contact the monitor every second, and log about this failure. Adding to that, no orchestration is possible.

The Postgres streaming replication does not need the monitor to be available in order to deliver its service guarantees to your application, so your Postgres service is still available when the monitor is not available.

To repair your installation after having lost a monitor, the following scenarios are to be considered.

The monitor node can be brought up again without data having been lost

This is typically the case in Cloud Native environments such as Kubernetes, where you could have a service migrated to another pod and re-attached to its disk volume. This scenario is well supported by pg_auto_failover, and no intervention is needed.

It is also possible to use synchronous archiving with the monitor so that it’s possible to recover from the current archives and continue operating without intervention on the Postgres nodes, except for updating their monitor URI. This requires an archiving setup that uses synchronous replication so that any transaction committed on the monitor is known to have been replicated in your WAL archive.

At the moment, you have to take care of that setup yourself. Here’s a quick summary of what needs to be done:

1. Schedule base backups

   Use pg_basebackup every once in a while to have a full copy of the monitor Postgres database available.

2. Archive WAL files in a synchronous fashion

   Use pg_receivewal --sync ... as a service to keep a WAL archive in sync with the monitor Postgres instance at all time.

3. Prepare a recovery tool on top of your archiving strategy

   Write a utility that knows how to create a new monitor node from your most recent pg_basebackup copy and the WAL files copy.

   Bonus points if that tool/script is tested at least once a day, so that you avoid surprises on the unfortunate day that you actually need to use it in production.

A future version of pg_auto_failover will include this facility, but the current versions don’t.

The monitor node can only be built from scratch again

If you don’t have synchronous archiving for the monitor set-up, then you might not be able to restore a monitor database with the expected up-to-date node metadata. Specifically we need the nodes state to be in sync with what each pg_autoctl process has received the last time they could contact the monitor, before it has been unavailable.

It is possible to register nodes that are currently running to a new monitor without restarting Postgres on the primary. For that, the procedure mentioned in Replacing the monitor online must be followed, using the following commands:

$ pg_autoctl disable monitor
$ pg_autoctl enable monitor

Can I use logical replication / CDC tools (Debezium, etc) with pg_auto_failover?

Yes, as long as every node in the formation runs Postgres 17 or more recent. See Change Data Capture (CDC) Continuity for the full picture; the short version:

• pg_auto_failover doesn’t create or manage your logical replication slots itself – that stays your CDC tool’s job, same as with any other Postgres cluster.

• What pg_auto_failover does manage, automatically, on Postgres 17+: the synchronized_standby_slots setting on the primary (kept in sync with the current failover candidates), the dbname Postgres needs in primary_conninfo to run its slot-sync worker, and sync_replication_slots / hot_standby_feedback on every standby. Together, these are what let a logical slot created with failover => true keep working after a promotion, instead of being silently left behind on the old primary.

• wal_level = logical is the default on every node pg_auto_failover creates (see Defaults and opting out) – you only need to set it yourself when opting back in on a cluster created with --wal-level replica, or one created before this default existed (see Turning on wal_level = logical on an already-running cluster). You still need to create your slot(s) with failover => true yourself: pg_auto_failover manages the infrastructure that keeps them working across a promotion, not slot creation itself.

On Postgres versions before 17, Postgres itself has no mechanism to keep a logical slot’s position correct on a standby without losing unconsumed changes, so there is nothing pg_auto_failover can safely do there: a logical slot on the primary does not survive a failover, the same as without pg_auto_failover. This was the situation discussed in issue #400 and issue #1030 before Postgres 17 added slot synchronization.