High availability reference queries

This guide provides a complete reference for all commands used to manage a Memgraph High Availability (HA) cluster.

Cluster registration commands

ADD COORDINATOR

Adds a coordinator to the cluster.

ADD COORDINATOR coordinatorId WITH CONFIG {
  "bolt_server": boltServer, 
  "coordinator_server": coordinatorServer, 
  "management_server": managementServer
}; 

Parameters

• coordinatorId (int) Unique ID for each coordinator. Typically incremented sequentially.
• boltServer (string) External Bolt endpoint: "IP_OR_DNS:PORT". Usually port 7687. Must be reachable by external applications.
• coordinatorServer (string) Internal raft endpoint: "HOSTNAME_OR_DNS:COORDINATOR_PORT".
• managementServer (string) Coordinator management endpoint: "HOSTNAME_OR_DNS:MANAGEMENT_PORT".

DNS/FQDN is recommended when IPs are ephemeral (e.g., Kubernetes).

Behavior & implications

• Can be run before or after registering data instances.
• Must be executed once for each coordinator.
• External Bolt server must be reachable; Raft and management endpoints may be internal.
• Writes cluster configuration to Raft log.

Example

ADD COORDINATOR 1 WITH CONFIG {
  "bolt_server": "my_outside_coordinator_1_IP:7687",
  "coordinator_server": "memgraph-coordinator-1.default.svc.cluster.local:12000",
  "management_server": "memgraph-coordinator-1.default.svc.cluster.local:10000"
};

REMOVE COORDINATOR

Removes a follower coordinator from the cluster.

REMOVE COORDINATOR coordinatorId;

Parameters

• coordinatorId (int) — previously registered coordinator ID.

Behavior & implications

• Leader coordinator cannot remove itself. To remove the leader, first trigger a leadership change with YIELD LEADERSHIP and then run REMOVE COORDINATOR once a new leader has been elected.

Example

REMOVE COORDINATOR 2;

UPDATE CONFIG

Updates the configuration of the data instance/coordinator in the cluster. Use this command to update the DNS of your bolt servers, for example, when migrating to new load balancers.

UPDATE CONFIG FOR ( INSTANCE instanceName | COORDINATOR coordinatorServerId ) configsMap=configMap ;

Parameters

• instanceName (string) — the name of the data instance for which the configuration changes will be applied.
• coordinatorServerId (int) — previously registered coordinator ID for which the configuration changes will be applied.
• configsMaps (int) — Map of KV pairs that signal which field should be updated. Currently, it is only supported to update bolt server of each instance.

Behavior & implications

• Only bolt server can be updated.

Example

UPDATE CONFIG FOR COORDINATOR 1 {'bolt_server': '127.0.0.1:7690'};
UPDATE CONFIG FOR INSTANCE instance_2 {'bolt_server': '127.0.0.1:7688'}

REGISTER INSTANCE

Registers a data instance in the cluster.

REGISTER INSTANCE instanceName ( AS ASYNC | AS STRICT_SYNC ) ? WITH CONFIG {
  "bolt_server": boltServer, 
  "management_server": managementServer, 
  "replication_server": replicationServer
};

Parameters

• instanceName Unique symbolic name of the data instance.
• AS ASYNC / AS STRICT_SYNC (optional) Selects replication mode. Default: SYNC.
• boltServer External Bolt endpoint: "IP_OR_DNS:PORT".
• managementServer Coordinator → data instance health check endpoint.
• replicationServer Replication endpoint, typically using port 20000.

Behavior

• The operation is first committed to the Raft log and acknowledged by a majority of coordinators.
• Coordinator connects via management_server to verify liveness.
• Coordinator begins periodic health checks.
• Instance is automatically demoted to REPLICA.
• Replication server is started on the data instance.
• If RPCs to the data instance fail (e.g., due to a transient network issue), the registration still succeeds. The reconciliation loop automatically retries the RPCs.

Replication mode rules

• Allowed combinations:

  • STRICT_SYNC + ASYNC
  • SYNC + ASYNC

• Forbidden combination:

  • STRICT_SYNC + SYNC

Notes

• In Kubernetes, use service DNS names (e.g. memgraph-data-1.default.svc.cluster.local).
• Local development uses localhost.
• Register instances only while every database on the MAIN is in the IN_MEMORY_TRANSACTIONAL storage mode. If the MAIN is in analytical mode, the query still returns success (the Raft commit is the success criterion) but the replica is not attached until the MAIN switches back, at which point the reconciliation loop attaches it. The data instance logs the reason for the rejection. See Bulk import in analytical mode.

Example

REGISTER INSTANCE instance1 WITH CONFIG {
  "bolt_server": "my_outside_instance1_IP:7687",
  "management_server": "memgraph-data-1.default.svc.cluster.local:10000",
  "replication_server": "memgraph-data-1.default.svc.cluster.local:20000"
};

UNREGISTER INSTANCE

Removes a data instance from the cluster.

UNREGISTER INSTANCE instanceName;

Parameters

• instanceName — name of the data instance.

Implications

• Do not unregister the MAIN instance; this may corrupt cluster state.
• A healthy MAIN must exist during the operation.
• The instance is removed from the Raft state first. If the RPC to unregister the replica from MAIN fails, the reconciliation loop automatically retries the operation.
• The unregistered instance keeps running and keeps all of its data. It is removed from the cluster, not wiped, so you do not need to clear its data directory before registering it back.
• The MAIN refuses the unregister RPC while any of its databases is in the IN_MEMORY_ANALYTICAL storage mode, and logs the reason. The instance is still removed from the Raft state, but the MAIN keeps its replication client until the reconciliation loop retries once every database is transactional again. Unregister instances only while the cluster is in transactional mode — see Bulk import in analytical mode.

Example

UNREGISTER INSTANCE instance_1;

Replication role management

SET INSTANCE ... TO MAIN

Promotes a replica to MAIN.

SET INSTANCE instanceName TO MAIN;

Behavior

• The promotion is first committed to the Raft log and acknowledged by a majority of coordinators.
• All other registered instances become replicas of the new MAIN.
• RPCs (PromoteToMainRpc, SwapAndUpdateUUID) are sent to data instances on a best-effort basis. If they fail, the reconciliation loop automatically retries them.

Implications

• Fails if a MAIN already exists.

Example

SET INSTANCE instance_0 TO MAIN;

DEMOTE INSTANCE

Demotes the current MAIN to a REPLICA.

DEMOTE INSTANCE instanceName;

Behavior

• The role change is first committed to the Raft log and acknowledged by a majority of coordinators.
• MAIN becomes REPLICA.
• The DemoteMainToReplicaRpc is sent on a best-effort basis. If it fails, the reconciliation loop automatically retries it.
• Returns an error if the instance is already a REPLICA.

Implications

• Failover is not automatic after demotion. You must manually promote another instance using SET INSTANCE ... TO MAIN.

Tip: Combine DEMOTE INSTANCE + SET INSTANCE ... TO MAIN for manual failover, useful during maintenance.

Example

DEMOTE INSTANCE instance1;

Coordinator leadership management

YIELD LEADERSHIP

Makes the current leader coordinator give up its Raft leadership so that another coordinator gets elected as the new leader.

YIELD LEADERSHIP;

Parameters

The query takes no parameters and returns no rows. On success, it returns an informational notification that the request was submitted.

Behavior

• Can be run on any coordinator. If the coordinator is a follower, the query is forwarded to the current leader, which yields its leadership. You no longer have to find the leader first.

• Running it on a data instance fails with:

  Only coordinator can run YIELD LEADERSHIP query.

• A coordinator that Raft already elected as leader but that has not yet finished taking over the cluster still yields its leadership. This makes the query usable as an escape hatch exactly when it is needed most — when a freshly elected leader is stuck and you want another coordinator to take over.

• The request is handed over to Raft and processed asynchronously. The query returns as soon as the request is submitted, not when the new leader is elected.

• Raft transfers leadership to another coordinator whose log is up to date. The old leader becomes a follower.

• When the new coordinator becomes the leader, it runs the reconciliation loop: it restores its view of the cluster from the Raft log and restarts health checks toward all data instances. If no MAIN is found at that point, it performs a failover.

Failure modes

Implications

• This changes only the coordinator leadership. Data instances keep their MAIN and REPLICA roles — this is not a data failover, and client queries against MAIN and REPLICAs are unaffected.
• During the short election window, cluster management queries (e.g. SHOW INSTANCES, registration queries) may temporarily fail because there is no leader to serve them. Read queries such as SHOW INSTANCES return no rows and a warning notification rather than a stale picture of the cluster. Retry once the new leader is elected.
• At least one other healthy coordinator must be able to take over. In a single-coordinator cluster, or when the other coordinators are down, the same coordinator remains (or becomes again) the leader.
• Failover of data instances is not triggered, but the new leader recomputes cluster state, so a cluster that was already missing a MAIN can fail over as part of the leadership change.
• Requires COORDINATOR_WRITE. A basic-auth session carries it implicitly; an SSO session needs a role that has been granted it.

Typical use cases

• Removing the current leader coordinator from the cluster, since the leader cannot remove itself — see REMOVE COORDINATOR.
• Taking the leader coordinator down for maintenance, a restart, or a rolling upgrade without waiting for its leadership to expire.
• Moving leadership away from a node that is under heavy load or in a degraded network zone.

Example

YIELD LEADERSHIP;

Verify the outcome by monitoring the cluster until a new coordinator reports the leader role:

SHOW INSTANCES;

Monitoring Commands

SHOW INSTANCES

Displays the state of all servers in the cluster.

SHOW INSTANCES;

Output includes

1. Network endpoints (bolt, coordinator, management)
2. Health state (up or down)
3. Role: MAIN, REPLICA, LEADER or FOLLOWER
4. Time since last health ping

A data instance that is currently down keeps the role recorded in the Raft log (main or replica) instead of being reported with an unknown role, so you can still tell which instance the cluster considers MAIN while it is unreachable.

Behavior

The query is strongly consistent: the result always comes from the leader coordinator, which is the only coordinator with an up-to-date view of the cluster.

1. If you are connected to the leader, it answers directly.

2. If you are connected to a follower, the follower forwards the request to the leader and returns the leader’s result.

3. If the leader cannot be reached, the query returns an empty result set together with a LeaderNotReachable warning notification:

   Couldn’t reach the leader coordinator, so the state of the cluster is unknown. Please retry the query.

   This happens when no leader is currently elected, when the connection to the leader is broken, or when the coordinator you are connected to was just elected leader but has not finished taking over the cluster yet.

SHOW INSTANCE

Displays information about the coordinator you’re connected to.

SHOW INSTANCE;

Output includes

1. Instance name
2. External Bolt server
3. Coordinator (Raft) endpoint
4. Management server endpoint
5. Cluster role (LEADER/FOLLOWER)

If ADD COORDINATOR has not been run, bolt_server will be empty.

SHOW REPLICATION LAG

Shows replication lag (in committed transactions) for all instances.

SHOW REPLICATION LAG;

Behavior

The lag data is collected by the leader coordinator from the current MAIN, so the query is answered by the leader — a follower forwards the request and returns the leader’s result.

Whenever the lag cannot be determined, the query returns no rows and a warning notification explaining why, so you know whether retrying will help:

Implications

• Lag values survive restarts (stored in snapshots + WAL).
• Useful during manual failover to evaluate risk of data loss.

SHOW ROUTING TABLE

Shows the routing table that a coordinator hands out to bolt+routing clients for the default database.

SHOW ROUTING TABLE;

Output includes

Each row contains a role and the list of Bolt servers serving that role:

Example output on a cluster with three coordinators, one MAIN and two REPLICAs:

+---------+------------------------------------------------------------------+
| role    | servers                                                          |
+---------+------------------------------------------------------------------+
| "WRITE" | ["localhost:7687"]                                               |
| "READ"  | ["localhost:7688", "localhost:7689"]                             |
| "ROUTE" | ["localhost:7690", "localhost:7691", "localhost:7692"]           |
+---------+------------------------------------------------------------------+

Behavior

• The query can only be run on a coordinator. Running it on a data instance fails with Only coordinator can run SHOW ROUTING TABLE query.
• The query is always answered from the leader’s state, so every coordinator returns the same routing table. If the leader cannot be contacted, an empty routing table is returned.
• Roles with no servers are omitted. For example, if no data instance is registered yet, only the ROUTE row is returned.
• The routing table is reported for the default database, which is the same database bolt+routing clients are routed to.

Implications

• Useful for verifying which instance clients will send writes to, and which instances they can read from, without inspecting driver internals.
• Because drivers cache the routing table for up to 5 minutes, the output of this query can differ from what a connected client is currently using. See routing table TTL and refresh behavior.

Coordinator runtime settings

Coordinator runtime settings are Raft-replicated and can be changed on a live cluster without downtime. Use SET COORDINATOR SETTING to modify a value and SHOW COORDINATOR SETTINGS to inspect all current values. Changes propagate automatically to every coordinator in the cluster.

Both queries can be run on any coordinator and are served by the leader — a follower forwards the request and returns the leader’s answer. If the leader cannot be reached, SHOW COORDINATOR SETTINGS returns no rows together with a LeaderNotReachable warning notification:

Couldn’t reach the leader coordinator, so the coordinator settings are unknown. Please retry the query.

instance_health_check_frequency_sec

How often the coordinator pings data instances, in seconds.

SET COORDINATOR SETTING 'instance_health_check_frequency_sec' TO '1' ;

Default: 1

instance_down_timeout_sec

How long to wait (in seconds) before marking an instance as down. Must be greater than or equal to instance_health_check_frequency_sec.

SET COORDINATOR SETTING 'instance_down_timeout_sec' TO '5' ;

Default: 5

enabled_reads_on_main

Allows or disallows reading from the MAIN instance.

SET COORDINATOR SETTING 'enabled_reads_on_main' TO 'true' ;

Default: false

sync_failover_only

Users can also choose whether failover to the ASYNC REPLICA is allowed by using the following query:

SET COORDINATOR SETTING 'sync_failover_only' TO 'false' ;

Default: true (only SYNC replicas are eligible). When the value is set to false, the ASYNC REPLICA is also considered, but there is an additional risk of experiencing data loss.

Setting to false allows failover to ASYNC replicas but may risk data loss.

In extreme cases, failover to an ASYNC REPLICA may be necessary when other SYNC REPLICAs are down and you want to manually perform a failover.

max_failover_replica_lag

Users can control the maximum transaction lag allowed during failover through configuration. If a REPLICA is behind the MAIN instance by more than the configured threshold, that REPLICA becomes ineligible for failover. This prevents data loss beyond the user’s acceptable limits.

To implement this functionality, we employ a caching mechanism on the cluster leader coordinator that tracks replicas’ lag. The cache gets updated with each StateCheckRpc response from REPLICAs. During the brief failover window on the cooordinators’ side, the new cluster leader may not have the current lag information for all data instances and in that case, any REPLICA can become MAIN. This trade-off is intentional and it avoids flooding Raft logs with frequently-changing lag data while maintaining failover safety guarantees in the large majority of situations.

The configuration value can be controlled using the query:

SET COORDINATOR SETTING 'max_failover_replica_lag' TO '10' ;

max_replica_read_lag

Users can control the maximum allowed REPLICA lag to maintain read consistency. When a REPLICA falls behind the current MAIN by more than max_replica_read_lag transactions, the bolt+routing protocol will exclude that REPLICA from read query routing to ensure data freshness.

The configuration value can be controlled using the query:

SET COORDINATOR SETTING 'max_replica_read_lag' TO '10' ;

deltas_batch_progress_size

Previously, this setting controlled how many deltas a REPLICA processed before reporting back to the MAIN that it is still working on the data (transactions, WALs, snapshots) the MAIN sent it. Counting deltas could not cover work that happens inside a single delta — populating an index, validating a constraint, clearing storage before a snapshot load, or aborting an interrupted two-phase commit. On a large dataset any one of those can run for minutes while the REPLICA reports nothing, so the MAIN would hit its RPC timeout and drop the connection mid-build, and the REPLICA would never converge.

As of 3.13, a REPLICA emits an in-progress message on a fixed interval for as long as the operation keeps making progress, regardless of how that work is divided into deltas. An operation that stops progressing stops reporting, so a genuinely stuck REPLICA is still caught by the MAIN’s timeout rather than masked by an unconditional keepalive.

SET COORDINATOR SETTING 'deltas_batch_progress_size' TO '50000';

The query above still succeeds and the value is still returned by SHOW COORDINATOR SETTINGS, but it is ignored.

global_read_only

Puts the entire cluster into a read-only state. When enabled, the current MAIN stops accepting write queries while it continues to serve reads and replicate existing data to REPLICAs. The main use case is performing no-downtime upgrades: disable writes across the cluster while you upgrade, without shutting the cluster down. It is also useful for freezing the dataset during a maintenance window, taking a consistent backup without racing new writes, or investigating an issue.

SET COORDINATOR SETTING 'global_read_only' TO 'true' ;

To return the cluster to normal read/write operation:

SET COORDINATOR SETTING 'global_read_only' TO 'false' ;

Default: false

The value is persisted in the coordinator’s Raft-replicated cluster state, so it survives coordinator restarts and leader re-elections, and is honored across failovers: a newly promoted MAIN comes up read-only when the cluster is in read-only mode, instead of silently accepting writes.

SET COORDINATOR SETTING requires COORDINATOR_WRITE and SHOW COORDINATOR SETTINGS requires COORDINATOR_READ. A basic-auth session carries both implicitly; an SSO session needs a role that has been granted them.

Coordinator role and privilege management

Coordinator roles are stored in the Raft-replicated cluster state, not in the auth store, so they survive restarts, follower catch-up and leader failover. Like the cluster registration queries, all of these can be run on any coordinator: writes are transparently forwarded to the leader and committed through the Raft log, and SHOW ROLES / SHOW PRIVILEGES FOR ROLE are strong reads served by the leader.

CREATE ROLE

Creates a coordinator role. New roles start with no privileges.

CREATE ROLE ifNotExists? roleName;

Behavior & implications

• Errors if the role already exists, unless IF NOT EXISTS is given.
• The role name must match the --auth-user-or-role-name-regex pattern, otherwise the query fails with Invalid role name '<name>'.
• Requires COORDINATOR_WRITE.

Example

CREATE ROLE dba;
CREATE ROLE IF NOT EXISTS analyst;

DROP ROLE

Removes a coordinator role.

DROP ROLE roleName;

Behavior & implications

• Errors with Role '<name>' doesn't exist. if the role is not present.
• Takes effect on already-connected sessions immediately — privileges are re-derived from the committed role set on every query, so a session that authenticated with the dropped role is denied its next privileged query without needing to reconnect.
• Requires COORDINATOR_WRITE.

Example

DROP ROLE analyst;

SHOW ROLES

Lists the coordinator roles. Returns one role column, name only.

SHOW ROLES;

Behavior & implications

• Strong read served by the leader. If no leader can be reached, the query fails rather than returning possibly-stale local state.
• Requires COORDINATOR_READ.

GRANT / REVOKE coordinator privileges

Grants or revokes a coordinator privilege on a role. Coordinators support exactly two privileges: COORDINATOR_READ and COORDINATOR_WRITE, where COORDINATOR_WRITE is a superset of COORDINATOR_READ.

GRANT ( ALL PRIVILEGES | COORDINATOR_READ | COORDINATOR_WRITE [, ...] ) TO ROLE? roleName;
REVOKE ( ALL PRIVILEGES | COORDINATOR_READ | COORDINATOR_WRITE [, ...] ) FROM ROLE? roleName;

Behavior & implications

• GRANT ALL PRIVILEGES grants both coordinator privileges; REVOKE ALL PRIVILEGES removes both.
• Errors with Role '<name>' doesn't exist. if the role is not present.
• Only COORDINATOR_READ and COORDINATOR_WRITE may appear in the privilege list. Any other privilege, DENY in any form, a USER target, fine-grained access control (ON NODES / ON EDGES), property permissions and GRANT DATABASE are all rejected on a coordinator.
• Like DROP ROLE, a REVOKE applies to already-connected sessions on their next query.
• Requires COORDINATOR_WRITE.

Example

GRANT ALL PRIVILEGES TO dba;
GRANT COORDINATOR_READ TO analyst;
REVOKE COORDINATOR_WRITE FROM analyst;

SHOW PRIVILEGES FOR ROLE

Reports the privileges granted to a coordinator role, one per row.

SHOW PRIVILEGES FOR ROLE? roleName;

Behavior & implications

• A role with no grants returns no rows.
• The trailing ON MAIN | CURRENT | DATABASE <db> clause is rejected — coordinators have no databases.
• SHOW PRIVILEGES FOR USER <user> is rejected — coordinators have no users.
• Strong read served by the leader.
• Requires COORDINATOR_READ.

Example

SHOW PRIVILEGES FOR ROLE dba;

+---------------------+
| privilege           |
+---------------------+
| COORDINATOR_READ    |
| COORDINATOR_WRITE   |
+---------------------+

SHOW CURRENT USER and SHOW CURRENT ROLE

Report the identity of the current session.

SHOW CURRENT USER;
SHOW CURRENT ROLE;

Behavior & implications

• No privilege and no license are required — these are self-service queries that reveal only the session’s own identity.
• SHOW CURRENT USER returns the principal the SSO module reported. It is session-local and works even when the leader is unreachable. A basic-auth passthrough session returns null.
• SHOW CURRENT ROLE returns the session’s roles filtered against the leader’s committed role set, so a dropped role stops being reported. A basic-auth passthrough session has no roles and returns null.

Error handling

If a Raft log commit fails for any cluster operation (register, unregister, promote, demote, add coordinator, role or privilege change), the error message will indicate:

Writing to Raft log failed. Please retry the operation.

When there is no leader to serve the query

Because every cluster query is served by the leader, queries fail (or return nothing) while the cluster has no usable leader — most commonly during a leader election, or right after one, while the new leader is still taking over the cluster.

State-changing queries (ADD COORDINATOR, REMOVE COORDINATOR, UPDATE CONFIG, REGISTER INSTANCE, UNREGISTER INSTANCE, SET INSTANCE ... TO MAIN, DEMOTE INSTANCE, FORCE RESET CLUSTER STATE) fail with an explicit error:

Couldn’t <operation> since coordinator is not a leader! Try contacting other coordinators as there might be leader election happening or other coordinators are down.

When the leader is known but the request still could not be executed there, the message instead names the current leader’s id and Bolt address so you can connect to it directly. If the request could not be forwarded at all, the error is:

Tried to forward the request to the current leader but the leader couldn’t be found!

Read queries (SHOW INSTANCES, SHOW COORDINATOR SETTINGS, SHOW REPLICATION LAG) do not fail — they return an empty result set with a warning notification (LeaderNotReachable, or ReplicationLagUnavailable for SHOW REPLICATION LAG). In both cases the operation is safe to retry once a leader is available.

Troubleshooting commands

FORCE RESET CLUSTER STATE

Resets cluster state when the cluster cannot reach a healthy configuration.

FORCE RESET CLUSTER STATE;

Behavior

1. All alive instances are demoted to REPLICA.
2. A new MAIN is selected from alive instances.
3. Down instances are demoted after they come back online.
4. Writes changes to Raft.

Implications