High availability reference queries

This guide provides a complete reference for all commands used to manage a Memgraph High Availability (HA) cluster.

Cluster registration commands

ADD COORDINATOR

Adds a coordinator to the cluster.

ADD COORDINATOR coordinatorId WITH CONFIG {
  "bolt_server": boltServer, 
  "coordinator_server": coordinatorServer, 
  "management_server": managementServer
}; 

Parameters

• coordinatorId (int) Unique ID for each coordinator. Typically incremented sequentially.
• boltServer (string) External Bolt endpoint: "IP_OR_DNS:PORT". Usually port 7687. Must be reachable by external applications.
• coordinatorServer (string) Internal raft endpoint: "HOSTNAME_OR_DNS:COORDINATOR_PORT".
• managementServer (string) Coordinator management endpoint: "HOSTNAME_OR_DNS:MANAGEMENT_PORT".

DNS/FQDN is recommended when IPs are ephemeral (e.g., Kubernetes).

Behavior & implications

• Can be run before or after registering data instances.
• Must be executed once for each coordinator.
• External Bolt server must be reachable; Raft and management endpoints may be internal.
• Writes cluster configuration to Raft log.

Example

ADD COORDINATOR 1 WITH CONFIG {
  "bolt_server": "my_outside_coordinator_1_IP:7687",
  "coordinator_server": "memgraph-coordinator-1.default.svc.cluster.local:12000",
  "management_server": "memgraph-coordinator-1.default.svc.cluster.local:10000"
};

REMOVE COORDINATOR

Removes a follower coordinator from the cluster.

REMOVE COORDINATOR coordinatorId;

Parameters

• coordinatorId (int) — previously registered coordinator ID.

Behavior & implications

• Leader coordinator cannot remove itself. To remove the leader, first trigger a leadership change.

Example

REMOVE COORDINATOR 2;

UPDATE CONFIG

Updates the configuration of the data instance/coordinator in the cluster. Use this command to update the DNS of your bolt servers, for example, when migrating to new load balancers.

UPDATE CONFIG FOR ( INSTANCE instanceName | COORDINATOR coordinatorServerId ) configsMap=configMap ;

Parameters

• instanceName (string) — the name of the data instance for which the configuration changes will be applied.
• coordinatorServerId (int) — previously registered coordinator ID for which the configuration changes will be applied.
• configsMaps (int) — Map of KV pairs that signal which field should be updated. Currently, it is only supported to update bolt server of each instance.

Behavior & implications

• Only bolt server can be updated.

Example

UPDATE CONFIG FOR COORDINATOR 1 {'bolt_server': '127.0.0.1:7690'};
UPDATE CONFIG FOR INSTANCE instance_2 {'bolt_server': '127.0.0.1:7688'}

REGISTER INSTANCE

Registers a data instance in the cluster.

REGISTER INSTANCE instanceName ( AS ASYNC | AS STRICT_SYNC ) ? WITH CONFIG {
  "bolt_server": boltServer, 
  "management_server": managementServer, 
  "replication_server": replicationServer
};

Parameters

• instanceName Unique symbolic name of the data instance.
• AS ASYNC / AS STRICT_SYNC (optional) Selects replication mode. Default: SYNC.
• boltServer External Bolt endpoint: "IP_OR_DNS:PORT".
• managementServer Coordinator → data instance health check endpoint.
• replicationServer Replication endpoint, typically using port 20000.

Behavior

• The operation is first committed to the Raft log and acknowledged by a majority of coordinators.
• Coordinator connects via management_server to verify liveness.
• Coordinator begins periodic health checks.
• Instance is automatically demoted to REPLICA.
• Replication server is started on the data instance.
• If RPCs to the data instance fail (e.g., due to a transient network issue), the registration still succeeds. The reconciliation loop automatically retries the RPCs.

Replication mode rules

• Allowed combinations:

  • STRICT_SYNC + ASYNC
  • SYNC + ASYNC

• Forbidden combination:

  • STRICT_SYNC + SYNC

Notes

• In Kubernetes, use service DNS names (e.g. memgraph-data-1.default.svc.cluster.local).
• Local development uses localhost.

Example

REGISTER INSTANCE instance1 WITH CONFIG {
  "bolt_server": "my_outside_instance1_IP:7687",
  "management_server": "memgraph-data-1.default.svc.cluster.local:10000",
  "replication_server": "memgraph-data-1.default.svc.cluster.local:20000"
};

UNREGISTER INSTANCE

Removes a data instance from the cluster.

UNREGISTER INSTANCE instanceName;

Parameters

• instanceName — name of the data instance.

Implications

• Do not unregister the MAIN instance; this may corrupt cluster state.
• A healthy MAIN must exist during the operation.
• The instance is removed from the Raft state first. If the RPC to unregister the replica from MAIN fails, the reconciliation loop automatically retries the operation.

Example

UNREGISTER INSTANCE instance_1;

Replication role management

SET INSTANCE ... TO MAIN

Promotes a replica to MAIN.

SET INSTANCE instanceName TO MAIN;

Behavior

• The promotion is first committed to the Raft log and acknowledged by a majority of coordinators.
• All other registered instances become replicas of the new MAIN.
• RPCs (PromoteToMainRpc, SwapAndUpdateUUID) are sent to data instances on a best-effort basis. If they fail, the reconciliation loop automatically retries them.

Implications

• Fails if a MAIN already exists.

Example

SET INSTANCE instance_0 TO MAIN;

DEMOTE INSTANCE

Demotes the current MAIN to a REPLICA.

DEMOTE INSTANCE instanceName;

Behavior

• The role change is first committed to the Raft log and acknowledged by a majority of coordinators.
• MAIN becomes REPLICA.
• The DemoteMainToReplicaRpc is sent on a best-effort basis. If it fails, the reconciliation loop automatically retries it.
• Returns an error if the instance is already a REPLICA.

Implications

• Failover is not automatic after demotion. You must manually promote another instance using SET INSTANCE ... TO MAIN.

Tip: Combine DEMOTE INSTANCE + SET INSTANCE ... TO MAIN for manual failover, useful during maintenance.

Example

DEMOTE INSTANCE instance1;

Monitoring Commands

SHOW INSTANCES

Displays the state of all servers in the cluster.

SHOW INSTANCES;

Output includes

1. Network endpoints (bolt, coordinator, management)
2. Health state
3. Role: MAIN, REPLICA, LEADER, FOLLOWER, or UNKNOWN
4. Time since last health ping

Behavior on followers

1. Follower attempts to query the leader for accurate state.
2. If leader unavailable, follower reports all servers as "down".

SHOW INSTANCE

Displays information about the coordinator you’re connected to.

SHOW INSTANCE;

Output includes

1. Instance name
2. External Bolt server
3. Coordinator (Raft) endpoint
4. Management server endpoint
5. Cluster role (LEADER/FOLLOWER)

If ADD COORDINATOR has not been run, bolt_server will be empty.

SHOW REPLICATION LAG

Shows replication lag (in committed transactions) for all instances.

SHOW REPLICATION LAG;

Implications

• Lag values survive restarts (stored in snapshots + WAL).
• Useful during manual failover to evaluate risk of data loss.

Coordinator runtime settings

Coordinator runtime settings are Raft-replicated and can be changed on a live cluster without downtime. Use SET COORDINATOR SETTING to modify a value and SHOW COORDINATOR SETTINGS to inspect all current values. Changes propagate automatically to every coordinator in the cluster.

instance_health_check_frequency_sec

How often the coordinator pings data instances, in seconds.

SET COORDINATOR SETTING 'instance_health_check_frequency_sec' TO '1' ;

Default: 1

instance_down_timeout_sec

How long to wait (in seconds) before marking an instance as down. Must be greater than or equal to instance_health_check_frequency_sec.

SET COORDINATOR SETTING 'instance_down_timeout_sec' TO '5' ;

Default: 5

enabled_reads_on_main

Allows or disallows reading from the MAIN instance.

SET COORDINATOR SETTING 'enabled_reads_on_main' TO 'true' ;

Default: false

sync_failover_only

Users can also choose whether failover to the ASYNC REPLICA is allowed by using the following query:

SET COORDINATOR SETTING 'sync_failover_only' TO 'false' ;

Default: true (only SYNC replicas are eligible). When the value is set to false, the ASYNC REPLICA is also considered, but there is an additional risk of experiencing data loss.

Setting to false allows failover to ASYNC replicas but may risk data loss.

In extreme cases, failover to an ASYNC REPLICA may be necessary when other SYNC REPLICAs are down and you want to manually perform a failover.

max_failover_replica_lag

Users can control the maximum transaction lag allowed during failover through configuration. If a REPLICA is behind the MAIN instance by more than the configured threshold, that REPLICA becomes ineligible for failover. This prevents data loss beyond the user’s acceptable limits.

To implement this functionality, we employ a caching mechanism on the cluster leader coordinator that tracks replicas’ lag. The cache gets updated with each StateCheckRpc response from REPLICAs. During the brief failover window on the cooordinators’ side, the new cluster leader may not have the current lag information for all data instances and in that case, any REPLICA can become MAIN. This trade-off is intentional and it avoids flooding Raft logs with frequently-changing lag data while maintaining failover safety guarantees in the large majority of situations.

The configuration value can be controlled using the query:

SET COORDINATOR SETTING 'max_failover_replica_lag' TO '10' ;

max_replica_read_lag

Users can control the maximum allowed REPLICA lag to maintain read consistency. When a REPLICA falls behind the current MAIN by more than max_replica_read_lag transactions, the bolt+routing protocol will exclude that REPLICA from read query routing to ensure data freshness.

The configuration value can be controlled using the query:

SET COORDINATOR SETTING 'max_replica_read_lag' TO '10' ;

deltas_batch_progress_size

Users can control how often REPLICAs report back to the MAIN that they’re still processing the data (transactions, WALs, snapshots) the MAIN has sent to them. The default value is 100’000, which should be enough for most of your transactions. However, if processing 100’000 deltas takes more than 30s (because you’re dealing with large deltas or you have older CPUs), you can set the configuration value deltas_batch_progress_size to a smaller value. This will avoid timeouts on REPLICAs so you won’t see the query exception “At least one SYNC replica has not committed”, but at the cost of lower throughput since REPLICAs will be sending in-progress messages to the MAIN more often.

SET COORDINATOR SETTING 'deltas_batch_progress_size' TO '50000';

global_read_only

Puts the entire cluster into a read-only state. When enabled, the current MAIN stops accepting write queries while it continues to serve reads and replicate existing data to REPLICAs. The main use case is performing no-downtime upgrades: disable writes across the cluster while you upgrade, without shutting the cluster down. It is also useful for freezing the dataset during a maintenance window, taking a consistent backup without racing new writes, or investigating an issue.

SET COORDINATOR SETTING 'global_read_only' TO 'true' ;

To return the cluster to normal read/write operation:

SET COORDINATOR SETTING 'global_read_only' TO 'false' ;

Default: false

The value is persisted in the coordinator’s Raft-replicated cluster state, so it survives coordinator restarts and leader re-elections, and is honored across failovers: a newly promoted MAIN comes up read-only when the cluster is in read-only mode, instead of silently accepting writes.

No privilege is required to run this query on the coordinators.

Error handling

If a Raft log commit fails for any cluster operation (register, unregister, promote, demote, add coordinator), the error message will indicate:

Writing to Raft log failed. Please retry the operation.

Troubleshooting commands

FORCE RESET CLUSTER STATE

Resets cluster state when the cluster cannot reach a healthy configuration.

FORCE RESET CLUSTER STATE;

Behavior

1. All alive instances are demoted to REPLICA.
2. A new MAIN is selected from alive instances.
3. Down instances are demoted after they come back online.
4. Writes changes to Raft.

Implications