Query the cluster in high availability ^Enterprise

Why the Bolt protocol is not enough

For a standalone Memgraph instance, the simplest way to connect is via the bolt:// protocol. However, in a high-availability cluster, this becomes insufficient for several reasons:

1. Bolt only connects to a single instance. In a cluster, you would need to maintain separate connections to every instance you want to query. A typical setup already includes five instances (3 coordinators + 2 data instances).

2. Clients do not know which instance is MAIN. Automatic failovers can happen at any time. Sending a write query to a REPLICA will fail, as replicas cannot accept writes.

To solve this, Memgraph supports a routing mechanism built on top of Bolt.

The Bolt+routing protocol (neo4j://) ensures that:

• Writes always go to the current MAIN instance.
• Reads are routed to either the MAIN or any REPLICA (depending on configuration).
• Clients automatically switch to the new MAIN after a failover, avoiding split-brain scenarios.

How the routing protocol works

The routing protocol works as follows:

1. The client sends a ROUTE Bolt message to any coordinator.

2. The coordinator returns a routing table containing:

   • Readers: REPLICAs (and optionally MAIN).
   • Writers: only the current MAIN.
   • Routers: all coordinator endpoints.

3. Using this routing table, the client forwards queries to the correct instance.

Leader and follower behavior

• If the client happens to contact the leader coordinator, it immediately receives the latest routing table.
• If it contacts a follower, the follower forwards the request to the leader and returns the leader’s result.
• If the leader cannot be reached at all, an empty routing table is returned and the driver retries against another coordinator.

Because leader state is synchronized via Raft, routing information is always accurate.

This ensures:

• Consistency: All clients receive the same routing table.
• Reliability: The leader always holds the latest cluster state.
• Transparency: Clients work seamlessly whether they connect to leaders or followers.

Inspecting the routing table

The routing table a coordinator hands out can be inspected manually with the SHOW ROUTING TABLE query:

SHOW ROUTING TABLE;

It returns one row per role (WRITE, READ, ROUTE) with the Bolt endpoints serving that role, for the default database. The query can only be run on a coordinator, and it is always answered from the leader’s state, so every coordinator returns the same result. If the leader cannot be contacted, an empty routing table is returned.

Routing table TTL and refresh behavior

Because routing is entirely client-side, the driver caches the routing table and decides when to refresh it. There are two triggers:

1. TTL expiration. Memgraph returns a TTL alongside the routing table, and it is hard-coded to 5 minutes. Once the TTL expires, the driver fetches a new routing table from a coordinator on the next request.

2. Driver-detected failure. If the driver notices that the cached routing table is wrong, it refreshes it before retrying:

   • execute_read refreshes the routing table when all read instances from the previous routing table become unavailable.
   • execute_write refreshes the routing table when the MAIN goes down.
   • session.run does not trigger a refresh on failure - it is up to the application to handle the error and retry.

Routing examples

WRITE query using Bolt+routing - routed to MAIN:

READ query using Bolt+routing - routed to a REPLICA:

Bolt+routing is entirely client-side, meaning:

• Drivers perform endpoint resolution internally.
• Routing decisions do not happen inside Memgraph itself.

For the details of the Bolt messages involved, see the official Bolt routing message documentation.

How to connect

Users only need to change the scheme they use for connecting to coordinators. This means instead of using bolt://<main_ip_address>, you should use neo4j://<coordinator_ip_adresss> to get an active connection to the current main instance in the cluster.

Examples in multiple programming languages can be found here.

Authentication

User accounts exist only on data instances. Coordinators do not store user accounts and therefore:

• Ignore the MEMGRAPH_USER and MEMGRAPH_PASSWORD environment variables.
• Do not accept authentication queries such as CREATE USER.

When using the bolt+routing protocol, provide credentials for users that exist on the data instances. The authentication flow works as follows:

1. The client authenticates and connects to a coordinator.
2. The coordinator returns a routing table.
3. The client connects to the appropriate data instance using the same credentials.
4. The data instance performs authentication and executes the query.

Authenticating against coordinators

From Memgraph 3.13, coordinators are no longer unconditionally open. They can enforce single sign-on and coordinator privileges, which changes the flow above in two ways:

• Step 1 becomes a real authentication. Once SSO is configured on the coordinators, basic auth is refused and the client must present a valid IdP token for a scheme listed in --auth-module-mappings.
• Step 2 requires a privilege. Serving the routing table requires COORDINATOR_READ. A session whose roles carry no coordinator privilege is denied the routing table with You don't have permission to read the routing table on the coordinator! — reported as a non-retryable client error so drivers do not retry it.