System configuration

Enhancing Memgraph’s performance can sometimes involve optimizing operating system-level configurations. These adjustments primarily relate to the settings governing resource allocation and process management.

The following configuration steps are written based on the Ubuntu 22.04 operating system. The steps may vary slightly depending on the Linux distribution you are using.

Handle numerous open files simultaneously

Every client connection, every database, every text index and every replica connection consumes file descriptors. When a process runs out of them, it fails with one of two errors:

• Too many open files (EMFILE): the per-process limit (RLIMIT_NOFILE) has been reached. This is the one you will normally hit.
• Too many open files in system (ENFILE): the system-wide limit (fs.file-max) has been reached.

When Memgraph reaches the per-process limit, new client connections fail at the Bolt handshake, and the Memgraph log shows:

Bolt accept failed (Too many open files); retrying every 500ms, further occurrences logged at most every 10s

Connections succeed again once other connections close.

Starting with v3.14, Memgraph raises its soft limit to the hard limit at startup. It never changes the hard limit, so only the hard limit matters. With a hard limit of 1024, Memgraph starts failing at roughly 950 client connections. The Memgraph package sets LimitNOFILE=1048576 in its systemd unit, so both limits are 1048576.

If the limit after the raise is below 65536, Memgraph logs a warning at startup. The warning begins with Open files limit is too low (soft N, hard N). Both numbers are the hard limit, so raise the hard limit. Memgraph still starts and serves queries.

A custom unit file or container setup with a high hard limit needs no action. A systemd unit without LimitNOFILE gets a soft limit of 1024 and a hard limit that depends on the systemd version. If the hard limit is low, or on releases before v3.14, follow the steps below.

Memgraph also logs the resulting limits as Open files limit (RLIMIT_NOFILE): soft N, hard N. at INFO level. The default log level is WARNING, so this line isn’t written unless you start Memgraph with --log-level=INFO. To check the limits Memgraph runs with, read /proc/<pid>/limits as shown in the steps below.

Sizing the limit

Memgraph needs roughly one descriptor per client connection plus files per database (WAL, snapshots, indexes); 65536 or more is recommended.

Linux with systemd

/etc/security/limits.conf does not apply to systemd services. Set the limit in the service unit instead:

[Service]
LimitNOFILE=1048576

grep 'open files' /proc/$(pidof memgraph)/limits

Docker

Pass the limit to the container with --ulimit:

docker run --ulimit nofile=1048576:1048576 -p 7687:7687 memgraph/memgraph

With Docker Compose:

services:
  memgraph:
    image: memgraph/memgraph
    ulimits:
      nofile:
        soft: 1048576
        hard: 1048576

Kubernetes

Starting with v3.14, Memgraph raises its soft limit to the container’s hard limit at startup, so only the hard limit counts. If you see the warning in the Memgraph log, raise the hard open files limit that containers get on the node.

System-wide limit

Modern kernels and systemd already set fs.file-max very high, so there is usually nothing to do. Do not lower it. Only if cat /proc/sys/fs/file-max returns a low value, raise it by adding fs.file-max=<value> to /etc/sysctl.conf and running sudo sysctl -p.

Increasing memory map areas

If users are working on a larger scale, there is a possibility that a transaction is hanging because Memgraph tries to allocate memory and it couldn’t as it was out of virtual memory space. User can adjust the vm.max_map_count parameter to define the maximum number of memory map areas a process can have.

Recommended values for the vm.max_map_count parameter

For optimal system performance, the vm.max_map_count value should be chosen in accordance with your system’s RAM, aiming for approximately one memory map area per 128 KB of system memory.

Immediate adjustment

For instance, if your recommended vm.max_map_count value is 524288, run sudo sysctl -w vm.max_map_count=524288 to immediately increase the memory map area limit to 524,288. This change takes effect right away but lasts only until the next reboot.

Persistent adjustment

To ensure that changes to vm.max_map_count persist after a reboot, follow these steps:

Addressing stack overflow issues

Stack overflow issues may occur due to a large query in Memgraph. To address these issues, there are two possible approaches:

1. Increase the stack size
2. Split the query into smaller chunks

Increase the stack size

With Docker

docker run --rm -it --name memgraph --ulimit stack=33554432:33554432 -p 7687:7687 -p 7444:7444 memgraph/memgraph

docker exec memgraph sh -c "ulimit -a | grep stack"

Without Docker

memgraph soft stack 32768
memgraph hard stack 32768

ulimit -s

Split the query into smaller chunks

To avoid stack overflow, divide your large query into smaller parts. This approach requires careful handling of variables and node references across queries. For creating edges between nodes created in separate queries, use the MATCH clause to reference nodes from previous queries:

// First Query
CREATE (a:Node {id: 1});
// Second Query
MATCH (a:Node {id: 1})
CREATE (b:Node {id: 2}), (a)-[:RELATES_TO]->(b);

This method reduces the stack size required for each query, potentially avoiding the stack overflow issue.

IPv6 networking configuration

If you are running Memgraph on a system with IPv6 enabled and want to connect via an IPv6 address, you may encounter issues where Memgraph is not reachable over IPv6.

Here are the possible error messages you might encounter:

Connection failure: couldn't connect to host: Connection refused

Error: Couldn't connect to [::]:7687 (resolved to ('[::]:7687',)):
Failed to read any data from server ResolvedIPv6Address(('::', 7687, 0, 0))

To fix this, disable IPv6 or configure Memgraph to use it.

Configure Memgraph to use IPv6

To configure Memgraph to use IPv6, explicitly instruct the Bolt server to listen on an IPv6 address. To do that, set the --bolt-address configuration to an IPv6 address when starting Memgraph. To start Memgraph with the Bolt server listening on all IPv6 interfaces (::), set --bolt-address=::.

The following command runs Memgraph in Docker with the --bolt-address set to :::

docker run -p 7687:7687 memgraph/memgraph:latest --bolt-address=::

When connecting to the running Memgraph instance with mgconsole or a certain driver, then the host parameter or URI must be set to :: (corresponding to IPv4 0.0.0.0) or ::1 (corresponding to IPv4 127.0.0.1). Network’s IPv6 can be used to connect from a client if everything is running in a explicit IPv6 Docker network.

If you are using monitoring server or metrics, they should be configured to connect to the IPv6 address as well.