Debugging Memgraph

Why enable user debugging?

User-driven debugging helps improve Memgraph’s performance and reliability by providing diagnostic data from your environment. This data assists us in reproducing and resolving issues faster, especially for bugs that are hard to replicate.

To help with this, our containers come equipped with user-friendly debugging tools, empowering you to identify and report problems more effectively.

Chose the right debug image

Memgraph provides Docker images built in RelWithDebInfo mode, including tools like perf, gdb and pgrep. This image is about 10% slower but enables detailed debugging.

To pull a debug image:

docker image pull <memgraph_version>-relwithdebinfo

For Memgraph MAGE:

docker image pull <MAGE_version>-memgraph-<memgraph_version>-relwithdebinfo

All the images in the RelWithDebInfo build mode have the suffix -relwithdebinfo.

Run Memgraph in debug mode

Run the Memgraph container in privileged mode to allow debugging tools like gdb and perf to function:

Below, we can see an example command of how to run a Memgraph container in the privileged mode:

docker container run --name mg --privileged -p 7687:7687 -p 9091:9091 memgraph --log-level=TRACE --also-log-to-stderr

Accessing the container

All debugging is performed inside the container. To enter the container, you need to execute the following command.

docker container exec -it -u root memgraph bash

The -u root command is there to enable root privileges inside the container, necessary for running debugging tools.

Debugging tools overview

Memgraph supports the following debug capabilities:

1. Using GDB: Attaching Memgraph with GDB and inspecting threads
2. Generating a core dump after Memgraph crashed
3. Running Memgraph in GDB inside Docker: Directly running Memgraph or MAGE in GDB inside a Docker container.
4. Using perf to identify performance bottlenecks

Using GDB

GDB and pgrep are already installed packages in the Memgraph container that has the debug symbols. Since Memgraph is already running there at port 7687, you can attach GDB to your running Memgraph with the following command:

gdb -p $(pgrep memgraph)

Most likely, the Memgraph process will have the PID number 1, but for certainty, we use pgrep.

Useful GDB commands:

In Memgraph, we usually want to see first what are all the threads currently in place. We do that by issuing:

info thread

By identifying a certain thread with a code that could belong to the Memgraph repository, we can issue the command

t <x>

where x is the specific thread number.

Seeing the backtrace can be done with the command

bt

Generating core dump via Docker

In order to generate a core dump, you need to do a few steps on the host and in the container image.

ulimit -c unlimited

docker container run --name mg --privileged -v /home/user/cores:/tmp/cores -p 7687:7687 -p 9091:9091 memgraph:2.16.0_17_050d5c985 --log-level=TRACE --also-log-to-stderr

ulimit -c unlimited
mkdir -p /tmp/cores
chmod a+rwx /tmp/cores
echo "/tmp/cores/core.%e.%p.%h.%t" > /proc/sys/kernel/core_pattern

docker container exec -it -u root memgraph bash

gdb /usr/lib/memgraph/memgraph --core=/core.memgraph.file

Generating core dump via Docker Compose

The setup with Docker Compose is similar to Docker. You will need to bind the volume, run Memgraph in privileged mode, and make sure you set no size limit on the generated core dump.

Below we can see an example Docker Compose file which can generate a core dump:

services:
  memgraph:
    image: memgraph:2.16.0_17_050d5c985
    container_name: mg
    privileged: true
    ports:
      - "7687:7687"
      - "7444:7444"
      - "9091:9091"
    volumes:
      - /home/josipmrden/cores:/tmp/cores
    command: ["--log-level=TRACE", "--also-log-to-stderr=true"]
    ulimits:
      core:
        hard: -1
        soft: -1
 
  lab:
    image: memgraph/lab:latest
    container_name: memgraph-lab
    ports:
      - "3000:3000"
    depends_on:
      - memgraph
    environment:
      - QUICK_CONNECT_MG_HOST=memgraph
      - QUICK_CONNECT_MG_PORT=7687

Running Memgraph in GDB inside Docker

To run Memgraph or MAGE in GDB inside a Docker container, you can use the following commands to override the entry point and create a bind mount for core dumps:

# create a directory with sufficient permissions to save the core on the host
mkdir -p cores
chmod 777 cores
 
# run memgraph via the debug script by overriding the entry point, and creating a bind mount for the core dump
docker run --rm \
  --name memgraph \
  -v "$(pwd)/cores:/tmp/cores" \
  --entrypoint /usr/lib/memgraph/run_with_gdb.sh \
  -p 7687:7687 \
  memgraph/memgraph:3.7-relwithdebinfo

Optionally, the core dump path can be overridden by setting the CORE_PATH variable:

docker run --rm \
  --name memgraph \
  -v "$(pwd)/cores:/some/other/dir/cores" \
  -e CORE_PATH=/some/other/dir/cores/core.dump \
  --entrypoint /usr/lib/memgraph/run_with_gdb.sh \
  -p 7687:7687 \
  memgraph/memgraph:3.7-relwithdebinfo

In the case where Memgraph crashes, a core dump will be created in the directory specified by the CORE_PATH variable (tmp/cores/core by default) and the full backtrace will be printed to the terminal.

Using heaptrack with Docker

All RelWithDebInfo images come with heaptrack installed. You can use it to track the memory usage of Memgraph.

Before starting the container, create a directory to store the heaptrack data:

mkdir -p /tmp/heaptrack
chmod a+rwx /tmp/heaptrack

Then start the container with the following command:

docker run -d --rm \
  --name memgraph \
  -v /tmp/heaptrack:/data \
  --entrypoint /usr/bin/heaptrack \
  memgraph/memgraph:3.6.1-relwithdebinfo \
  --output /data/heaptrack.memgraph \
  -- \
  /usr/lib/memgraph/memgraph

docker run -d --rm \
  --name memgraph \
  -p 7687:7687 \
  -v /tmp/heaptrack:/data \
  --entrypoint /usr/bin/heaptrack \
  memgraph/memgraph-mage:3.6.1-relwithdebinfo \
  --output /data/heaptrack.memgraph \
  --use-inject /usr/lib/memgraph/memgraph

To stop memgraph gracefully:

docker exec memgraph bash -c "kill -SIGINT \$(pidof memgraph)"

Then the heaptrack GUI can be used to inspect the heaptrack data on the host machine:

heaptrack /tmp/heaptrack/heaptrack.memgraph.gz

sudo apt install heaptrack

sudo dnf install heaptrack

Profiling with perf

Perfing is the most common operation that is run when Memgraph is hanging or performing slowly.

In the next steps, the instructions are provided on how to check which parts of Memgraph are stalling during query execution, so we can use the information to make the system better.

Prior to performing perf instructions, you will need to bound the Memgraph binary to the local filesystem. You can start Memgraph with the volume binded like this:

docker container run --name mg -p 7687:7687 --privileged -v memgraph-binary:/usr/lib/memgraph <memgraph_image> --log-level=TRACE --also-log-to-stderr

perf record -p $(pgrep memgraph) --call-graph dwarf sleep 5

apt install hotspot

docker cp <container_id>:<path_to_perf.data> .

ln -s <path_to_docker_volume_debug_symbols_binary> /usr/lib/memgraph/memgraph

hotspot perf.data

Debugging Memgraph under Kubernetes (k8s)

General commands

To being with, the master of all kubectl commands is:

kubectl get all

Managing nodes:

kubectl get nodes --show-labels # Show all nodes and their labels.
kubectl get nodes -o wide       # Show additional information about the nodes.
kubectl top nodes               # Get the current memory usage.

Managing pods:

kubectl get pods --show-labels               # Show all pods and their labels.
kubectl get pods -o wide                     # Inspect how pods get scheduled.
kubectl describe pod <pod-name>              # Inspect pod config (args, envs, ...).
kubectl get pod <pod-name> -o yaml           # Get pod yaml config.
kubectl exec -it <pod-name> -- /bin/bash     # Login to a running pod.
kubectl logs <pod-name>                      # Get logs for a running pod.
kubectl logs memgraph-data-0-0 | tail -n 100 # Filter last logs from a running pod.
kubectl logs --previous <pod-name>           # Get logs from a crashed pod.
kubectl logs <pod-name> -c <container-name>  # Get logs from a specific pod, e.g., debugging init containers.
kubectl cp <pod-name>:<pod-path> .           # Copy logs from a running pod.

Events:

kubectl get events --all-namespaces  --sort-by='.metadata.creationTimestamp' # List all events by creation time.
kubectl get events --namespace <namespace-name>                              # List all events in the given namespace.

Cluster:

kubectl port-forward <pod-name> <host-port>:<pod-port> # Forward/connect port on host to the pod port.
kubectl cluster-info dump                              # Dump current cluster state to stdout.

StatefulSets:

kubectl get statefulsets                  # Show all StatefulSets.
kubectl get pvc                           # Get all PersistentVolumeClaims.
kubectl get pvc -l app=<statefulset-name> # Get the PersistentVolumeClaims for the StatefulSet.

Debugging Memgraph pods

You can attach GDB to a running Memgraph pod using ephemeral debug containers. This approach injects a debug container into an existing pod — no need to redeploy or create a separate privileged pod.

Requirements: kubectl 1.32+, Kubernetes 1.25+ (ephemeral containers must be enabled).

./scripts/debug-memgraph.sh memgraph-data-0-0

# Specify container, namespace, or image explicitly
./scripts/debug-memgraph.sh memgraph-data-0-0 -c memgraph-data -n my-namespace
 
# Use a custom debug image
DEBUG_IMAGE=ubuntu:24.04 ./scripts/debug-memgraph.sh memgraph-data-0-0

kubectl get pods -o wide

apiVersion: v1
kind: Pod
metadata:
  name: debug
spec:
  containers:
  - args:
    - "3600"
    command:
    - sleep
    image: ubuntu:22.04
    name: debug
    imagePullPolicy: Always
    securityContext:
      privileged: true
  hostPID: true
  nodeName: <node-where-your-pod-is-running>  # must match target pod's node
  restartPolicy: Never

kubectl apply -f scripts/perf_pod.yaml

./scripts/find-memgraph-pid.sh memgraph-data-0-0

kubectl exec -it debug -- bash
apt-get update && apt-get install -y gdb procps
gdb -p <PID>

kubectl delete pod debug

Kubernetes official documentation on how to debug running pods covers additional techniques including node-level debugging.

Handling core dumps

When Memgraph crashes, for example, due to segmentation faults (SIGSEGV), core dumps can provide invaluable insight for debugging. The Memgraph Helm charts provide an easy way to enable persistent core dump storage using the createCoreDumpsClaim option.

To enable core dumps, create a values.yaml file with at least the following setting:

createCoreDumpsClaim: true

If you’re running the Memgraph high-availability chart, you can automatically upload core dumps to S3.

Setting this value to true will also enable the use of GDB inside Memgraph containers when using our provided charts.

This instructs the Helm chart to create a PersistentVolumeClaim (PVC) to store core dumps generated by the Memgraph process.

Important configuration notes

By default the storage size is 10GiB. Core dumps can be as large as your node’s total RAM, so it’s recommended to set this explicitly and make sure to adjust the coreDumpsStorageSize under values.yaml file.

Make sure to use the relwithdebinfo image of Memgraph by setting the image.tag also under values.yaml file.

Run the following command to install Memgraph with the debugging configuration:

helm install my-release memgraph/memgraph -f values.yaml

The core dumps are written to a mounted volume inside the container (the default is /var/core/memgraph, it’s possible to tweak that by changing the coreDumpsMountPath under values.yaml). You can use kubectl exec or kubectl cp to access the files for post-mortem analysis.

If you have k8s cluster under any major cloud provider + you want to store the dumps under S3, probably the best repo to check out is the core-dump-handler.

Profiling Memgraph in Kubernetes

Profile a Memgraph process running inside a Kubernetes pod using perf and generate flame graphs.

Prerequisites

• kubectl configured with access to your cluster
• A running Memgraph deployment (standalone or HA)

Step 1: Identify the target pod

kubectl get pods -o wide

In this example, we want to profile memgraph-data-1-0, which is currently the MAIN instance. Note the NODE it is running on — the debug pod must be scheduled on the same node.

Step 2: Deploy the debug pod

Edit perf_pod.yaml and set nodeName to match the target pod’s node:

apiVersion: v1
kind: Pod
metadata:
  name: debug
spec:
  containers:
  - args:
    - "3600"
    command:
    - sleep
    image: ubuntu:22.04
    name: debug
    imagePullPolicy: Always
    securityContext:
      privileged: true
  hostPID: true
  nodeName: aks-nodepool1-38123842-vmss000004  # <-- must match target pod's node
  restartPolicy: Never

kubectl apply -f scripts/perf_pod.yaml

The debug pod needs privileged: true and hostPID: true so it can see host processes and access /proc/<pid>/cgroup to match processes to pods.

Step 3: Find the Memgraph PID

Since multiple Memgraph processes may be visible from the host PID namespace (due to Kubernetes multi-tenancy), we need to match the correct one to our target pod. The find-memgraph-pid.sh script does this automatically — it resolves the pod’s UID, lists all memgraph processes inside the debug pod, and matches via /proc/<pid>/cgroup:

./scripts/find-memgraph-pid.sh memgraph-data-1-0

Output:

Pod:          memgraph-data-1-0
UID:          c8707c88-631b-467c-af9f-26e9dac8e780
UID fragment: 26e9dac8e780
Debug pod:    debug

Found memgraph PIDs: 1335771 1396816
cgroup match: /proc/1396816/cgroup:0::/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod...26e9dac8e780...

Matched memgraph PID: 1396816

Use -d to specify a different debug pod name, or -n for a non-default namespace:

./scripts/find-memgraph-pid.sh memgraph-data-1-0 -d my-debug-pod -n memgraph # Default pod name is 'debug', default namespace is 'default'

Step 4: Install perf in the debug pod

kubectl exec -it debug -- bash

Inside the debug pod:

apt-get update && apt-get install -y linux-tools-common linux-tools-generic

Note (AKS / cloud kernels): apt-get install linux-tools-$(uname -r) will fail if the host kernel is a cloud-specific variant (e.g., 5.15.0-1102-azure) because the matching package isn’t in standard Ubuntu repos. Use linux-tools-generic instead — the generic perf binary works in most cases. If it complains about a version mismatch, invoke it directly:

/usr/lib/linux-tools/*/perf record ...

Step 5: Record a perf profile

perf record -p <PID> --call-graph dwarf sleep 30

Replace <PID> with the PID from Step 3. Adjust the duration (sleep 30) as needed — run your workload during this window.

Step 6: Generate a flame graph

apt-get install -y git
git clone https://github.com/brendangregg/FlameGraph
perf script | ./FlameGraph/stackcollapse-perf.pl > out.perf-folded
./FlameGraph/flamegraph.pl out.perf-folded > perf.svg

Step 7: Copy results and clean up

From your local machine:

kubectl cp debug:perf.svg perf.svg
kubectl cp debug:perf.data perf.data    # optional: raw perf data for later analysis
kubectl delete pod debug

Open perf.svg in a browser to explore the interactive flame graph.

Specific cloud provider instructions

• AWS
• Azure
• GCP

The k8s quick reference is an amazing set of commands!