Kubernetes

Memgraph can be deployed on Kubernetes. The easiest way to do that is with Helm, the package manager for Kubernetes. Helm uses a packaging format called charts. A chart is a collection of files that describe a related set of Kubernetes resources.

Currently, we prepared and released the following charts:

• Memgraph standalone Helm chart
• Memgraph high availability Helm chart
• Memgraph Lab Helm chart

The Helm charts are published on Artifact Hub. For details on the implementation of the Helm charts, check Memgraph Helm charts repository.

Due to numerous possible use cases and deployment setups via Kubernetes, the provided Helm charts are a starting point you can modify according to your needs. This page will highlight some of the specific parts of the Helm charts that you might want to adjust.

Memgraph standalone Helm chart

Memgraph is a stateful application (database), hence the Helm chart for standalone Memgraph is configured to deploy Memgraph as a Kubernetes StatefulSet workload.

It will deploy a single Memgraph instance in a single pod.

Typically, when deploying a stateful application like Memgraph, a StatefulSet workload is used to ensure that each pod has a unique identity and stable network identity. When deploying Memgraph, it is also necessary to define a PersistentVolumeClaims to store the data directory (/var/lib/memgraph). This enables the data to be persisted even if the pod is restarted or deleted.

Storage configuration

By default, the Helm chart will create a PersistentVolumeClaim (PVC) for storage and logs. If the storage class for PVC is not defined, PVC will use the default one available in the cluster. The storage class can be configured in the values.yaml file. To avoid losing your data, make sure you have Retain reclaim policy. If you delete PersistentVolumeClaim without having Retain reclaim policy, you will lose your data because PersistentVolume will be deleted.

An example of a storage class for AWS EBS volumes:

storageClass:
  name: "gp2"
  provisioner: "kubernetes.io/aws-ebs"
  storageType: "gp2"
  fsType: "ext4"
  reclaimPolicy: "Retain"
  volumeBindingMode: "Immediate"

Default template for a storage class is part of the Helm chart and can be found in the repository.

More details on the configuration options can be found in the configuration section.

Secrets

The Helm chart allows you to use Kubernetes secrets to store Memgraph credentials. By default, the secrets are disabled. If you want to use secrets, you can enable them in the values.yaml file.

The secrets are prepared to work for environment variables MEMGRAPH_USER and MEMGRAPH_PASSWORD.

System configuration

The Helm chart will set the linux kernel vm.max_map_count parameter to 262144 by default to ensure Memgraph won’t run into issues with memory mapping.

Installing Memgraph standalone Helm chart

To include a standalone Memgraph into your Kubernetes cluster, you need to add the repository and install Memgraph.

The steps below will work in the Minikube environment, but you can also use them in other Kubernetes environments with minor adjustments.

Add the repository

Add the Memgraph Helm chart repository to your local Helm setup by running the following command:

helm repo add memgraph https://memgraph.github.io/helm-charts

Make sure to update the repository to fetch the latest Helm charts available:

helm repo update

Install Memgraph

To install Memgraph Helm chart, run the following command:

helm install <release-name> memgraph/memgraph

Replace <release-name> with the name of the release you chose.

Access Memgraph

Once Memgraph is installed, you can access it using the provided services and endpoints, such as various client libraries, command-line interface mgconsole or visual user interface Memgraph Lab.

Configuration options

The following table lists the configurable parameters of the Memgraph chart and their default values.

To change the default chart values, provide your own values.yaml file during the installation:

helm install <resource-name> memgraph/memgraph -f values.yaml

Default chart values can also be changed by setting the values of appropriate parameters:

helm install <resource-name> memgraph/memgraph --set <flag1>=<value1>,<flag2>=<value2>,...

memgraphConfig: 
  - "--also-log-to-stderr=true"
  - "--log-level=TRACE"

Memgraph high availability Helm chart

A Helm chart for deploying Memgraph in high availability (HA) setup. This helm chart requires Memgraph Enterprise license.

Memgraph HA cluster includes 3 coordinators and 2 data instances by default. Since multiple Memgraph instances are used, it is advised to use multiple workers nodes in Kubernetes. Our advice is that each Memgraph instance gets on its own node. The size of nodes on which data pods will reside depends on the computing power and the memory you need to store data. Coordinator nodes can be smaller and machines with basic requirements met (8-16 GB of RAM) will be enough.

Installing the Memgraph HA Helm chart

To include Memgraph HA cluster as a part of your Kubernetes cluster, you need to add the repository and install Memgraph.

Add the repository

Add the Memgraph Helm chart repository to your local Helm setup by running the following command:

helm repo add memgraph https://memgraph.github.io/helm-charts

Make sure to update the repository to fetch the latest Helm charts available:

helm repo update

Install Memgraph HA

Since Memgraph HA requires an Enterprise license, you need to provide the license and organization name during the installation.

helm install <release-name> memgraph/memgraph-high-availability --set env.MEMGRAPH_ENTERPRISE_LICENSE=<your-license>,env.MEMGRAPH_ORGANIZATION_NAME=<your-organization-name>

Replace <release-name> with a name of your choice for the release and set the Enterprise license.

Changing the default chart values

To change the default chart values, run the command with the specified set of flags:

helm install <resource-name> memgraph/memgraph-high-availability --set <flag1>=<value1>,<flag2>=<value2>,...

Or you can modify a values.yaml file and override the desired values:

helm install <resource-name> memgraph/memgraph-high-availability -f values.yaml

Upgrade Helm chart

To upgrade the helm chart you can use:

helm upgrade <release-name> memgraph/memgraph-high-availability --set <flag1>=<value1>,<flag2>=<value2>

Again it is possible use both --set and values.yaml to set configuration options.

Uninstall Helm chart

Uninstallation is done with:

helm uninstall <release-name>

Uninstalling the chart won’t trigger deletion of persistent volume claims (PVCs) which means that even if the reclaim policy of the default storage class is set to ‘Delete’ (which often is), the data from persistent volumes (PVs) won’t be lost. However, we advise users to set the reclaim policy to ‘Retain’ as suggessted in the Storage chapter.

Security context

All instances are started as StatefulSet with one pod. The pod has two or three containers depending on whether the sysctlInitContainer.enabled is used. The init container is used to set permissions on volume mounts. It is used as root user with CHOWN capability and without privileged access. The memgraph-coordinator container is the one which actually runs Memgraph image. The process is run by non-root memgraph user without any Linux capabilities. Privileges cannot escalate.

High availability storage

Memgraph HA chart will always use persistent volume claims (PVCs) for storing data and logs. By default they will use 1Gi so most likely you will need to set this parameter to a higher value. Access mode is by default set to ReadWriteOnce (RWO) and can be changed to one of values: ReadWriteOnce, ReadOnlyMany, ReadWriteMany and ReadWriteOncePod. All PVCs will use a default storage class. You can use parameters libStorageClassName and logStorageClassName to specify which storage classes will be used for storing data and logs on data and coordinator instances. Default storage classes usually have a reclaim policy set to Delete which means that if you delete PVCs, attached PVs will also get deleted. We advise users to set the reclaim policy to Retain One of the ways to achieve this is to run the following script after starting your cluster:

#!/bin/bash
 
# Get all Persistent Volume names
PVS=$(kubectl get pv --no-headers -o custom-columns=":metadata.name")
 
# Loop through each PV and patch it
for pv in $PVS; do
  echo "Patching PV: $pv"
  kubectl patch pv $pv -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'
done
 
echo "All PVs have been patched."

Network configuration

All instances which are part of Memgraph HA cluster use the internal Cluster IP network for communicating between themselves. By default, management port is on all instances opened on 10000, replication port on data instances is opened on 20000 while coordinator port is opened on 12000. You can change this configuration by specifying ports.managementPort, ports.replicationPort and ports.coordinatorPort.

External network configuration

Since Memgraph HA works with client-side routing, the resolution of all DNS entries happens outside internal Cluster IP network. Because of that, we need one more type of network which will be used for clients accessing instances inside the cluster.

• IngressNginx: Using only one load balancer you can access all instances from outside.
• NodePort: The cheapest option because it will open port on each node. For this to work, you need to enable public IPs on each node.
• LoadBalancer: This option will use one load balancer for each instance in the cluster. Load balancer is mostly interesting here because it comes with public IP out of the box.

For coordinators, there is an additional option of using CommonLoadBalancer. In this scenario, there is one load balancer sitting in front of coordinators. You can save the cost of two load balancers compared to LoadBalancer option since usually you don’t need to distinguish specific coordinators while using Memgraph capabilities. The default Bolt port is opened on 7687 but you can change it by setting ports.boltPort.

Read more in the guide for installing Memgraph HA with IngressNginx.

Node affinity

Since HA Memgraph deploys multiple pods, you can control on which nodes pods get distributed in the cluster.

The Memgraph HA Helm chart provides the following node affinity options:

• default: Default affinity will try to schedule the data pods and coordinator pods on the nodes where there is no other pod with the same role. If there is no such node, the pods will still be scheduled on the same node, and deployment will not fail.
• unique: This is achieved with the affinity.unique set to true. This option will try to deploy the data pods and coordinator pods on different nodes in the cluster so that each pod is on a unique node. If there are no sufficient nodes, this deployment will fail.
• parity: This is achieved with the affinity.parity set to true. This option will try to deploy the data pods and coordinator pods on the same node with maximum one coordinator and one data pod on the node. If there are no sufficient nodes to deploy the pods, this deployment will fail. Coordinators get scheduled first. After that, data pods are looking for the nodes with coordinators.
• nodeSelection: This is achieved with the affinity.nodeSelection set to true. This option will try to deploy the data pods and coordinator pods on the nodes with specific labels. You can set the labels with the affinity.dataNodeLabelValue and affinity.coordinatorNodeLabelValue parameters. If all the nodes with labels are occupied by the pods with the same role, the deployment will fail.

Sysctl options

You can use sysctlInitContainer configuration parameter to increase the vm_max_map_count which is necessary for high memory loads in Memgraph.

Authentication

By default, there is no user or password configured for Memgraph instances. You can use secrets configuration parameter to create the user with the password.

Setting up the cluster

Although there are many configuration options you can use to set up HA cluster (especially for networking), the set-up process stays always the same:

1. Provision the cluster

2. Label nodes based on the affinity strategy

3. Install memgraph/memgraph-high-availability chart using helm install

4. Install any other auxiliary resources, e.g ingress-nginx if you are using that type of external network configuration.

5. Connect instances to form the cluster

The last step, connecting instances, needs to be done manually to provide information to instances about their external addresses on which they are available. Otherwise, client-side routing wouldn’t work. To connect instances, we use the following queries:

ADD COORDINATOR 1 WITH CONFIG {"bolt_server": "<bolt-server-coord1>", "management_server":  "memgraph-coordinator-1.default.svc.cluster.local:10000", "coordinator_server":  "memgraph-coordinator-1.default.svc.cluster.local:12000"};
ADD COORDINATOR 2 WITH CONFIG {"bolt_server": "<bolt-server-coord2>", "management_server":  "memgraph-coordinator-2.default.svc.cluster.local:10000", "coordinator_server":  "memgraph-coordinator-2.default.svc.cluster.local:12000"};
ADD COORDINATOR 3 WITH CONFIG {"bolt_server": "<bolt-server-coord3>", "management_server":  "memgraph-coordinator-3.default.svc.cluster.local:10000", "coordinator_server":  "memgraph-coordinator-3.default.svc.cluster.local:12000"};
REGISTER INSTANCE instance_0 WITH CONFIG {"bolt_server": "<bolt-server-instance0>", "management_server": "memgraph-data-0.default.svc.cluster.local:10000", "replication_server": "memgraph-data-0.default.svc.cluster.local:20000"};
REGISTER INSTANCE instance_1 WITH CONFIG {"bolt_server": "<bolt-server-instance1>", "management_server": "memgraph-data-1.default.svc.cluster.local:10000", "replication_server": "memgraph-data-1.default.svc.cluster.local:20000"};
SET INSTANCE instance_1 TO MAIN;

Note that the only part which you need to change from the above template is the Bolt server address. The value of the Bolt server address depends on the type of the external network you are using for accessing instances from outside.

Using Memgraph HA chart with IngressNginx resource

The most cost-friendly way to manage a Memgraph HA cluster in K8s is using a IngressNginx contoller. This controller is capable of routing TCP messages on Bolt level protocol to the K8s Memgraph services. To achieve this, it uses only a single LoadBalancer which means there is only a single external IP for connecting to the cluster. Users can connect to any coordinator or data instance by distinguishing bolt ports. The only step required is to install Memgraph HA chart:

helm install mem-ha-test ./charts/memgraph-high-availability --set \
env.MEMGRAPH_ENTERPRISE_LICENSE=<licence>,\
env.MEMGRAPH_ORGANIZATION_NAME=<organization>,affinity.nodeSelection=true,\
externalAccessConfig.dataInstance.serviceType=IngressNginx,externalAccessConfig.coordinator.serviceType=IngressNginx

The chart will also install IngressNginx automatically with all required configuration.

Configuration options

The following table lists the configurable parameters of the Memgraph HA chart and their default values.

For the data and coordinators sections, each item in the list has the following parameters:

The args section contains a list of arguments for the instance.

For all available database settings, refer to the configuration settings docs.

Memgraph Lab Helm chart

A Helm chart for deploying Memgraph Lab on Kubernetes.

Installing the Memgraph Lab Helm chart

To install the Memgraph Lab Helm chart, follow the steps below:

helm install <release-name> memgraph/memgraph-lab

Replace <release-name> with a name of your choice for the release.

Changing the default chart values

To change the default chart values, run the command with the specified set of flags:

helm install <resource-name> memgraph/memgraph-lab --set <flag1>=<value1>,<flag2>=<value2>,...

Or you can modify a values.yaml file and override the desired values:

helm install <resource-name> memgraph/memgraph-lab -f values.yaml

Configuration options

The following table lists the configurable parameters of the Memgraph Lab chart and their default values.

Memgraph Lab can be further configured with environment variables in your values.yaml file.

env:
  - name: QUICK_CONNECT_MG_HOST
    value: memgraph
  - name: QUICK_CONNECT_MG_PORT
    value: "7687"
  - name: KEEP_ALIVE_TIMEOUT_MS
    value: 65000

In case you added Nginx Ingress service or web server for a reverse proxy, update the following proxy timeout annotations to avoid potential timeouts:

proxy_read_timeout X;
proxy_connect_timeout X;
proxy_send_timeout X;

where X is the number of seconds the connection (request query) can be alive. Additionally, update the Memgraph Lab KEEP_ALIVE_TIMEOUT_MS environment variable to a higher value to ensure that Memgraph Lab stays connected to Memgraph when running queries over 65 seconds.

Refer to the Memgraph Lab documentation for details on how to connect to and interact with Memgraph.