Set up HA cluster with K8s Enterprise
Users are advised to first read the guide on how replication works, followed by the guide on how high availability works, and how to query the cluster.
Install Memgraph HA on Kubernetes
To deploy a Memgraph High Availability (HA) cluster on Kubernetes, you must first add the Memgraph Helm repository and then install the HA Helm chart.
Add the Helm repository
Add the Memgraph Helm chart repository to your local Helm setup by running the following command:
helm repo add memgraph https://memgraph.github.io/helm-chartsMake sure to update the repository to fetch the latest Helm charts available:
helm repo updateInstall Memgraph HA
Since Memgraph HA requires an Enterprise
license, you must provide
the license and organization name to the chart through a Kubernetes Secret.
Breaking change: Starting with Memgraph HA chart version 1.0.0, the HA chart no longer accepts
the license and organization name as plaintext values via env.MEMGRAPH_ENTERPRISE_LICENSE
and env.MEMGRAPH_ORGANIZATION_NAME. Both values are now read from a Kubernetes
Secret referenced via secretKeyRef, and the secret must exist before you run
helm install — the StatefulSets will fail to start otherwise. The previous
env.* values have been removed from values.yaml.
Create the secret first, then install the chart:
kubectl create secret generic memgraph-secrets \
--from-literal=MEMGRAPH_ENTERPRISE_LICENSE=<your-license> \
--from-literal=MEMGRAPH_ORGANIZATION_NAME=<your-organization-name>
helm install <release-name> memgraph/memgraph-high-availabilityReplace <release-name> with a name of your choice for the release. The
secret name and keys are configurable via secrets.name, secrets.licenseKey
and secrets.organizationKey (defaults: memgraph-secrets,
MEMGRAPH_ENTERPRISE_LICENSE, MEMGRAPH_ORGANIZATION_NAME).
The cluster will be fully connected once installation completes. Note that the install command may take a moment while instances establish connections. If clients connect from outside the cluster, update the Bolt server address on each instance to use its external IP as explained in the section on setting up the cluster.
latest tag can lead to unexpected behavior if pods restart and pull newer,
incompatible images. Install Memgraph HA with kind
For local development, we suggest using kind. Running:
kind create clustergives you a fully initialized environment that is sufficient for running the HA chart.
Install Memgraph HA with minikube
If you are installing Memgraph HA chart locally with minikube, we are strongly
recommending to enable csi-hostpath-driver and use its storage class.
Otherwise,
you could have problems with attaching PVCs to pods.
Enable csi-hostpath-driver
minikube addons disable storage-provisioner
minikube addons disable default-storageclass
minikube addons enable volumesnapshots
minikube addons enable csi-hostpath-driverCreate a StorageClass (save as sc.yaml)
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: csi-hostpath-delayed
provisioner: hostpath.csi.k8s.io
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: DeleteApply the StorageClass
kubectl apply -f sc.yaml
Configure the Helm chart
In your values.yaml, set:
storage:
libStorageClassName: csi-hostpath-delayedConfigure the Helm chart
Override default chart values
You can customize the Memgraph HA Helm chart either inline with --set flags or
by using a values.yaml file.
Option 1: Override values inline
helm install <release-name> memgraph/memgraph-high-availability \
--set <flag1>=<value1>,<flag2>=<value2>,...Option 2: Use a values file
helm install <release-name> memgraph/memgraph-high-availability \
-f values.yamlYou can also combine both approaches. Values specified with --set override
those in values.yaml.
Upgrade Helm chart
To upgrade the helm chart you can use:
helm upgrade <release-name> memgraph/memgraph-high-availability --set <flag1>=<value1>,<flag2>=<value2>Again it is possible use both --set and values.yaml to set configuration
options.
If you’re using IngressNginx and performing an upgrade, the attached public IP
should remain the same. It will only change if the release includes specific
updates that modify it—and such changes will be documented.
Uninstall Helm chart
Uninstallation is done with:
helm uninstall <release-name>Uninstalling the chart does not delete PersistentVolumeClaims (PVCs). Even
if the default StorageClass reclaim policy is Delete, data on the underlying
PersistentVolumes (PVs) will not be removed automatically when uninstalling the
chart.
However, we still recommend configuring the reclaim policy to Retain, as
described in the High availability storage
section.
Runtime environment & security
Security context
All Memgraph HA instances run as Kubernetes StatefulSet workloads, each with a
single pod. Depending on configuration, the pod contains two or three
containers:
- memgraph-coordinator - runs the Memgraph binary.
- Optional sysctl init container - enabled when
sysctlInitContainer.enabledis set. - Optional fix-ownership init container - enabled when
fixOwnershipInitContainer.enabledis set. See Manual ownership fix.
Memgraph processes run as the non-root memgraph user with no Linux capabilities
and no privilege escalation. The security context is especially important when deploying the HA chart to Red Hat OpenShift. In a
sandboxed environment, you will have to disable the init-sysctl init container. You will most likely also need to change memgraphUserId
and memgraphGroupId so that they fall within OpenShift’s allowed UID/GID range.
High availability storage
Memgraph HA always uses PersistentVolumeClaims (PVCs) to store database files and logs.
- Default storage size: 1Gi (you will likely need to increase this).
- Default access mode:
ReadWriteOnce(can be set toReadOnlyMany,ReadWriteMany, orReadWriteOncePod). - PVCs use the cluster’s default StorageClass, unless overridden.
You can explicitly set storage classes using:
storage.libStorageClassName- for data volumesstorage.logStorageClassName- for log volumes
Most default StorageClasses use a Delete reclaim policy, meaning deleting the
PVC deletes the underlying PersistentVolume (PV). We recommend switching to
Retain.
After your cluster is running, you can patch all PVs:
#!/bin/bash
PVS=$(kubectl get pv --no-headers -o custom-columns=":metadata.name")
for pv in $PVS; do
echo "Patching PV: $pv"
kubectl patch pv $pv -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'
done
echo "All PVs have been patched."Kubernetes uses Storage Object in Use Protection, preventing deletion of
PVCs while still attached to pods.
Similarly, PVs will remain until their PVCs are fully removed.
If a PVC is stuck terminating, you can remove its finalizers:
kubectl patch pvc PVC_NAME -p '{"metadata":{"finalizers": []}}' --type=mergeNetwork configuration
All Memgraph HA components communicate internally using ClusterIP network for communicating between themselves.
Default ports:
- Management: 10000
- Replication (data instances): 20000
- Coordinator communication: 12000
You can change this configuration by specifying:
ports:
managementPort: <value>
replicationPort: <value>
coordinatorPort: <value>External network configuration
Memgraph HA uses client-side routing, so DNS resolution happens on the driver. Because of that, we need one more type of network which will be used for clients accessing instances from outside the cluster. Our HA supports out of the box following K8s resources used to setup external access:
- IngressNginx - one LoadBalancer for all instances. TCP port bases are configured under
externalAccessConfig.ingress. - NodePort - exposes ports on each node (requires public node IPs).
- LoadBalancer - one LoadBalancer per instance (highest cost).
- CommonLoadBalancer (coordinators only) - single LB for all coordinators.
- Gateway API - uses Kubernetes Gateway API resources (Gateway + TCPRoute). Configured under
externalAccessConfig.gateway.
For coordinators, there is an additional option of using CommonLoadBalancer.
In this scenario, there is one load balancer sitting in front of coordinators.
You can save the cost of two load balancers compared to LoadBalancer option
since usually you don’t need to distinguish specific coordinators while using
Memgraph capabilities. Note that if you will be connecting to the coordinator directly for some reason (e.g to run show instances query),
you can run show instance query to see which coordinator you got routed to.
The default Bolt port is opened on 7687 but you can change it by setting ports.boltPort.
For more detailed IngressNginx setup, see Use Memgraph HA chart with IngressNginx.
Note however that Ingress Nginx is getting retired and one of the alternatives is using the Kubernetes Gateway API with controllers like Envoy Gateway, Istio, Cilium, Traefik, or Kong. The HA chart has native Gateway API support — see Use Memgraph HA chart with Gateway API.
By default, the chart does not expose any external network services.
Recommended external access setup
External access methods can be mixed per tier: coordinators and data
instances each pick their own method independently. Our recommendation is to
expose coordinators through a CommonLoadBalancer and data instances
through IngressNginx or the Gateway API. This splits external access across
two independent load balancers, so routing stays available even if a single
load balancer goes down: if the load balancer in front of the data instances
fails, clients can still reach the coordinators and get routing information,
and if the coordinators’ load balancer fails, connections to the data
instances keep working. If everything sits behind one shared load balancer,
that load balancer is a single point of failure for all external access.
To set up the recommended combination with the Gateway API, leave
externalAccessConfig.dataInstance.serviceType empty — the Gateway only
exposes tiers whose serviceType is empty, so it will expose the data
instances only:
externalAccessConfig:
coordinator:
serviceType: "CommonLoadBalancer"
annotations:
external-dns.alpha.kubernetes.io/hostname: "memgraph.coordinators.example.com"
# dataInstance.serviceType stays empty, so the Gateway exposes only the data instances
gateway:
enabled: true
gatewayClassName: "eg"
annotations:
external-dns.alpha.kubernetes.io/hostname: "memgraph.data.example.com"Or with IngressNginx for the data instances:
externalAccessConfig:
dataInstance:
serviceType: "IngressNginx"
annotations:
external-dns.alpha.kubernetes.io/hostname: "memgraph.data.example.com"
coordinator:
serviceType: "CommonLoadBalancer"
annotations:
external-dns.alpha.kubernetes.io/hostname: "memgraph.coordinators.example.com"In both cases every coordinator is reachable at the shared
memgraph.coordinators.example.com:<boltPort> address, and each data instance
at memgraph.data.example.com:<dataPortBase + id>. With the external-dns
hostnames set, the cluster-setup Job registers the external bolt_server
addresses automatically (see Update bolt server).
All other combinations are valid as well — for coordinators + data instances, for example:
CommonLoadBalancer+LoadBalancerLoadBalancer+LoadBalancerIngressNginx+IngressNginx(everything behind the single ingress-nginx load balancer)- Gateway API for everything (leave both
serviceTypefields empty and enable the gateway) NodePortfor either tier, combined with any of the above
The only invalid combination is enabling the Gateway API
(externalAccessConfig.gateway.enabled: true) while both tiers have a
serviceType set: a tier with a serviceType is excluded from the Gateway,
so the Gateway would apply to nothing and chart rendering fails.
Per-instance external access annotations
When using LoadBalancer or NodePort external access, you can set annotations
globally via externalAccessConfig.dataInstance.annotations and
externalAccessConfig.coordinator.annotations. These apply to every external
Service of that type.
If you need different annotations per instance — for example, to assign unique
DNS hostnames via external-dns — use the externalAccessAnnotations field on
individual entries in data[] or coordinators[]. Per-instance annotations are
merged with the global annotations, and per-instance values take precedence
when the same key appears in both.
externalAccessConfig:
dataInstance:
serviceType: "LoadBalancer"
annotations:
service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
data:
- id: "0"
externalAccessAnnotations:
external-dns.alpha.kubernetes.io/hostname: "data-0.memgraph.example.com"
- id: "1"
externalAccessAnnotations:
external-dns.alpha.kubernetes.io/hostname: "data-1.memgraph.example.com"In this example, each data instance’s external Service gets the shared
aws-load-balancer-scheme annotation plus its own unique external-dns
hostname. Bolt and management ports are not set per-instance — they come from
ports.boltPort and ports.managementPort.
With serviceType: LoadBalancer and a per-instance external-dns hostname
set like above, the cluster-setup Job automatically registers each
instance’s routing bolt_server as <hostname>:<boltPort> (see Update bolt
server). Do not set the external-dns hostname at
the tier level (externalAccessConfig.dataInstance.annotations /
externalAccessConfig.coordinator.annotations) when using LoadBalancer:
each instance gets its own LoadBalancer Service, so a tier-level hostname
would put one DNS record in front of every load balancer and register the
same bolt_server for all instances.
Per-instance internal access annotations
Each data instance and coordinator also has an internal ClusterIP Service used
for in-cluster communication. You can set per-instance annotations on these
internal Services using the internalAccessAnnotations field on individual
entries in data[] or coordinators[]. This is useful for integrations or other tooling that consumes annotations on the internal
Services.
data:
- id: "0"
internalAccessAnnotations:
mycompany.io/service-mesh: "enabled"
- id: "1"
internalAccessAnnotations:
mycompany.io/service-mesh: "enabled"
coordinators:
- id: "1"
internalAccessAnnotations:
mycompany.io/service-mesh: "enabled"Node affinity
Memgraph HA deploys multiple pods, and you can control pod placement with affinity settings.
Supported strategies:
-
default Attempts to to schedule the data pods and coordinator pods on the nodes where there is no other pod with the same role. If there is no such node, the pods will still be scheduled on the same node, and deployment will not fail.
-
unique (
affinity.unique = true) Each coordinator and data pod must be placed on separate nodes. If not enough nodes exist, deployment fails. Coordinators get scheduled first. After that, data pods are looking for the nodes with coordinators. -
parity (
affinity.parity = true) Schedules at most one coordinator + one data pod per node. Coordinators schedule first; data pods follow. -
nodeSelection (
affinity.nodeSelection = true) Pods are scheduled onto explicitly labeled nodes usingaffinity.dataNodeLabelValueandaffinity.coordinatorNodeLabelValue. If all the nodes with labels are occupied by the pods with the same role, the deployment will fail.
When using nodeSelection, ensure that nodes are labeled correctly.
Default role label key: role
Default values: data-node, coordinator-node
Example:
kubectl label nodes <node-name> role=data-nodeA full AKS example is available in the chart repository.
Sysctl options
Use the sysctlInitContainer to configure kernel parameters required for
high-memory workloads, such as increasing:
Manual ownership fix
Some storage drivers (notably rancher.io/local-path) do not honor pod-level
fsGroup, leaving the volume root owned by root:root. Because Memgraph runs
as a non-root user, its storage directory ownership assertion (process euid ==
data directory owner uid) fails on startup.
When fixOwnershipInitContainer.enabled is set to true, an init container
runs as root before Memgraph starts and chowns the lib, log, and core-dumps
mount points to memgraphUserId:memgraphGroupId. The container drops all Linux
capabilities except CHOWN, uses a read-only root filesystem, and disables
privilege escalation.
To enable it:
fixOwnershipInitContainer:
enabled: true
image:
repository: docker.io/library/busybox
tag: 1.37.0
pullPolicy: IfNotPresentThe container only chowns the mount paths that exist for the role — /var/log/memgraph
is included when storage.<role>.createLogStorageClaim is true, and
storage.<role>.coreDumpsMountPath is included when storage.<role>.createCoreDumpsClaim
is true.
Authentication
By default, Memgraph HA starts without authentication enabled.
Breaking change: The HA chart no longer creates a Memgraph user from the
USER/PASSWORD keys of the memgraph-secrets Secret. The secrets.enabled,
secrets.userKey and secrets.passwordKey values have been removed because
the previous implementation also applied these env variables to coordinators,
which run without auth. The memgraph-secrets Secret is now reserved for the
license and organization name.
To configure credentials, connect to a data instance after installation and create users with Cypher, for example:
CREATE USER memgraph IDENTIFIED BY 'memgraph';Run the same statements on every data instance you want the user to exist on. Coordinators run without authentication and do not need user setup.
Bolt SSL/TLS
Each data instance and coordinator can independently terminate Bolt
connections over TLS. When enabled, the chart mounts a pre-existing
Kubernetes Secret containing the certificate and private key at
/etc/memgraph/ssl/ and auto-appends --bolt-cert-file=/etc/memgraph/ssl/tls.crt
and --bolt-key-file=/etc/memgraph/ssl/tls.key to the instance’s args.
Breaking change in HA chart version with TLS config: The previous way of
enabling Bolt TLS — passing --bolt-cert-file / --bolt-key-file through
data[].args / coordinators[].args and mounting the certificate Secret
through storage.{data,coordinators}.extraVolumes / extraVolumeMounts — is
no longer supported. Setting --bolt-cert-file or --bolt-key-file in args
now causes helm install to fail with a template error. Migrate to the
tls.bolt block on each instance instead.
To enable Bolt TLS, first create a Kubernetes Secret holding the certificate and private key in the release namespace:
kubectl create secret tls bolt-tls-secret \
--cert=path/to/tls.crt \
--key=path/to/tls.keyThen enable tls.bolt on each instance that should terminate TLS:
data:
- id: "0"
tls:
bolt:
enabled: true
secretName: bolt-tls-secret
certSecretPath: tls.crt
keySecretPath: tls.key
- id: "1"
tls:
bolt:
enabled: true
secretName: bolt-tls-secret
certSecretPath: tls.crt
keySecretPath: tls.key
coordinators:
- id: "1"
tls:
bolt:
enabled: true
secretName: bolt-tls-secret
- id: "2"
tls:
bolt:
enabled: true
secretName: bolt-tls-secret
- id: "3"
tls:
bolt:
enabled: true
secretName: bolt-tls-secretcertSecretPath and keySecretPath are the keys inside the Secret holding
the certificate and key respectively (default tls.crt and tls.key).
The chart fails the install if tls.bolt.enabled is true but
tls.bolt.secretName is empty.
When a coordinator has tls.bolt.enabled: true, the cluster-setup job
that registers coordinators and data instances automatically uses
--use-ssl when connecting to coordinator 1.
Intra-cluster SSL/TLS
Independently of Bolt TLS, each data instance and coordinator can encrypt the
internal communication between cluster members (coordinator-to-coordinator and
coordinator-to-data traffic). When tls.intraCluster.enabled is true, the
chart mounts a pre-existing Kubernetes Secret containing the certificate,
private key and CA bundle at /etc/memgraph/intra_cluster_tls/ and
auto-appends --cluster-cert-file=/etc/memgraph/intra_cluster_tls/tls.crt,
--cluster-key-file=/etc/memgraph/intra_cluster_tls/tls.key and
--cluster-ca-file=/etc/memgraph/intra_cluster_tls/ca.crt to the instance’s
args.
To enable intra-cluster TLS, first create a Kubernetes Secret holding the
certificate, private key and CA bundle for each instance in the release
namespace. For example, for data-0:
kubectl create secret generic intra-tls-data-0-secret \
--from-file=tls.crt=path/to/tls.crt \
--from-file=tls.key=path/to/tls.key \
--from-file=ca.crt=path/to/ca.crtThen enable tls.intraCluster on each instance that should encrypt internal
traffic:
data:
- id: "0"
tls:
intraCluster:
enabled: true
secretName: intra-tls-data-0-secret
certSecretPath: tls.crt
keySecretPath: tls.key
caSecretPath: ca.crt
- id: "1"
tls:
intraCluster:
enabled: true
secretName: intra-tls-data-1-secret
certSecretPath: tls.crt
keySecretPath: tls.key
caSecretPath: ca.crt
coordinators:
- id: "1"
tls:
intraCluster:
enabled: true
secretName: intra-tls-coord-1-secret
- id: "2"
tls:
intraCluster:
enabled: true
secretName: intra-tls-coord-2-secret
- id: "3"
tls:
intraCluster:
enabled: true
secretName: intra-tls-coord-3-secretcertSecretPath, keySecretPath and caSecretPath are the keys inside the
Secret holding the certificate, private key and CA bundle respectively
(default tls.crt, tls.key and ca.crt). The chart fails the install if
tls.intraCluster.enabled is true but tls.intraCluster.secretName is
empty. Enable it on every instance that participates in encrypted intra-cluster
communication.
Setting up the cluster
Although many configuration options exist, especially for networking, the workflow for creating a Memgraph HA cluster follows these steps:
- Provision the Kubernetes cluster. Ensure your nodes, storage, and networking are ready.
- Label nodes according to your chosen affinity strategy (optional). For example, when using
nodeSelection, label nodes asdata-nodeorcoordinator-node. - Create the
memgraph-secretsKubernetes secret holdingMEMGRAPH_ENTERPRISE_LICENSEandMEMGRAPH_ORGANIZATION_NAME(required — the chart reads these viasecretKeyRef). - Install the Memgraph HA Helm chart using
helm install. This creates a fully connected cluster. - Install auxiliary components for external access, such as
ingress-nginx(optional). - Update Bolt server addresses if clients will connect from outside the cluster (optional).
Customize the cluster-setup Job
After helm install, the chart runs a cluster-setup Job as a Helm
post-install and post-upgrade hook. This Job registers all coordinators and
data instances via mgconsole, producing a fully connected cluster, and re-runs
on every helm upgrade to reconcile cluster membership (see Scale the cluster
up and down).
You can customize the labels and security context applied to this Job through
the clusterSetup block:
clusterSetup.labels- custom labels applied to both the Job and its pod template. Useful for matching network policies, cost-allocation, or observability selectors that target the setup Job.clusterSetup.podSecurityContext- pod-level security context for thecluster-setuppod.clusterSetup.containerSecurityContext- container-level security context for thecluster-setupcontainer.
By default the Job runs hardened: as the non-root memgraph user
(runAsUser: 101, runAsGroup: 103), with a read-only root filesystem, all
Linux capabilities dropped, no privilege escalation, and the RuntimeDefault
seccomp profile. Override these values when deploying to a restricted
environment such as Red Hat OpenShift, where the allowed UID/GID range differs:
clusterSetup:
labels:
app.kubernetes.io/component: cluster-setup
podSecurityContext:
runAsUser: 1000680000
runAsGroup: 1000680000
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
containerSecurityContext:
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
readOnlyRootFilesystem: true
runAsNonRoot: true
seccompProfile:
type: RuntimeDefaultScale the cluster up and down
The cluster-setup Job runs as both a post-install and a post-upgrade hook,
so it re-runs on every helm upgrade and reconciles the registered cluster with
your chart values. All operations are idempotent — existing coordinators and data
instances are left untouched:
- Scale out: add an entry to
coordinators[]ordata[]and runhelm upgrade. The new coordinator is registered withADD COORDINATORand the new data instance withREGISTER INSTANCE. - Scale down: remove an entry from
coordinators[]ordata[]and runhelm upgrade. Members that are registered in the cluster but no longer present in the values are removed withREMOVE COORDINATOR/UNREGISTER INSTANCE.
Instance names are keyed by their stable id, not by their position in the
array, so removal is position-independent: dropping a middle entry unregisters
exactly that member and leaves the others’ names and endpoints intact. Data
instance names are derived as instance_<id+1> (so ids 0, 1, 2 map to
instance_1, instance_2, instance_3), and coordinator names as
coordinator_<id>.
Scale-down safety: the Job never removes the current MAIN data instance or
the leader coordinator, because unregistering either can corrupt cluster state.
When you scale one of those down, its pod is deleted and the cluster fails over /
re-elects; on a later helm upgrade the removed member is no longer MAIN/leader
and is cleaned up then. To remove it immediately, first promote/elect a different
member, then upgrade — or run UNREGISTER INSTANCE / REMOVE COORDINATOR
manually on the leader coordinator.
Breaking change for non-default data instance IDs: data instance names are
now derived from each entry’s id (instance_<id+1>) instead of its position in
the data array. For the shipped default ids (0, 1, 2) the names are unchanged
(instance_1, instance_2, instance_3), so those deployments upgrade safely.
If you use non-sequential or non-zero-based ids (e.g. 5, 6, 7), the
registered instance names change on upgrade: the cluster-setup Job will
REGISTER the new names and UNREGISTER the old ones during reconciliation. The
current MAIN is protected, but replicas are torn down and re-registered under
their new names. Review your data[].id values before upgrading, and expect a
brief replica re-registration if they don’t follow the default 0-based
sequential scheme. In that case you may need to run UNREGISTER INSTANCE and
REMOVE COORDINATOR manually to consolidate the cluster state.
Each data[].id must be a unique, non-negative integer given as a string (e.g.
"0", "1", "2"). Non-numeric ids are not supported, as they would all
collapse to the same instance name.
Update bolt server
This step is required only when:
- Clients access the database from outside the cluster, and
- You’re using bolt+routing for client-side routing
When you expose the cluster through IngressNginx, Gateway API, the
coordinator CommonLoadBalancer, or per-instance LoadBalancer Services
and set an external-dns hostname via annotations, the cluster-setup Job
sets each instance’s routing bolt_server to the external hostname and port
automatically, and reconciles it on every helm upgrade (including reverting
to the internal address when external access is turned off). In that case you
don’t need to run the queries below manually. For LoadBalancer, set the
hostname per instance via data[].externalAccessAnnotations /
coordinators[].externalAccessAnnotations — each instance has its own
LoadBalancer Service, so a tier-level hostname would register the same
bolt_server for every instance. This does not apply to NodePort (or to
instances without a hostname annotation), where the internal bolt_server is
kept and you update it manually.
Each instance must know its external address for routing to work correctly. Run the following queries on the leader coordinator:
UPDATE CONFIG FOR COORDINATOR 1 WITH CONFIG {"bolt_server": "<bolt-server-coord1>"};
UPDATE CONFIG FOR COORDINATOR 2 WITH CONFIG {"bolt_server": "<bolt-server-coord2>"};
UPDATE CONFIG FOR COORDINATOR 3 WITH CONFIG {"bolt_server": "<bolt-server-coord3>"};
UPDATE CONFIG FOR INSTANCE instance_0 WITH CONFIG {"bolt_server": "<bolt-server-instance0>"};
UPDATE CONFIG FOR INSTANCE instance_1 WITH CONFIG {"bolt_server": "<bolt-server-instance1>"};Note that the only the bolt_server values are provided. The correct
value depends on the type of external access you configured (LoadBalancer IP,
Ingress host/port, NodePort, etc.).
Refer to the Memgraph HA User API docs for the full set of commands and usage patterns.
Use Memgraph HA chart with Gateway API
The Memgraph HA Helm chart has native support for the Kubernetes Gateway API. When enabled, the chart automatically creates TCPRoute resources for each data and coordinator instance. You can either let the chart create its own Gateway or attach routes to a pre-existing one.
The Gateway only exposes tiers whose serviceType is empty: a tier with
externalAccessConfig.dataInstance.serviceType or
externalAccessConfig.coordinator.serviceType set gets its external access
from that service type instead and is excluded from the Gateway (no listeners
or TCPRoutes are created for it). This lets you mix methods per tier — for
example, coordinators via CommonLoadBalancer and data instances via the
Gateway, which is our recommended setup.
Enabling the gateway while both tiers have a serviceType set fails chart
rendering, since the Gateway would then apply to nothing.
Prerequisites
Before enabling Gateway API in the chart, you need:
-
The Gateway API CRDs installed in your cluster. Some controllers ship them with their own Helm chart (Envoy Gateway does), while others expect you to install them yourself (Traefik does). The controller-specific guides below cover both cases.
-
A Gateway API controller installed in your cluster. Examples include Envoy Gateway, Istio, Cilium, Traefik, and Kong. Step-by-step guides are available for two of them: Envoy Gateway and Traefik.
-
A GatewayClass resource that references your controller. A GatewayClass is a cluster-scoped resource that defines which controller manages Gateways — each Gateway references a GatewayClass by name. The Memgraph Helm chart does not create a GatewayClass; you must create one yourself or use one provided by your controller installation (the Envoy Gateway chart does not create one, the Traefik chart does).
The chart exposes Bolt through TCPRoute, which is part of the Gateway API
experimental channel. A standard-channel CRD bundle is not enough — whatever
you install must include tcproutes.gateway.networking.k8s.io, and the
controller itself may need experimental support turned on (Traefik does, see
Deploy with Traefik).
You must ensure the GatewayClass exists before enabling the gateway feature in the chart. If you create your own Gateway (Option 1 below), the chart requires gatewayClassName to reference an existing GatewayClass, and will fail with an error if it is not set.
Option 1: Chart-managed Gateway
When you want the chart to create its own Gateway along with TCPRoute resources, set externalAccessConfig.gateway.enabled to true and provide the gatewayClassName:
externalAccessConfig:
gateway:
enabled: true
gatewayClassName: "eg"The chart will create:
- A Gateway (
gateway.networking.k8s.io/v1) with TCP listeners auto-generated for each data and coordinator instance whose tier has an emptyserviceType. - A TCPRoute (
gateway.networking.k8s.io/v1alpha2) per such instance, routing traffic from the Gateway listener to the instance’s Bolt port.
Data instance ports are assigned as dataPortBase + data instance id (default: 9000, 9001, …) and coordinator ports as coordinatorPortBase + coordinator id (default: 10001, 10002, 10003). The coordinator base port is kept well above the data base port so the two ranges never overlap. You can customize the base ports:
externalAccessConfig:
gateway:
enabled: true
gatewayClassName: "eg"
dataPortBase: 9000
coordinatorPortBase: 10000You can also set annotations and labels on the Gateway resource:
externalAccessConfig:
gateway:
enabled: true
gatewayClassName: "eg"
annotations:
example.io/owner: "memgraph"
labels:
app: memgraph-haTo install with a chart-managed Gateway (assuming the memgraph-secrets
Secret with the license and organization name already exists, see Install
Memgraph HA):
helm install memgraph-ha memgraph/memgraph-high-availability \
--set externalAccessConfig.gateway.enabled=true \
--set externalAccessConfig.gateway.gatewayClassName=egOption 2: Existing (external) Gateway
When you already have a Gateway resource in your cluster (for example, a shared Gateway serving multiple services including Memgraph Lab), you can have the chart create only TCPRoute resources that attach to it:
externalAccessConfig:
gateway:
enabled: true
existingGatewayName: "memgraph-gateway"In this mode, the chart skips Gateway creation and only creates TCPRoute resources. The gatewayClassName is not required.
If the existing Gateway is in a different namespace, specify it:
externalAccessConfig:
gateway:
enabled: true
existingGatewayName: "memgraph-gateway"
existingGatewayNamespace: "gateway-system"To install with an existing Gateway (assuming the memgraph-secrets Secret
with the license and organization name already exists, see Install Memgraph
HA):
helm install memgraph-ha memgraph/memgraph-high-availability \
--set externalAccessConfig.gateway.enabled=true \
--set externalAccessConfig.gateway.existingGatewayName=memgraph-gatewayWhen using an existing Gateway, ensure it has listeners whose names match the
TCPRoute sectionName references. The chart expects listener names in the format
data-{id}-bolt for data instances and coordinator-{id}-bolt for coordinators.
The listener ports are your choice in this mode, but keeping them aligned with the
chart defaults avoids surprises. For example, the default HA setup (2 data
instances, 3 coordinators) needs these listeners:
data-0-bolton port 9000data-1-bolton port 9001coordinator-1-bolton port 10001coordinator-2-bolton port 10002coordinator-3-bolton port 10003
A standalone Gateway manifest with these pre-configured listeners is available in the Helm charts repository.
TCPRoute API version: TCPRoute uses v1alpha2, which is the latest available API version. It is supported by Envoy Gateway and other major implementations but is not yet GA. Gateway and HTTPRoute are both GA (v1).
Deploy with Envoy Gateway
Envoy Gateway is the controller used in the examples above. Its Helm chart bundles the Gateway API CRDs, so no separate CRD installation is needed, and it programs listener ports dynamically — the Bolt ports only have to be configured in the Memgraph chart.
Install Envoy Gateway
helm install eg oci://docker.io/envoyproxy/gateway-helm --version v1.2.4 \
-n envoy-gateway-system --create-namespaceChart version v1.2.4 ships Gateway API bundle version v1.2.1, which includes
the TCPRoute CRD.
Create the GatewayClass
The Envoy Gateway chart does not create a GatewayClass, so create one that points at its controller:
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: eg
spec:
controllerName: gateway.envoyproxy.io/gatewayclass-controllerkubectl apply -f gatewayclass.yamlInstall the Memgraph HA chart
Assuming the memgraph-secrets Secret with the license and organization name
already exists (see Install Memgraph HA):
helm install memgraph-ha memgraph/memgraph-high-availability \
--set externalAccessConfig.gateway.enabled=true \
--set externalAccessConfig.gateway.gatewayClassName=egVerify the setup
kubectl get gatewayclass eg
kubectl get gateway -o wide
kubectl get tcprouteThe GatewayClass should be ACCEPTED=True and the Gateway PROGRAMMED=True with
an external address. Envoy Gateway provisions a separate Envoy deployment and
LoadBalancer Service for the Gateway.
Deploy with Traefik
Traefik can serve the same setup, but its Gateway API support differs from Envoy Gateway in two ways that affect the Memgraph chart:
- Its Helm chart ships no Gateway API CRDs, so you install them yourself, and
TCPRoutesupport has to be explicitly enabled. - Traefik does not open arbitrary listener ports. Every port used by a Gateway listener must also be declared as a Traefik entryPoint, otherwise the listener is silently ignored.
Install the Gateway API CRDs
Install the experimental channel, which is where TCPRoute lives:
kubectl apply --server-side --force-conflicts \
-f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.5.1/experimental-install.yamlUse --server-side. A plain kubectl apply stores the whole manifest in the
kubectl.kubernetes.io/last-applied-configuration annotation and the HTTPRoute
CRD exceeds the 256 KB annotation limit, failing with
metadata.annotations: Too long: may not be more than 262144 bytes.
--force-conflicts is needed only when the CRDs were previously created with a
client-side apply.
Configure Traefik
Create a traefik-values.yaml file. The ports entries are the entryPoints
backing the Gateway listeners the Memgraph chart creates — data instances on
dataPortBase + data instance id, coordinators on
coordinatorPortBase + coordinator id:
providers:
kubernetesGateway:
enabled: true
# Required for TCPRoute support.
experimentalChannel: true
statusAddress:
service:
# Copies the Traefik LoadBalancer address into Gateway .status.addresses,
# which is what external-dns reads.
enabled: true
# The Memgraph chart creates its own Gateway, so Traefik's default one is not needed.
gateway:
enabled: false
# Creates the GatewayClass "traefik" (controller traefik.io/gateway-controller).
gatewayClass:
enabled: true
service:
type: LoadBalancer
ports:
data-0:
port: 9000
exposedPort: 9000
protocol: TCP
expose:
default: true
data-1:
port: 9001
exposedPort: 9001
protocol: TCP
expose:
default: trueA ready-to-use version of this file is available in the Helm charts repository.
Declare one entryPoint per Gateway listener, and remember to add new ones when
you scale the cluster — a third data instance
needs port 9002, a fourth coordinator needs port 10004. If a tier is exposed
through a serviceType instead of the Gateway, it has no listeners and needs no
entryPoints; with the recommended setup
(coordinators via CommonLoadBalancer, data instances via the Gateway) only the
data-* entryPoints are required.
Install Traefik
helm repo add traefik https://traefik.github.io/charts
helm repo update
helm install traefik traefik/traefik -n traefik --create-namespace \
-f traefik-values.yamlThe values above were verified against Traefik chart 41.0.2 (Traefik v3.7.6).
The providers.kubernetesGateway.* options require Traefik 3.x.
Install the Memgraph HA chart
Point gatewayClassName at the GatewayClass created by the Traefik chart.
Assuming the memgraph-secrets Secret with the license and organization name
already exists (see Install Memgraph HA):
helm install memgraph-ha memgraph/memgraph-high-availability \
--set externalAccessConfig.gateway.enabled=true \
--set externalAccessConfig.gateway.gatewayClassName=traefikVerify the setup
kubectl get gatewayclass traefik
kubectl get gateway -o wide
kubectl get tcproute
# The chart names the Gateway <release-name>-memgraph-high-availability-gateway
kubectl describe gateway memgraph-ha-memgraph-high-availability-gatewayIf a listener reports Detached or UnsupportedProtocol, the matching entryPoint
is missing from the Traefik values. Unlike Envoy Gateway, Traefik routes all
Gateways through the single Traefik Service, so the external address comes from
the traefik Service in the traefik namespace.
Use Memgraph HA chart with IngressNginx
One of the most cost-efficient ways to expose a Memgraph HA cluster is by using IngressNginx. The controller supports TCP routing (including the Bolt protocol), allowing all Memgraph instances to share:
- a single LoadBalancer, and
- a single external IP address.
Clients connect to any coordinator or data instance by using different Bolt ports.
To install Memgraph HA with IngressNginx enabled (assuming the
memgraph-secrets Secret with the license and organization name already
exists, see Install Memgraph HA):
helm install mem-ha-test ./charts/memgraph-high-availability --set \
affinity.nodeSelection=true,\
externalAccessConfig.dataInstance.serviceType=IngressNginx,\
externalAccessConfig.coordinator.serviceType=IngressNginxWhen using these settings, the chart will automatically install and configure IngressNginx, including all required TCP routing setup for Memgraph.
The IngressNginx service type can be combined with the Gateway API
(externalAccessConfig.gateway.enabled) as long as at least one tier keeps an
empty serviceType — the Gateway only exposes tiers whose serviceType is
empty, so a tier using IngressNginx is simply excluded from the Gateway.
Enabling the gateway while both tiers have a serviceType set fails chart
rendering.
All IngressNginx tiers share a single ingress-nginx controller Service (one
LoadBalancer, one external IP). When you set an external-dns hostname via
annotations, the data instance and coordinator annotations are merged onto that
shared Service (the coordinator value wins on a key collision), so set the same
hostname on both tiers. Instances behind the shared Service are told apart by
port, not host: data instances are reachable at ingress.dataPortBase + data instance id and coordinators at ingress.coordinatorPortBase + coordinator id.
Probes
Memgraph HA uses standard Kubernetes startup, readiness, and liveness probes to ensure correct container operation.
-
Startup probe Determines when Memgraph has fully started. It succeeds only after database recovery completes. Liveness and readiness probes do not run until startup succeeds.
-
Readiness probe Indicates when the instance is ready to accept client traffic.
-
Liveness probe Determines when the container should be restarted if it becomes unresponsive.
Default timing
-
On data instances, the startup probe must succeed within 2 hours. If recovery (e.g., from backup) may take longer, increase the timeout.
-
Liveness and readiness probes must succeed at least once every 5 minutes for the pod to be considered healthy.
Probe endpoints
- Coordinators: probed on the NuRaft server
- Data instances: probed on the Bolt server
Breaking change (HA Helm chart 1.0.0): The probe target port is no longer configurable through
container.data.{readinessProbe,livenessProbe,startupProbe}.tcpSocket.port or
container.coordinators.{readinessProbe,livenessProbe,startupProbe}.tcpSocket.port.
Probes are now hard-coded to a tcpSocket check against ports.boltPort for data
instances and ports.coordinatorPort for coordinators. Only the probe timings
(failureThreshold, timeoutSeconds, periodSeconds) remain configurable.
Remove any tcpSocket overrides from your values.yaml and change the bolt or
coordinator port via ports.boltPort / ports.coordinatorPort instead.
Debugging
There are different ways in which you can debug Memgraph’s HA cluster in
production. One way is to send us logs from all instances if you notice some
issue. That’s why we advise users to set the log level to TRACE if possible.
Note however that running TRACE log level has some performance costs,
especially when logging to stderr in addition to files. If performance is your
concern, first set commonArgs.data.logging.also_log_to_stderr and
commonArgs.coordinators.logging.also_log_to_stderr to false since logging
to files is cheaper. If you’re still unhappy with the performance overhead of
logging, set commonArgs.{data,coordinators}.logging.log_level to DEBUG
(higher log levels like INFO or CRITICAL are also fine) and keep
also_log_to_stderr: true. These settings replace the --log-level,
--also-log-to-stderr, --log-file and --log-retention-days flags that the
chart now appends to instance args automatically — setting them directly in
data[].args or coordinators[].args is rejected. Configure log retention via
commonArgs.{data,coordinators}.logging.log_retention_days (defaults to 35).
By default, the chart provisions a dedicated log PVC for every data and
coordinator pod. If you only log to stderr and don’t need a persistent log
volume, you can disable the log PVC by setting
storage.data.createLogStorageClaim and/or
storage.coordinators.createLogStorageClaim to false. When you do this you
must also set the corresponding commonArgs.{data,coordinators}.logging.log_file
to "" to disable file logging — installing the chart with file logging
enabled but no log volume is rejected.
If you notice your application is crashing, you will be able to collect core
dumps by setting storage.data.createCoreDumpsClaim and
storage.coordinators.createCoreDumpsClaim to true. That will trigger the
creation of an init container which will be run in privileged mode as the root user
to set up all the necessary things on your nodes to be able to collect core
dumps. You can then create the debug pod and attach the PVC containing core dumps to
that pod to be able to extract core dumps outside of the K8s nodes. The example
of such a debug pod is the following YAML file:
apiVersion: v1
kind: Pod
metadata:
name: debug-coredump
spec:
containers:
- name: debug
image: ubuntu:22.04
command: ["sleep", "infinity"]
volumeMounts:
- name: coredumps
mountPath: /var/core/memgraph
volumes:
- name: coredumps
persistentVolumeClaim:
claimName: memgraph-data-0-core-dumps-storage-memgraph-data-0-0
restartPolicy: NeverThere is also a possibility of automatically uploading core dumps to S3. To do that, set coreDumpUploader.enabled to true and configure the S3 bucket,
AWS region, and credentials secret in the coreDumpUploader section. Note that the createCoreDumpsClaim flag for the relevant role (data/coordinators)
must also be set to true, as the uploader sidecar mounts the same PVC used for core dump storage. Core dumps are uploaded to
s3://<s3BucketName>/<s3Prefix>/<pod-hostname>/<core-dump-filename>.
Graceful termination
When a pod is stopped (e.g., during upgrades, rescheduling, or scale-down),
Kubernetes sends SIGTERM and waits up to terminationGracePeriodSeconds
for the container to exit cleanly before forcefully killing it with SIGKILL.
The HA chart defaults this value to 30 seconds for both data
and coordinator pods, configurable via:
container.data.terminationGracePeriodSecondscontainer.coordinators.terminationGracePeriodSeconds
since --storage-snapshot-on-exit is explicitly set to false by default.
Using --storage-snapshot-on-exit with HA
If you enable the --storage-snapshot-on-exit
flag on data instances, Memgraph will attempt to create a full snapshot of the
database during shutdown. Snapshot creation time scales with dataset size and
can easily exceed the default grace period on larger deployments.
If terminationGracePeriodSeconds is shorter than the time needed to write the
on-exit snapshot, Kubernetes will SIGKILL the Memgraph process mid-write,
leaving the snapshot incomplete and defeating the purpose of the flag.
When enabling --storage-snapshot-on-exit, set
container.data.terminationGracePeriodSeconds to a value that comfortably
covers the expected snapshot duration for your dataset. Benchmark the snapshot
time on a representative dataset and add a safety margin.
Monitoring
Memgraph HA integrates with Kubernetes monitoring tools through:
- The kube-prometheus-stack Helm chart
- Memgraph’s Prometheus exporter
The chart kube-prometheus-stack should be installed independently from HA
chart with the following command:
helm install kube-prometheus-stack oci://ghcr.io/prometheus-community/charts/kube-prometheus-stack \
-f kube_prometheus_stack_values.yaml \
-f kube_prometheus_stack_memgraph_dashboard.yaml \
--namespace monitoring \
--create-namespacekube_prometheus_stack_values.yaml is optional. A template is available in the
upstream chart’s
repository.
kube_prometheus_stack_memgraph_dashboard.yaml is also optional - it provides a generic dashboard which shows the metrics
that Memgraph exports for both standalone and HA deployments. This dashboard file can be downloaded from
here.
If you install the kube-prometheus-stack in a non-default namespace, allow
cross-namespace scraping. You can allow this by adding the following
configuration to your kube_prometheus_stack_values.yaml file:
prometheus:
prometheusSpec:
serviceMonitorSelectorNilUsesHelmValues: falseEnable monitoring in the Memgraph HA chart
To enable the Memgraph Prometheus exporter and ServiceMonitor:
prometheus:
enabled: true
namespace: monitoring
memgraphExporter:
port: 9115
pullFrequencySeconds: 5
repository: memgraph/mg-exporter
tag: 0.2.3
serviceMonitor:
enabled: true
kubePrometheusStackReleaseName: kube-prometheus-stack
interval: 15sIf you set prometheus.enabled to false, resources from
charts/memgraph-high-availability/templates/mg-exporter.yaml will still be
installed into the monitoring namespace.
prometheus.serviceMonitor.enabled defaults to false; set it to true only
when you have kube-prometheus-stack (or another Prometheus Operator) in the
cluster to consume the ServiceMonitor resource.
Refer to the configuration table later in the document for details on all parameters.
mg-exporter TLS
When any data instance or coordinator has tls.bolt.enabled: true, the
chart automatically configures the mg-exporter to scrape that instance over
https:// instead of http://. Each instance entry in the exporter config
also gets skip_tls_verify and (optionally) ca_file derived from
prometheus.memgraphExporter.tls:
prometheus:
memgraphExporter:
tls:
skipVerify: true
caSecretName: ""
caSecretKey: ca.crtskipVerify— whentrue(default), the exporter does not verify the Memgraph server certificate. Convenient for self-signed certs but not suitable for production.caSecretName— name of a pre-created Secret holding the CA bundle that signed Memgraph’s certificate. When set andskipVerifyisfalse, the chart mounts the Secret at/etc/mg-exporter/ssland passesca_file=/etc/mg-exporter/ssl/<caSecretKey>to the exporter.caSecretKey— key inside the Secret holding the CA certificate (defaultca.crt).
Example with strict CA verification:
prometheus:
memgraphExporter:
tls:
skipVerify: false
caSecretName: bolt-ca-bundle
caSecretKey: ca.crtScrape Memgraph directly (without mg-exporter)
From Memgraph 3.11, Memgraph can serve metrics in Prometheus
OpenMetrics format on its
HTTP monitoring port, so the cluster can be scraped directly and mg-exporter can be
skipped entirely.
Set the top-level scrapeMemgraphDirectly: true and leave prometheus.enabled: false:
# Every coordinator and data instance serves OpenMetrics; the chart exposes their
# monitoring ports automatically.
scrapeMemgraphDirectly: true
prometheus:
enabled: false # no mg-exporter is deployed
serviceMonitor:
enabled: true # in-cluster: kube-prometheus-stack scrapes each instance directlyscrapeMemgraphDirectly only selects the metrics format and target — on its own it
does not scrape anything. You still enable a scraper:
- In-cluster:
prometheus.serviceMonitor.enabled: trueprovisions aServiceMonitorthatkube-prometheus-stackuses to scrape each Memgraph instance directly. - Remote:
vmagentRemote.enabled: truemakes vmagent scrape the instances directly and remote-write the metrics (see Remote metrics and logs).
The same switch governs both paths. When it is false, those scrapers read the
mg-exporter instead, which requires prometheus.enabled: true.
Direct scraping requires Memgraph >= 3.11. When scrapeMemgraphDirectly: true, the
chart runs every coordinator and data instance with --metrics-format=OpenMetrics and
exposes their monitoring ports automatically. The metric names differ from the legacy
JSON exporter, so use the bundled Memgraph OpenMetrics dashboard.
Grafana dashboard for direct scraping
Set prometheus.grafanaDashboard.enabled: true to ship the bundled “Memgraph
OpenMetrics” Grafana dashboard as a ConfigMap (labelled grafana_dashboard) for a
Grafana sidecar (e.g. the one bundled with kube-prometheus-stack) to auto-load. The
dashboard uses a datasource template variable, so it binds to Grafana’s default
Prometheus. The ConfigMap must live in a namespace the sidecar watches — set
prometheus.grafanaDashboard.namespace to Grafana’s namespace (or run the sidecar with
searchNamespace: ALL).
prometheus:
grafanaDashboard:
enabled: true
namespace: monitoring
label: grafana_dashboardUninstall kube-prometheus-stack
helm uninstall kube-prometheus-stack --namespace monitoringNote: The stack’s CRDs are not deleted automatically and must be removed manually:
kubectl delete crd alertmanagerconfigs.monitoring.coreos.com
kubectl delete crd alertmanagers.monitoring.coreos.com
kubectl delete crd podmonitors.monitoring.coreos.com
kubectl delete crd probes.monitoring.coreos.com
kubectl delete crd prometheusagents.monitoring.coreos.com
kubectl delete crd prometheuses.monitoring.coreos.com
kubectl delete crd prometheusrules.monitoring.coreos.com
kubectl delete crd scrapeconfigs.monitoring.coreos.com
kubectl delete crd servicemonitors.monitoring.coreos.com
kubectl delete crd thanosrulers.monitoring.coreos.comRemote metrics and logs
The HA chart supports optional remote observability:
vmagentRemotefor shipping metrics with Prometheusremote_writevectorRemotesidecars for shipping Memgraph logs to Loki-compatible endpoints
Prerequisites:
- keep
prometheus.enabled: truesomg-exporteris deployed — or setscrapeMemgraphDirectly: truewithprometheus.enabled: falseto have vmagent scrape Memgraph’s OpenMetrics endpoint directly (no exporter; see Scrape Memgraph directly) - if you only need remote shipping and not local scraping, set
prometheus.serviceMonitor.enabled: falseto avoid duplicate scraping - configure
vectorRemote.dataand/orvectorRemote.coordinatorsdepending on which pod roles should ship logs - when
vectorRemote.enabled: true, add--monitoring-port=<vectorRemote.websocketPort>and--monitoring-address=0.0.0.0to each instanceargs
Example values.yaml:
prometheus:
enabled: true
namespace: monitoring
serviceMonitor:
enabled: false
vmagentRemote:
enabled: true
namespace: monitoring
remoteWrite:
url: "https://<prom-remote-write>/api/v1/write"
# Optional: only set basicAuth when your remote_write endpoint requires basic auth.
basicAuth:
secretName: monitoring-basic-auth
usernameKey: username
passwordKey: password
externalLabels:
cluster_id: "memgraph-testing-cluster-53"
service_name: "memgraph-ha"
cluster_env: "self-hosted-large-01"
vectorRemote:
enabled: true
data: true
coordinators: true
websocketPort: 7444
logsEndpoint: "https://<loki-endpoint>"
# Optional: only set auth when your endpoint requires basic auth.
auth:
secretName: monitoring-basic-auth
usernameKey: username
passwordKey: password
extraLabels:
cluster_id: "memgraph-testing-cluster-53"
service_name: "memgraph-ha"
cluster_env: "self-hosted-large-01"
data:
- id: "0"
args:
- "--monitoring-port=7444"
- "--monitoring-address=0.0.0.0"
- id: "1"
args:
- "--monitoring-port=7444"
- "--monitoring-address=0.0.0.0"
coordinators:
- id: "1"
args:
- "--monitoring-port=7444"
- "--monitoring-address=0.0.0.0"
- id: "2"
args:
- "--monitoring-port=7444"
- "--monitoring-address=0.0.0.0"
- id: "3"
args:
- "--monitoring-port=7444"
- "--monitoring-address=0.0.0.0"The chart auto-appends --bolt-port, --management-port, --coordinator-port,
--coordinator-id, --coordinator-hostname, --data-directory, --log-level,
--also-log-to-stderr, --log-file, --bolt-cert-file, --bolt-key-file,
--cluster-cert-file, --cluster-key-file and --cluster-ca-file
from ports.*, commonArgs.{data,coordinators}.logging.* and the per-instance
tls.bolt.* / tls.intraCluster.* blocks. Setting any of these in data[].args or
coordinators[].args causes helm install to fail with a template error.
Create credentials secret in the namespace where vmagent runs (usually monitoring):
kubectl create secret generic monitoring-basic-auth -n monitoring \
--from-literal=username='<username>' \
--from-literal=password='<password>'For HA Vector sidecars, create the same secret in the Memgraph release namespace as well:
kubectl create secret generic monitoring-basic-auth -n <memgraph-namespace> \
--from-literal=username='<username>' \
--from-literal=password='<password>'Kubernetes infrastructure metrics
vmagentRemote can additionally scrape Kubernetes infrastructure metrics
(kube-state-metrics, node-exporter, kubelet) required by
kube-prometheus-stack Kubernetes and Node dashboards, and remote-write them
to your centralized monitoring cluster.
Enable Kubernetes scraping by extending your existing vmagentRemote values:
vmagentRemote:
# ... existing fields (enabled, remoteWrite, externalLabels) ...
kubernetes:
enabled: true
kubeStateMetrics:
enabled: true
jobName: kube-state-metrics
targets:
- kube-prometheus-stack-kube-state-metrics.monitoring.svc.cluster.local:8080
nodeExporter:
enabled: true
jobName: node-exporter
targets:
- kube-prometheus-stack-prometheus-node-exporter.monitoring.svc.cluster.local:9100
kubelet:
enabled: true
jobName: kubelet
metricsPath: /metrics/cadvisor
apiServerAddress: kubernetes.default.svc:443
insecureSkipVerify: falseNotes:
- RBAC and
ServiceAccountresources are created only when an enabled scrape job requires Kubernetes API access (for examplekubelet.enabled=trueornodeExporter.useKubernetesDiscovery=true). - Keep
jobNamevalues aligned with dashboard and recording-rule expectations unless you also update those queries. - Dashboards that rely on precomputed recording-rule series still require rule evaluation in your monitoring stack.
A ready-to-use example values file is available in the Helm charts repository:
examples/remote-monitoring/values-ha-k8s-metrics.yaml.
Configuration options
The following table lists the configurable parameters of the Memgraph HA chart and their default values.
| Parameter | Description | Default |
|---|---|---|
image.repository | Memgraph Docker image repository | docker.io/memgraph/memgraph |
image.tag | Specific tag for the Memgraph Docker image. Overrides the image tag whose default is chart version. | 3.1.0 |
image.pullPolicy | Image pull policy | IfNotPresent |
memgraphUserId | The user id that is hardcoded in Memgraph and Mage images | 101 |
memgraphGroupId | The group id that is hardcoded in Memgraph and Mage images | 103 |
clusterDomain | Kubernetes cluster domain used to build the internal FQDN service addresses (<svc>.<namespace>.svc.<clusterDomain>) of data instances and coordinators. Override if your cluster uses a custom domain. | cluster.local |
storage.data.libPVCSize | Size of the lib storage PVC for data instances | 1Gi |
storage.data.libStorageAccessMode | Access mode used for lib storage on data instances | ReadWriteOnce |
storage.data.libStorageClassName | The name of the storage class used for storing data on data instances | "" |
storage.data.createLogStorageClaim | Create a PVC for logs on data instances. When false, commonArgs.data.logging.log_file must be "". | true |
storage.data.logPVCSize | Size of the log PVC for data instances | 1Gi |
storage.data.logStorageAccessMode | Access mode used for log storage on data instances | ReadWriteOnce |
storage.data.logStorageClassName | The name of the storage class used for storing logs on data instances | "" |
storage.data.createCoreDumpsClaim | Create a PVC for core dumps on data instances | false |
storage.data.coreDumpsStorageClassName | Storage class name for core dumps PVC on data instances | "" |
storage.data.coreDumpsStorageSize | Size of the core dumps PVC on data instances | 10Gi |
storage.data.coreDumpsMountPath | Mount path for core dumps on data instances | /var/core/memgraph |
storage.data.coreDumpsImage.repository | Image repository for the data instance core-dumps init container. | docker.io/library/busybox |
storage.data.coreDumpsImage.tag | Image tag for the data instance core-dumps init container. | 1.37.0 |
storage.data.coreDumpsImage.pullPolicy | Image pull policy for the data instance core-dumps init container. | IfNotPresent |
storage.data.extraVolumes | Additional volumes to add to data instance pods | [] |
storage.data.extraVolumeMounts | Additional volume mounts to add to data instance containers | [] |
storage.coordinators.libPVCSize | Size of the lib storage PVC for coordinators | 1Gi |
storage.coordinators.libStorageAccessMode | Access mode used for lib storage on coordinators | ReadWriteOnce |
storage.coordinators.libStorageClassName | The name of the storage class used for storing data on coordinators | "" |
storage.coordinators.createLogStorageClaim | Create a PVC for logs on coordinators. When false, commonArgs.coordinators.logging.log_file must be "". | true |
storage.coordinators.logPVCSize | Size of the log PVC for coordinators | 1Gi |
storage.coordinators.logStorageAccessMode | Access mode used for log storage on coordinators | ReadWriteOnce |
storage.coordinators.logStorageClassName | The name of the storage class used for storing logs on coordinators | "" |
storage.coordinators.createCoreDumpsClaim | Create a PVC for core dumps on coordinators | false |
storage.coordinators.coreDumpsStorageClassName | Storage class name for core dumps PVC on coordinators | "" |
storage.coordinators.coreDumpsStorageSize | Size of the core dumps PVC on coordinators | 10Gi |
storage.coordinators.coreDumpsMountPath | Mount path for core dumps on coordinators | /var/core/memgraph |
storage.coordinators.coreDumpsImage.repository | Image repository for the coordinator core-dumps init container. | docker.io/library/busybox |
storage.coordinators.coreDumpsImage.tag | Image tag for the coordinator core-dumps init container. | 1.37.0 |
storage.coordinators.coreDumpsImage.pullPolicy | Image pull policy for the coordinator core-dumps init container. | IfNotPresent |
storage.coordinators.extraVolumes | Additional volumes to add to coordinator pods | [] |
storage.coordinators.extraVolumeMounts | Additional volume mounts to add to coordinator containers | [] |
externalAccessConfig.coordinator.serviceType | IngressNginx, NodePort, CommonLoadBalancer or LoadBalancer. By default, no external service will be created. | "" |
externalAccessConfig.coordinator.annotations | Annotations for external services attached to coordinators. | {} |
externalAccessConfig.dataInstance.serviceType | IngressNginx, NodePort or LoadBalancer. By default, no external service will be created. | "" |
externalAccessConfig.dataInstance.annotations | Annotations for external services attached to data instances. | {} |
externalAccessConfig.ingress.dataPortBase | Base port for data instance ports on the IngressNginx controller (dataPortBase + data instance id). | 9000 |
externalAccessConfig.ingress.coordinatorPortBase | Base port for coordinator ports on the IngressNginx controller (coordinatorPortBase + coordinator id). Kept well above dataPortBase so the data and coordinator port ranges never overlap. | 10000 |
externalAccessConfig.gateway.enabled | Enable Gateway API external access. Only exposes tiers whose serviceType is empty; fails rendering if both tiers have a serviceType set. | false |
externalAccessConfig.gateway.gatewayClassName | Name of a pre-existing GatewayClass. Required when creating a new Gateway. | "" |
externalAccessConfig.gateway.existingGatewayName | Name of an existing Gateway to attach routes to. Skips Gateway creation. | "" |
externalAccessConfig.gateway.existingGatewayNamespace | Namespace of the existing Gateway. Defaults to release namespace. | "" |
externalAccessConfig.gateway.annotations | Annotations for the Gateway resource. | {} |
externalAccessConfig.gateway.labels | Labels for the Gateway resource. | {} |
externalAccessConfig.gateway.dataPortBase | Base port for data instance Gateway listeners (dataPortBase + data instance id). | 9000 |
externalAccessConfig.gateway.coordinatorPortBase | Base port for coordinator Gateway listeners (coordinatorPortBase + coordinator id). Kept well above dataPortBase so the data and coordinator port ranges never overlap. | 10000 |
headlessService.enabled | Specifies whether headless services will be used inside K8s network on all instances. | false |
ports.boltPort | Bolt port used on coordinator and data instances. | 7687 |
ports.managementPort | Management port used on coordinator and data instances. | 10000 |
ports.replicationPort | Replication port used on data instances. | 20000 |
ports.coordinatorPort | Coordinator port used on coordinators. | 12000 |
ports.metricsPort | Metrics port for coordinators and data instances. Opened only if prometheus.enabled is set to true. | 9091 |
affinity.unique | Schedule pods on different nodes in the cluster | false |
affinity.parity | Schedule pods on the same node with maximum one coordinator and one data node | false |
affinity.nodeSelection | Schedule pods on nodes with specific labels | false |
affinity.roleLabelKey | Label key for node selection | role |
affinity.dataNodeLabelValue | Label value for data nodes | data-node |
affinity.coordinatorNodeLabelValue | Label value for coordinator nodes | coordinator-node |
container.data.terminationGracePeriodSeconds | Grace period for data instance pod termination | 1800 |
container.data.livenessProbe.failureThreshold | Failure threshold for liveness probe | 20 |
container.data.livenessProbe.timeoutSeconds | Timeout for liveness probe | 10 |
container.data.livenessProbe.periodSeconds | Period seconds for liveness probe | 5 |
container.data.readinessProbe.failureThreshold | Failure threshold for readiness probe | 20 |
container.data.readinessProbe.timeoutSeconds | Timeout for readiness probe | 10 |
container.data.readinessProbe.periodSeconds | Period seconds for readiness probe | 5 |
container.data.startupProbe.failureThreshold | Failure threshold for startup probe | 1440 |
container.data.startupProbe.timeoutSeconds | Timeout for probe | 10 |
container.data.startupProbe.periodSeconds | Period seconds for startup probe | 10 |
container.data.terminationGracePeriodSeconds | Grace period for data pod termination. Increase when --storage-snapshot-on-exit is enabled so the snapshot has time to finish. | 30 |
container.coordinators.livenessProbe.failureThreshold | Failure threshold for liveness probe | 20 |
container.coordinators.livenessProbe.timeoutSeconds | Timeout for liveness probe | 10 |
container.coordinators.livenessProbe.periodSeconds | Period seconds for liveness probe | 5 |
container.coordinators.readinessProbe.failureThreshold | Failure threshold for readiness probe | 20 |
container.coordinators.readinessProbe.timeoutSeconds | Timeout for readiness probe | 10 |
container.coordinators.readinessProbe.periodSeconds | Period seconds for readiness probe | 5 |
container.coordinators.startupProbe.failureThreshold | Failure threshold for startup probe | 20 |
container.coordinators.startupProbe.timeoutSeconds | Timeout for probe | 10 |
container.coordinators.startupProbe.periodSeconds | Period seconds for startup probe | 10 |
container.coordinators.terminationGracePeriodSeconds | Grace period for coordinators pod termination. | 30. |
data | Configuration for data instances | See data section |
coordinators | Configuration for coordinator instances | See coordinators section |
sysctlInitContainer.enabled | Enable the init container to set sysctl parameters | true |
sysctlInitContainer.maxMapCount | Value for vm.max_map_count to be set by the init container | 524288 |
sysctlInitContainer.image.repository | Image repository for the sysctl init container | library/busybox |
sysctlInitContainer.image.tag | Image tag for the sysctl init container | 1.37.0 |
sysctlInitContainer.image.pullPolicy | Image pull policy for the sysctl init container | IfNotPresent |
fixOwnershipInitContainer.enabled | Enable the init container that chowns lib/log/core-dump mounts to memgraphUserId:memgraphGroupId before Memgraph starts. Use when the storage driver does not honor fsGroup. | false |
fixOwnershipInitContainer.image.repository | Image repository for the fix-ownership init container. | docker.io/library/busybox |
fixOwnershipInitContainer.image.tag | Image tag for the fix-ownership init container. | 1.37.0 |
fixOwnershipInitContainer.image.pullPolicy | Image pull policy for the fix-ownership init container. | IfNotPresent |
secrets.name | Name of the Kubernetes Secret holding the Memgraph Enterprise license and organization name. Must exist before helm install. | memgraph-secrets |
secrets.licenseKey | Key in the Secret whose value is exposed as MEMGRAPH_ENTERPRISE_LICENSE to data and coordinator pods. | MEMGRAPH_ENTERPRISE_LICENSE |
secrets.organizationKey | Key in the Secret whose value is exposed as MEMGRAPH_ORGANIZATION_NAME to data and coordinator pods. | MEMGRAPH_ORGANIZATION_NAME |
resources.coordinators | CPU/Memory resource requests/limits for coordinators. Left empty by default. | {} |
resources.data | CPU/Memory resource requests/limits for data instances. Left empty by default. | {} |
scrapeMemgraphDirectly | Serve metrics in OpenMetrics format and scrape Memgraph directly instead of mg-exporter (requires Memgraph >= 3.11). Only selects the format/target — still enable a scraper via prometheus.serviceMonitor.enabled and/or vmagentRemote.enabled. | false |
prometheus.enabled | If set to true, K8s resources representing Memgraph’s Prometheus exporter will be deployed. | false |
prometheus.namespace | Namespace in which kube-prometheus-stack and Memgraph’s Prometheus exporter are installed. When empty, the release namespace is used. | "" |
prometheus.memgraphExporter.port | The port on which Memgraph’s Prometheus exporter is available. | 9115 |
prometheus.memgraphExporter.pullFrequencySeconds | How often will Memgraph’s Prometheus exporter pull data from Memgraph instances. | 5 |
prometheus.memgraphExporter.repository | The repository where Memgraph’s Prometheus exporter image is available. | docker.io/memgraph/prometheus-exporter |
prometheus.memgraphExporter.tag | The tag of Memgraph’s Prometheus exporter image. | 0.2.3 |
prometheus.memgraphExporter.tls.skipVerify | When true, mg-exporter does not verify Memgraph’s server certificate. Only applied when scraping instances with tls.bolt.enabled=true. | true |
prometheus.memgraphExporter.tls.caSecretName | Name of a pre-created Secret containing the CA bundle. When set (and skipVerify=false), the chart mounts it at /etc/mg-exporter/ssl. | "" |
prometheus.memgraphExporter.tls.caSecretKey | Key inside the Secret holding the CA certificate. | ca.crt |
prometheus.memgraphExporter.extraVolumes | Additional volumes mounted on the mg-exporter Deployment (e.g. ConfigMaps with custom exporter configs). | [] |
prometheus.memgraphExporter.extraVolumeMounts | Additional volume mounts for the mg-exporter container. | [] |
prometheus.serviceMonitor.enabled | If enabled, a ServiceMonitor object will be deployed. | false |
prometheus.serviceMonitor.kubePrometheusStackReleaseName | The release name under which kube-prometheus-stack chart is installed. | kube-prometheus-stack |
prometheus.serviceMonitor.interval | How often will Prometheus pull data from Memgraph’s Prometheus exporter. | 15s |
prometheus.grafanaDashboard.enabled | Ship the bundled “Memgraph OpenMetrics” Grafana dashboard as a ConfigMap for a Grafana sidecar to auto-load. | false |
prometheus.grafanaDashboard.namespace | Namespace for the dashboard ConfigMap; must be one the Grafana sidecar watches. Defaults to prometheus.namespace, else release namespace. | "" |
prometheus.grafanaDashboard.label | Label the Grafana sidecar selects dashboards by. | grafana_dashboard |
vmagentRemote.enabled | Deploy a vmagent Deployment that scrapes mg-exporter (or Memgraph directly when scrapeMemgraphDirectly=true) and remote-writes to a Prometheus-compatible endpoint. | false |
vmagentRemote.namespace | Namespace for the vmagent Deployment and its resources. Defaults to prometheus.namespace when empty. | "" |
vmagentRemote.image.repository | vmagent image repository. | victoriametrics/vmagent |
vmagentRemote.image.tag | vmagent image tag. | v1.139.0 |
vmagentRemote.image.pullPolicy | vmagent image pull policy. | IfNotPresent |
vmagentRemote.remoteWrite.url | Prometheus remote_write URL. Required when vmagentRemote.enabled=true. | "" |
vmagentRemote.remoteWrite.basicAuth.secretName | Kubernetes Secret holding basic-auth credentials for remote_write. When empty, basic auth is not configured. | "" |
vmagentRemote.remoteWrite.basicAuth.usernameKey | Key in the basic-auth Secret holding the username. | username |
vmagentRemote.remoteWrite.basicAuth.passwordKey | Key in the basic-auth Secret holding the password. | password |
vmagentRemote.scrapeInterval | Global scrape_interval applied to vmagent scrape jobs. | 15s |
vmagentRemote.externalLabels | External labels attached to every scraped sample before remote-write. | {} |
vmagentRemote.resources | Resource requests/limits for the vmagent container. | {} |
vmagentRemote.httpPort | vmagent local HTTP listen port for metrics/debug (the remote-write target is remoteWrite.url). | 8429 |
vmagentRemote.kubernetes.enabled | Enable scraping of Kubernetes infrastructure metrics used by kube-prometheus dashboards. | false |
vmagentRemote.kubernetes.kubeStateMetrics.enabled | Scrape kube-state-metrics. | true |
vmagentRemote.kubernetes.kubeStateMetrics.jobName | Prometheus job label for kube-state-metrics. Keep aligned with dashboard/recording-rule expectations. | kube-state-metrics |
vmagentRemote.kubernetes.kubeStateMetrics.targets | Static scrape targets for kube-state-metrics. | [kube-prometheus-stack-kube-state-metrics.monitoring.svc.cluster.local:8080] |
vmagentRemote.kubernetes.nodeExporter.enabled | Scrape node-exporter. | true |
vmagentRemote.kubernetes.nodeExporter.jobName | Prometheus job label for node-exporter. | node-exporter |
vmagentRemote.kubernetes.nodeExporter.useKubernetesDiscovery | Discover node-exporter pods via Kubernetes SD so namespace/pod/node labels are present for recording rules. | false |
vmagentRemote.kubernetes.nodeExporter.podMetricsPort | Pod port used by Kubernetes SD to match node-exporter pods. | "9100" |
vmagentRemote.kubernetes.nodeExporter.appNameLabel | Expected value of app.kubernetes.io/name on node-exporter pods. | prometheus-node-exporter |
vmagentRemote.kubernetes.nodeExporter.appInstanceLabel | Expected value of app.kubernetes.io/instance on node-exporter pods. | kube-prometheus-stack-prometheus-node-exporter |
vmagentRemote.kubernetes.nodeExporter.targets | Static fallback targets for node-exporter when useKubernetesDiscovery=false. | [kube-prometheus-stack-prometheus-node-exporter.monitoring.svc.cluster.local:9100] |
vmagentRemote.kubernetes.kubelet.enabled | Scrape kubelet metrics via the Kubernetes API server node proxy. | true |
vmagentRemote.kubernetes.kubelet.jobName | Prometheus job label for kubelet. Keep as kubelet so kube-prometheus dashboards and rules still match. | kubelet |
vmagentRemote.kubernetes.kubelet.metricsPath | Metrics path for the primary kubelet scrape (cAdvisor). | /metrics/cadvisor |
vmagentRemote.kubernetes.kubelet.additionalMetricsEnabled | Enable a second kubelet scrape job for /metrics alongside the cAdvisor job. | true |
vmagentRemote.kubernetes.kubelet.additionalJobName | Prometheus job label for the additional kubelet scrape. | kubelet-metrics |
vmagentRemote.kubernetes.kubelet.additionalMetricsPath | Metrics path for the additional kubelet scrape. | /metrics |
vmagentRemote.kubernetes.kubelet.apiServerAddress | Kubernetes API server address used to proxy kubelet scrapes. | kubernetes.default.svc:443 |
vmagentRemote.kubernetes.kubelet.insecureSkipVerify | Skip TLS verification of the kube-apiserver serving cert when scraping kubelet. | false |
labels.coordinators.podLabels | Enables you to set labels on a pod level for coordinators. | {} |
labels.coordinators.statefulSetLabels | Enables you to set labels on a stateful set level for coordinators. | {} |
labels.coordinators.serviceLabels | Enables you to set labels on a service level for coordinators. | {} |
labels.data.podLabels | Enables you to set labels on a pod level for data instances. | {} |
labels.data.statefulSetLabels | Enables you to set labels on a stateful set level for data instances. | {} |
labels.data.serviceLabels | Enables you to set labels on a service level for data instances. | {} |
clusterSetup.labels | Custom labels applied to both the post-install cluster-setup Job and its pod template. | {} |
clusterSetup.podSecurityContext | Pod-level security context for the cluster-setup Job pod. See Customize the cluster-setup Job. | runAsUser: 101, runAsGroup: 103, runAsNonRoot: true, seccompProfile.type: RuntimeDefault |
clusterSetup.containerSecurityContext | Container-level security context for the cluster-setup container. See Customize the cluster-setup Job. | allowPrivilegeEscalation: false, capabilities.drop: ["ALL"], readOnlyRootFilesystem: true, runAsNonRoot: true, seccompProfile.type: RuntimeDefault |
updateStrategy.type | Update strategy for StatefulSets. Possible values are RollingUpdate and OnDelete | RollingUpdate |
extraEnv.data | Env variables that users can define and are applied to data instances | [] |
extraEnv.coordinators | Env variables that users can define and are applied to coordinators | [] |
commonArgs.data.logging.log_level | Log level applied to every data instance via --log-level. Must not be empty. | TRACE |
commonArgs.data.logging.also_log_to_stderr | When true, appends --also-log-to-stderr to every data instance. Must be a boolean. | true |
commonArgs.data.logging.log_file | Log-file path applied to every data instance via --log-file. Empty disables file logging. | /var/log/memgraph/memgraph.log |
commonArgs.data.logging.log_retention_days | Number of days to retain log files on every data instance via --log-retention-days. | 35 |
commonArgs.coordinators.logging.log_level | Log level applied to every coordinator via --log-level. Must not be empty. | TRACE |
commonArgs.coordinators.logging.also_log_to_stderr | When true, appends --also-log-to-stderr to every coordinator. Must be a boolean. | true |
commonArgs.coordinators.logging.log_file | Log-file path applied to every coordinator via --log-file. Empty disables file logging. | /var/log/memgraph/memgraph.log |
commonArgs.coordinators.logging.log_retention_days | Number of days to retain log files on every coordinator via --log-retention-days. | 35 |
userContainers.data | Additional sidecar containers for data instance pods | [] |
userContainers.coordinators | Additional sidecar containers for coordinator pods | [] |
tolerations.data | Tolerations for data instance pods | [] |
tolerations.coordinators | Tolerations for coordinator pods | [] |
initContainers.data | Init containers that users can define that will be applied to data instances. | [] |
initContainers.coordinators | Init containers that users can define that will be applied to coordinators. | [] |
coreDumpUploader.enabled | Enable the core dump S3 uploader sidecar. Requires storage.<role>.createCoreDumpsClaim to be true. | false |
coreDumpUploader.image.repository | Docker image repository for the uploader sidecar | amazon/aws-cli |
coreDumpUploader.image.tag | Docker image tag for the uploader sidecar | 2.33.28 |
coreDumpUploader.image.pullPolicy | Image pull policy for the uploader sidecar | IfNotPresent |
coreDumpUploader.s3BucketName | S3 bucket name where core dumps will be uploaded | "" |
coreDumpUploader.s3Prefix | S3 key prefix (folder) for uploaded core dumps | core-dumps |
coreDumpUploader.awsRegion | AWS region of the S3 bucket | us-east-1 |
coreDumpUploader.pollIntervalSeconds | How often (in seconds) the sidecar checks for new core dump files | 30 |
coreDumpUploader.secretName | Name of the K8s Secret containing AWS credentials | aws-s3-credentials |
coreDumpUploader.accessKeySecretKey | Key in the K8s Secret for AWS_ACCESS_KEY_ID | AWS_ACCESS_KEY_ID |
coreDumpUploader.secretAccessKeySecretKey | Key in the K8s Secret for AWS_SECRET_ACCESS_KEY | AWS_SECRET_ACCESS_KEY |
For the data and coordinators sections, each item in the list has the
following parameters:
| Parameter | Description | Default |
|---|---|---|
id | ID of the instance | 0 for data, 1 for coordinators |
internalAccessAnnotations | Per-instance annotations for the internal ClusterIP Service. | {} |
externalAccessAnnotations | Per-instance annotations for the external access Service, merged with global annotations. | {} |
tls.bolt.enabled | Enable Bolt TLS termination on this instance. The chart auto-appends --bolt-cert-file / --bolt-key-file and mounts the certificate Secret at /etc/memgraph/ssl. | false |
tls.bolt.secretName | Name of a pre-existing Kubernetes Secret holding the Bolt TLS certificate and private key. Required when tls.bolt.enabled=true. | bolt-tls-secret |
tls.bolt.certSecretPath | Key inside the Secret holding the TLS certificate. | tls.crt |
tls.bolt.keySecretPath | Key inside the Secret holding the TLS private key. | tls.key |
tls.intraCluster.enabled | Enable TLS on internal cluster communication for this instance. The chart auto-appends --cluster-cert-file / --cluster-key-file / --cluster-ca-file and mounts the certificate Secret at /etc/memgraph/intra_cluster_tls. | false |
tls.intraCluster.secretName | Name of a pre-existing Kubernetes Secret holding the intra-cluster TLS certificate, private key and CA bundle. Required when tls.intraCluster.enabled=true. | intra-tls-<instance>-secret |
tls.intraCluster.certSecretPath | Key inside the Secret holding the TLS certificate. | tls.crt |
tls.intraCluster.keySecretPath | Key inside the Secret holding the TLS private key. | tls.key |
tls.intraCluster.caSecretPath | Key inside the Secret holding the CA bundle. | ca.crt |
args | Per-instance Memgraph CLI flags. Append-only — see the note below for flags the chart manages. | ["--storage-snapshot-on-exit=false"] for data, [] for coordinators |
The args field accepts any Memgraph CLI flag except the following, which
the chart appends automatically and rejects when set per-instance:
--bolt-port, --management-port, --coordinator-port, --coordinator-id,
--coordinator-hostname, --data-directory, --log-level,
--also-log-to-stderr, --log-file, --bolt-cert-file, --bolt-key-file,
--cluster-cert-file, --cluster-key-file and --cluster-ca-file.
Configure those through ports.*, commonArgs.{data,coordinators}.logging.*
and the per-instance tls.bolt.* / tls.intraCluster.* blocks instead.
For all available database settings, refer to the configuration settings docs.
In-Service Software Upgrade (ISSU)
Memgraph’s High Availability supports in-service software upgrades (ISSU). This guide explains the process when using HA Helm charts. The procedure is very similar for native deployments.
Some Memgraph versions require additional upgrade steps beyond the standard ISSU procedure. Check the Migrating to v3.9 HA page for version-specific instructions before proceeding.
Important: Although the upgrade process is designed to complete
successfully, unexpected issues may occur. We strongly recommend doing a backup
of your lib directory on all of your StatefulSets or native instances
depending on the deployment type.
Prerequisites
If you are using HA Helm charts, set the following configuration before doing any upgrade.
updateStrategy.type: OnDeleteDepending on the infrastructure on which you have your Memgraph cluster, the details will differ a bit, but the backbone is the same.
Prepare a backup of all data from all instances. This ensures you can safely downgrade cluster to the last stable version you had.
-
For native deployments, tools like
cporrsyncare sufficient. -
For Kubernetes, create a
VolumeSnapshotClasswith the yaml file fimilar to this:apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshotClass metadata: name: csi-azure-disk-snapclass driver: disk.csi.azure.com deletionPolicy: DeleteApply it:
kubectl apply -f azure_class.yaml- On Google Kubernetes Engine, the default CSI driver is
pd.csi.storage.gke.ioso make sure to change the fielddriver. - On AWS EKS, refer to the AWS snapshot controller docs.
- On Google Kubernetes Engine, the default CSI driver is
Create snapshots
Now you can create a VolumeSnapshot of the lib directory using the yaml file:
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
name: coord-3-snap # Use a unique name for each instance
namespace: default
spec:
volumeSnapshotClassName: csi-azure-disk-snapclass
source:
persistentVolumeClaimName: memgraph-coordinator-3-lib-storage-memgraph-coordinator-3-0Apply it:
kubectl apply -f azure_snapshot.yamlRepeat for every instance in the cluster.
Update configuration
Next you should update image.tag field in the values.yaml configuration file
to the version to which you want to upgrade your cluster.
-
In your
values.yaml, update the image version:image: tag: <new_version> -
Apply the upgrade:
helm upgrade <release> <chart> -f <path_to_values.yaml>
Since we are using updateStrategy.type=OnDelete, this step will not restart
any pod, rather it will just prepare pods for running the new version.
- For native deployments, ensure the new binary is available.
Upgrade procedure (zero downtime)
Our procedure for achieving zero-downtime upgrades consists of restarting one instance at a time. Memgraph uses primary–secondary replication. To avoid downtime:
- Upgrade replicas first.
- Upgrade the main instance.
- Upgrade coordinator followers
- Upgrade the coordinator leader.
In order to find out on which pod/server the current main and the current cluster leader sits, run:
SHOW INSTANCES;Query behavior during the upgrade
Your applications can stay connected to the cluster throughout the upgrade. What they will observe depends on the query type and the replication mode:
-
Read queries always work, regardless of the replication mode used. There is no specific exception you need to catch — Neo4j drivers are preconfigured with automatic retries, so reads hitting an instance that is restarting are transparently retried.
One exception applies to minimal clusters: with 1 main and 1 replica, you need to enable the
enabled_reads_on_maincoordinator setting so that, while the only replica is being upgraded, read queries are routed to the main instance:SET COORDINATOR SETTING 'enabled_reads_on_main' TO 'true'; -
Write queries depend on the replication mode:
- ASYNC — writes keep working without errors.
- SYNC — writes keep working, but a ReplicationException can happen
because a commit couldn’t be replicated to the instance that is being
restarted (e.g. the old main while it is coming back up). The transaction
is still committed on the MAIN, so your application should catch the
exception and continue. You can parse the error and check for the exact
message, for example:
Failed to replicate to SYNC replica 'instance_1': replica is not reachable or not in sync with the main. - STRICT_SYNC — writes won’t work while a data instance is being restarted. Because of the two-phase commit protocol, a transaction cannot be committed unless every STRICT_SYNC replica confirms it, so writes are aborted until the restarted instance rejoins the cluster.
If these write-side effects are not acceptable for your workload, you can pause
writes across the whole cluster for the duration of the upgrade. The
recommended way is the
global_read_only
coordinator setting. Connect to the coordinator leader and run:
SET COORDINATOR SETTING 'global_read_only' TO 'true';When enabled, the current MAIN stops accepting write queries while it keeps serving reads and replicating existing data to REPLICAs. The setting blocks all write sources on the MAIN — user Cypher writes, TTL background expiry, and stream- and trigger-driven writes — so the dataset stays frozen while replicas catch up and you swap binaries. The value is persisted in the coordinator’s Raft-replicated cluster state, so it survives coordinator restarts and leader re-elections, and is honored across failovers: a newly promoted MAIN comes up read-only instead of silently accepting writes.
While the cluster runs mixed versions during the rolling upgrade, read-only mode is honored best-effort: older instances behave as before until every node is upgraded, after which the cluster self-heals to the requested state.
Upgrade replicas
Before restarting any replica, check whether replicas are caught up with the MAIN:
SHOW REPLICATION LAG;If the replica you are about to restart is lagging behind, wait until it catches up before deleting its pod.
If you are using K8s, the upgrade can be performed by deleting the pod. Start by
deleting the replica pod (in this example replica is running on the pod
memgraph-data-1-0):
kubectl delete pod memgraph-data-1-0Native deployment: stop the old binary and start the new one.
Before starting the upgrade of the next pod, it is important to wait until all pods are ready. Otherwise, you may end up with a data loss. On K8s you can easily achieve that by running:
kubectl wait --for=condition=ready pod --allFor the native deployment, check if all your instances are alived manually.
This step should be repeated for all of your replicas in the cluster.
Upgrade the main
When the main pod is deleted, the coordinator detects that the main is down and
performs an automatic failover, promoting one of the already upgraded
replicas to be the new main. Until the failover completes, the cluster does not
accept write queries. The length of this time window depends on how quickly the
failover happens, which is governed by two coordinator settings: the
coordinator pings data instances every
instance_health_check_frequency_sec
seconds and declares an instance down after it hasn’t responded for
instance_down_timeout_sec
seconds. Lowering these values shortens the write-unavailability window.
Upgrade the main pod:
kubectl delete pod memgraph-data-0-0
kubectl wait --for=condition=ready pod --allUpgrade coordinators
The upgrade of coordinators is done in exactly the same way. Start by upgrading followers and finish with deleting the leader pod:
kubectl delete pod memgraph-coordinator-3-0
kubectl wait --for=condition=ready pod --all
kubectl delete pod memgraph-coordinator-2-0
kubectl wait --for=condition=ready pod --all
kubectl delete pod memgraph-coordinator-1-0
kubectl wait --for=condition=ready pod --allVerify upgrade
Your upgrade should be finished now, to check that everything works, run:
SHOW VERSION;It should show you the new Memgraph version.
If you paused writes with global_read_only before the upgrade, return the
cluster to normal read/write operation once all nodes are upgraded and healthy:
SET COORDINATOR SETTING 'global_read_only' TO 'false';Rollback
If during the upgrade, you figured out that an error happened or even after
upgrading all of your pods something doesn’t work (e.g. write queries don’t
pass), you can safely downgrade your cluster to the previous version using
VolumeSnapshots you took on K8s or file backups for native deployments.
-
Kubernetes:
helm uninstall <release>In
values.yaml, for all instances set:restoreDataFromSnapshot: trueMake sure to set correct name of the snapshot you will use to recover your instances.
-
Native deployments: restore from your file backups.
If you’re doing an upgrade on minikube, it is important to make sure that the
snapshot resides on the same node on which the StatefulSet is installed.
Otherwise, it won’t be able to restore StatefulSet's attached
PersistentVolumeClaim from the VolumeSnapshot.