# Set up HA cluster with K8s Enterprise
> **Note**
>
> Users are advised to first read the guide on [how replication
> works](https://memgraph.com/docs/clustering/replication/how-replication-works), followed by the guide on
> [how high availability
> works](https://memgraph.com/docs/clustering/high-availability/how-high-availability-works), and how to [query
> the cluster](https://memgraph.com/docs/clustering/high-availability/querying-the-cluster-in-high-availability).
## Install Memgraph HA on Kubernetes
To deploy a Memgraph High Availability (HA) cluster on Kubernetes, you must
first [add the Memgraph Helm repository](#add-the-helm-repository) and then
[install the HA Helm chart](#install-memgraph-ha).
### Add the Helm repository
Add the Memgraph Helm chart repository to your local Helm setup by running the
following command:
```
helm repo add memgraph https://memgraph.github.io/helm-charts
```
Make sure to update the repository to fetch the latest Helm charts available:
```
helm repo update
```
### Install Memgraph HA
Since Memgraph HA requires an [Enterprise
license](https://memgraph.com/docs/database-management/enabling-memgraph-enterprise), you must provide
the license and organization name to the chart through a Kubernetes `Secret`.
> **Warning**
>
> **Breaking change**: Starting with Memgraph HA chart version 1.0.0, the HA chart no longer accepts
> the license and organization name as plaintext values via `env.MEMGRAPH_ENTERPRISE_LICENSE`
> and `env.MEMGRAPH_ORGANIZATION_NAME`. Both values are now read from a Kubernetes
> `Secret` referenced via `secretKeyRef`, and the secret **must exist before you run
> `helm install`** — the StatefulSets will fail to start otherwise. The previous
> `env.*` values have been removed from `values.yaml`.
Create the secret first, then install the chart:
```
kubectl create secret generic memgraph-secrets \
--from-literal=MEMGRAPH_ENTERPRISE_LICENSE= \
--from-literal=MEMGRAPH_ORGANIZATION_NAME=
helm install memgraph/memgraph-high-availability
```
Replace `` with a name of your choice for the release. The
secret name and keys are configurable via `secrets.name`, `secrets.licenseKey`
and `secrets.organizationKey` (defaults: `memgraph-secrets`,
`MEMGRAPH_ENTERPRISE_LICENSE`, `MEMGRAPH_ORGANIZATION_NAME`).
The cluster will be fully connected once installation completes. Note that the
install command may take a moment while instances establish connections. If
clients connect from outside the cluster, update the Bolt server address on
each instance to use its external IP as explained in the section on setting up
the cluster.
> **Info**
>
> **Tip:** Always install a specific chart version. Using
> the `latest` tag can lead to unexpected behavior if pods restart and pull newer,
> incompatible images.
### Install Memgraph HA with `kind`
For local development, we suggest using `kind`. Running:
```
kind create cluster
```
gives you a fully initialized environment that is sufficient for running the HA chart.
### Install Memgraph HA with `minikube`
If you are installing Memgraph HA chart locally with `minikube`, we are strongly
recommending to enable `csi-hostpath-driver` and use its storage class.
Otherwise,
you could have problems with attaching PVCs to pods.
#### Enable csi-hostpath-driver
```
minikube addons disable storage-provisioner
minikube addons disable default-storageclass
minikube addons enable volumesnapshots
minikube addons enable csi-hostpath-driver
```
#### Create a StorageClass (save as sc.yaml)
```
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: csi-hostpath-delayed
provisioner: hostpath.csi.k8s.io
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Delete
```
#### Apply the StorageClass
```kubectl apply -f sc.yaml```
#### Configure the Helm chart
In your `values.yaml`, set:
```yaml
storage:
libStorageClassName: csi-hostpath-delayed
```
## Configure the Helm chart
### Override default chart values
You can customize the Memgraph HA Helm chart either inline with `--set` flags or
by using a `values.yaml` file.
**Option 1: Override values inline**
```bash
helm install memgraph/memgraph-high-availability \
--set =,=,...
```
**Option 2: Use a values file**
```bash
helm install memgraph/memgraph-high-availability \
-f values.yaml
```
You can also combine both approaches. Values specified with `--set` override
those in `values.yaml`.
### Upgrade Helm chart
To upgrade the helm chart you can use:
```
helm upgrade memgraph/memgraph-high-availability --set =,=
```
Again it is possible use both `--set` and values.yaml to set configuration
options.
If you're using `IngressNginx` and performing an upgrade, the attached public IP
should remain the same. It will only change if the release includes specific
updates that modify it—and such changes will be documented.
### Uninstall Helm chart
Uninstallation is done with:
```
helm uninstall
```
Uninstalling the chart does **not** delete PersistentVolumeClaims (PVCs). Even
if the default StorageClass reclaim policy is `Delete`, data on the underlying
PersistentVolumes (PVs) will not be removed automatically when uninstalling the
chart.
However, we still recommend configuring the reclaim policy to `Retain`, as
described in the [High availability storage](#high-availability-storage)
section.
## Runtime environment & security
### Security context
All Memgraph HA instances run as Kubernetes `StatefulSet` workloads, each with a
single pod. Depending on configuration, the pod contains two or three
containers:
- **memgraph-coordinator** - runs the Memgraph binary.
- **Optional sysctl init container** - enabled when `sysctlInitContainer.enabled` is set.
- **Optional fix-ownership init container** - enabled when `fixOwnershipInitContainer.enabled` is set. See [Manual ownership fix](#manual-ownership-fix).
Memgraph processes run as the non-root **memgraph** user with **no Linux capabilities
and no privilege escalation**. The security context is especially important when deploying the HA chart to Red Hat OpenShift. In a
sandboxed environment, you will have to disable the `init-sysctl` init container. You will most likely also need to change `memgraphUserId`
and `memgraphGroupId` so that they fall within OpenShift's allowed UID/GID range.
### High availability storage
Memgraph HA always uses PersistentVolumeClaims (PVCs) to store database files
and logs.
- Default storage size: **1Gi** (you will likely need to increase this).
- Default access mode: `ReadWriteOnce` (can be set to `ReadOnlyMany`,
`ReadWriteMany`, or `ReadWriteOncePod`).
- PVCs use the cluster’s **default StorageClass**, unless overridden.
You can explicitly set storage classes using:
- `storage.libStorageClassName` - for data volumes
- `storage.logStorageClassName` - for log volumes
Most default StorageClasses use a `Delete` reclaim policy, meaning deleting the
PVC deletes the underlying PersistentVolume (PV). We recommend switching to
`Retain`.
---
After your cluster is running, you can patch all PVs:
```bash
#!/bin/bash
PVS=$(kubectl get pv --no-headers -o custom-columns=":metadata.name")
for pv in $PVS; do
echo "Patching PV: $pv"
kubectl patch pv $pv -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'
done
echo "All PVs have been patched."
```
> **Info**
>
> Kubernetes uses **Storage Object in Use Protection**, preventing deletion of
> PVCs while still attached to pods.
> Similarly, PVs will remain until their PVCs are fully removed.
>
> If a PVC is stuck terminating, you can remove its finalizers:
>
> ```bash
kubectl patch pvc PVC_NAME -p '{"metadata":{"finalizers": []}}' --type=merge
```
### Extra volumes
Besides the PVCs the chart creates for database files and logs, you can attach
your own volumes to Memgraph pods at two levels:
- **Per role** — `storage.data.extraVolumes` / `storage.data.extraVolumeMounts`
apply to every data instance, and `storage.coordinators.extraVolumes` /
`storage.coordinators.extraVolumeMounts` apply to every coordinator.
- **Per instance** — `extraVolumes` / `extraVolumeMounts` on an individual entry
in `data[]` or `coordinators[]` apply only to that one instance.
The two levels are additive: a pod gets the volumes defined for its role plus
the ones defined on its own instance entry. Volume names must be unique across
both levels, otherwise Kubernetes rejects the pod spec because of duplicate
volume names. Both levels are rendered through `tpl`, so Helm templating such as
`{{ .Release.Name }}` works inside the volume definitions.
A typical use case for per-instance volumes is mounting a dataset that only one
instance needs, for example a PVC holding CSV files to import. First create the
claim:
```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-csv-import
namespace: memgraph
annotations:
helm.sh/resource-policy: keep
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
```
Then mount it on a single data instance:
```yaml
data:
- id: "0"
extraVolumes:
- name: csv-import
persistentVolumeClaim:
claimName: pvc-csv-import
extraVolumeMounts:
- name: csv-import
mountPath: /import
readOnly: true
- id: "1"
```
> **Info**
>
> With the `ReadWriteOnce` access mode, only one pod can mount the claim at a
> time, so attach it to a single instance. If several instances need the same
> data, use a claim with the `ReadWriteMany` access mode, or define the volume at
> the role level instead.
### Network configuration
All Memgraph HA components communicate internally using ClusterIP network for
communicating between themselves.
**Default ports:**
- Management: **10000**
- Replication (data instances): **20000**
- Coordinator communication: **12000**
You can change this configuration by specifying:
```yaml
ports:
managementPort:
replicationPort:
coordinatorPort:
```
#### External network configuration
Memgraph HA uses **client-side routing**, so DNS resolution happens on the driver. Because of that, we need
one more type of network which will be used for clients accessing instances
from outside the cluster. Our HA supports out of the box following K8s resources used to setup external access:
- **IngressNginx** - one LoadBalancer for all instances. TCP port bases are configured under `externalAccessConfig.ingress`.
- **NodePort** - exposes ports on each node (requires public node IPs).
- **LoadBalancer** - one LoadBalancer per instance (highest cost).
- **CommonLoadBalancer (coordinators only)** - single LB for all coordinators.
- **Gateway API** - uses Kubernetes Gateway API resources (Gateway + TCPRoute), with one listener per data instance and a single shared listener load balancing across all coordinators. Configured under `externalAccessConfig.gateway`.
For coordinators, there is an additional option of using `CommonLoadBalancer`.
In this scenario, there is one load balancer sitting in front of coordinators.
You can save the cost of two load balancers compared to `LoadBalancer` option
since usually you don't need to distinguish specific coordinators while using
Memgraph capabilities. Note that if you will be connecting to the coordinator directly for some reason (e.g to run `show instances` query),
you can run `show instance` query to see which coordinator you got routed to.
The default Bolt port is opened on 7687 but you can change it by setting `ports.boltPort`.
For more detailed IngressNginx setup, see [Use Memgraph HA chart with
IngressNginx](#use-memgraph-ha-chart-with-ingressnginx).
Note however that Ingress Nginx is getting retired and one of the alternatives is using the [Kubernetes Gateway API](https://gateway-api.sigs.k8s.io/) with
controllers like Envoy Gateway, Istio, Cilium, Traefik, or Kong. The HA chart has native Gateway API support — see [Use Memgraph HA chart with Gateway API](#use-memgraph-ha-chart-with-gateway-api).
By default, the chart does **not** expose any external network services.
#### Recommended external access setup
External access methods can be **mixed per tier**: coordinators and data
instances each pick their own method independently. Our recommendation is to
expose **coordinators through a `CommonLoadBalancer`** and **data instances
through IngressNginx or the Gateway API**. This splits external access across
two independent load balancers, so routing stays available even if a single
load balancer goes down: if the load balancer in front of the data instances
fails, clients can still reach the coordinators and get routing information,
and if the coordinators' load balancer fails, connections to the data
instances keep working. If everything sits behind one shared load balancer,
that load balancer is a single point of failure for all external access.
To set up the recommended combination with the Gateway API, leave
`externalAccessConfig.dataInstance.serviceType` empty — the Gateway only
exposes tiers whose `serviceType` is empty, so it will expose the data
instances only:
```yaml
externalAccessConfig:
coordinator:
serviceType: "CommonLoadBalancer"
annotations:
external-dns.alpha.kubernetes.io/hostname: "memgraph.coordinators.example.com"
# dataInstance.serviceType stays empty, so the Gateway exposes only the data instances
gateway:
enabled: true
gatewayClassName: "eg"
annotations:
external-dns.alpha.kubernetes.io/hostname: "memgraph.data.example.com"
```
Or with IngressNginx for the data instances:
```yaml
externalAccessConfig:
dataInstance:
serviceType: "IngressNginx"
annotations:
external-dns.alpha.kubernetes.io/hostname: "memgraph.data.example.com"
coordinator:
serviceType: "CommonLoadBalancer"
annotations:
external-dns.alpha.kubernetes.io/hostname: "memgraph.coordinators.example.com"
```
In both cases every coordinator is reachable at the shared
`memgraph.coordinators.example.com:` address, and each data instance
at `memgraph.data.example.com:`. With the `external-dns`
hostnames set, the `cluster-setup` Job registers the external `bolt_server`
addresses automatically (see [Update bolt server](#update-bolt-server)).
All other combinations are valid as well — for coordinators + data instances,
for example:
- `CommonLoadBalancer` + `LoadBalancer`
- `LoadBalancer` + `LoadBalancer`
- `IngressNginx` + `IngressNginx` (everything behind the single ingress-nginx load balancer)
- Gateway API for everything (leave both `serviceType` fields empty and enable the gateway) — data instances get a port each, all coordinators share the Bolt port
- `NodePort` for either tier, combined with any of the above
The only invalid combination is enabling the Gateway API
(`externalAccessConfig.gateway.enabled: true`) while **both** tiers have a
`serviceType` set: a tier with a `serviceType` is excluded from the Gateway,
so the Gateway would apply to nothing and chart rendering fails.
#### Per-instance external access annotations
When using `LoadBalancer` or `NodePort` external access, you can set annotations
globally via `externalAccessConfig.dataInstance.annotations` and
`externalAccessConfig.coordinator.annotations`. These apply to every external
Service of that type.
If you need different annotations per instance — for example, to assign unique
DNS hostnames via `external-dns` — use the `externalAccessAnnotations` field on
individual entries in `data[]` or `coordinators[]`. Per-instance annotations are
merged with the global annotations, and **per-instance values take precedence**
when the same key appears in both.
```yaml
externalAccessConfig:
dataInstance:
serviceType: "LoadBalancer"
annotations:
service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
data:
- id: "0"
externalAccessAnnotations:
external-dns.alpha.kubernetes.io/hostname: "data-0.memgraph.example.com"
- id: "1"
externalAccessAnnotations:
external-dns.alpha.kubernetes.io/hostname: "data-1.memgraph.example.com"
```
In this example, each data instance's external Service gets the shared
`aws-load-balancer-scheme` annotation plus its own unique `external-dns`
hostname. Bolt and management ports are not set per-instance — they come from
`ports.boltPort` and `ports.managementPort`.
With `serviceType: LoadBalancer` and a per-instance `external-dns` hostname
set like above, the `cluster-setup` Job automatically registers each
instance's routing `bolt_server` as `:` (see [Update bolt
server](#update-bolt-server)). Do **not** set the `external-dns` hostname at
the tier level (`externalAccessConfig.dataInstance.annotations` /
`externalAccessConfig.coordinator.annotations`) when using `LoadBalancer`:
each instance gets its own LoadBalancer Service, so a tier-level hostname
would put one DNS record in front of every load balancer and register the
same `bolt_server` for all instances.
#### Per-instance internal access annotations
Each data instance and coordinator also has an internal ClusterIP Service used
for in-cluster communication. You can set per-instance annotations on these
internal Services using the `internalAccessAnnotations` field on individual
entries in `data[]` or `coordinators[]`. This is useful for integrations or other tooling that consumes annotations on the internal
Services.
```yaml
data:
- id: "0"
internalAccessAnnotations:
mycompany.io/service-mesh: "enabled"
- id: "1"
internalAccessAnnotations:
mycompany.io/service-mesh: "enabled"
coordinators:
- id: "1"
internalAccessAnnotations:
mycompany.io/service-mesh: "enabled"
```
### Node affinity
Memgraph HA deploys multiple pods, and you can control pod placement with
affinity settings.
Supported strategies:
- **default** Attempts to to schedule the data pods and coordinator pods on the
nodes where there is no other pod with the same role. If there is no such node,
the pods will still be scheduled on the same node, and deployment will not
fail.
- **unique** (`affinity.unique = true`)
Each coordinator and data pod must be placed on separate nodes. If not enough
nodes exist, deployment fails. Coordinators get scheduled first. After that,
data pods are looking for the nodes with coordinators.
- **parity** (`affinity.parity = true`)
Schedules at most **one coordinator + one data pod per node**.
Coordinators schedule first; data pods follow.
- **nodeSelection** (`affinity.nodeSelection = true`)
Pods are scheduled onto explicitly labeled nodes using
`affinity.dataNodeLabelValue` and `affinity.coordinatorNodeLabelValue`. If all
the nodes with labels are occupied by the pods with the same role, the
deployment will fail.
> **Info**
>
> When using `nodeSelection`, ensure that nodes are labeled correctly.
> Default role label key: `role`
> Default values: `data-node`, `coordinator-node`
>
> Example:
>
> ```bash
kubectl label nodes role=data-node
```
>
> A full AKS example is available in the [chart
> repository](https://github.com/memgraph/helm-charts/tree/main/charts/memgraph-high-availability/aks).
All strategies above spread pods across a topology domain defined by
`affinity.topologyKey`. The default, `kubernetes.io/hostname`, treats every
node as its own domain. Set it to a different node label to spread pods across
a wider domain, for example across availability zones:
```yaml
affinity:
topologyKey: topology.kubernetes.io/zone
unique: true
```
The same topology key is applied to both coordinator and data pod rules, so the
label must be present on every node that should take part in scheduling.
### Sysctl options
Use the `sysctlInitContainer` to configure kernel parameters required for
high-memory workloads, such as increasing:
- [`vm.max_map_count`](https://memgraph.com/docs/database-management/system-configuration#increasing-memory-map-areas)
### Manual ownership fix
Some storage drivers (notably `rancher.io/local-path`) do not honor pod-level
`fsGroup`, leaving the volume root owned by `root:root`. Because Memgraph runs
as a non-root user, its storage directory ownership assertion (process euid ==
data directory owner uid) fails on startup.
When `fixOwnershipInitContainer.enabled` is set to `true`, an init container
runs as root before Memgraph starts and `chown`s the lib, log, and core-dumps
mount points to `memgraphUserId:memgraphGroupId`. The container drops all Linux
capabilities except `CHOWN`, uses a read-only root filesystem, and disables
privilege escalation.
To enable it:
```yaml
fixOwnershipInitContainer:
enabled: true
image:
repository: docker.io/library/busybox
tag: 1.37.0
pullPolicy: IfNotPresent
```
The container only chowns the mount paths that exist for the role — `/var/log/memgraph`
is included when `storage..createLogStorageClaim` is `true`, and
`storage..coreDumpsMountPath` is included when `storage..createCoreDumpsClaim`
is `true`.
### Authentication
By default, Memgraph HA starts **without authentication** enabled.
> **Warning**
>
> **Breaking change**: The HA chart no longer creates a Memgraph user from the
> `USER`/`PASSWORD` keys of the `memgraph-secrets` Secret. The `secrets.enabled`,
> `secrets.userKey` and `secrets.passwordKey` values have been removed because
> the previous implementation also applied these env variables to coordinators,
> which run without auth. The `memgraph-secrets` Secret is now reserved for the
> license and organization name.
To configure credentials, connect to a data instance after installation and
create users with Cypher, for example:
```cypher
CREATE USER memgraph IDENTIFIED BY 'memgraph';
```
Run the same statements on every data instance you want the user to exist on.
Coordinators run without authentication and do not need user setup.
### Bolt SSL/TLS
Each data instance and coordinator can independently terminate Bolt
connections over TLS. When enabled, the chart mounts a pre-existing
Kubernetes Secret containing the certificate and private key at
`/etc/memgraph/ssl/` and auto-appends `--bolt-cert-file=/etc/memgraph/ssl/tls.crt`
and `--bolt-key-file=/etc/memgraph/ssl/tls.key` to the instance's args.
> **Warning**
>
> **Breaking change in HA chart version with TLS config**: The previous way of
> enabling Bolt TLS — passing `--bolt-cert-file` / `--bolt-key-file` through
> `data[].args` / `coordinators[].args` and mounting the certificate Secret
> through `storage.{data,coordinators}.extraVolumes` / `extraVolumeMounts` — is
> no longer supported. Setting `--bolt-cert-file` or `--bolt-key-file` in `args`
> now causes `helm install` to fail with a template error. Migrate to the
> `tls.bolt` block on each instance instead.
To enable Bolt TLS, first create a Kubernetes Secret holding the certificate
and private key in the release namespace:
```bash
kubectl create secret tls bolt-tls-secret \
--cert=path/to/tls.crt \
--key=path/to/tls.key
```
Then enable `tls.bolt` on each instance that should terminate TLS:
```yaml
data:
- id: "0"
tls:
bolt:
enabled: true
secretName: bolt-tls-secret
certSecretPath: tls.crt
keySecretPath: tls.key
- id: "1"
tls:
bolt:
enabled: true
secretName: bolt-tls-secret
certSecretPath: tls.crt
keySecretPath: tls.key
coordinators:
- id: "1"
tls:
bolt:
enabled: true
secretName: bolt-tls-secret
- id: "2"
tls:
bolt:
enabled: true
secretName: bolt-tls-secret
- id: "3"
tls:
bolt:
enabled: true
secretName: bolt-tls-secret
```
`certSecretPath` and `keySecretPath` are the keys inside the Secret holding
the certificate and key respectively (default `tls.crt` and `tls.key`).
The chart fails the install if `tls.bolt.enabled` is `true` but
`tls.bolt.secretName` is empty.
When a coordinator has `tls.bolt.enabled: true`, the cluster-setup job
that registers coordinators and data instances automatically uses
`--use-ssl` when connecting to coordinator 1.
### Intra-cluster SSL/TLS
Independently of Bolt TLS, each data instance and coordinator can encrypt the
internal communication between cluster members (coordinator-to-coordinator and
coordinator-to-data traffic). When `tls.intraCluster.enabled` is `true`, the
chart mounts a pre-existing Kubernetes Secret containing the certificate,
private key and CA bundle at `/etc/memgraph/intra_cluster_tls/` and
auto-appends `--cluster-cert-file=/etc/memgraph/intra_cluster_tls/tls.crt`,
`--cluster-key-file=/etc/memgraph/intra_cluster_tls/tls.key` and
`--cluster-ca-file=/etc/memgraph/intra_cluster_tls/ca.crt` to the instance's
args.
To enable intra-cluster TLS, first create a Kubernetes Secret holding the
certificate, private key and CA bundle for each instance in the release
namespace. For example, for `data-0`:
```bash
kubectl create secret generic intra-tls-data-0-secret \
--from-file=tls.crt=path/to/tls.crt \
--from-file=tls.key=path/to/tls.key \
--from-file=ca.crt=path/to/ca.crt
```
Then enable `tls.intraCluster` on each instance that should encrypt internal
traffic:
```yaml
data:
- id: "0"
tls:
intraCluster:
enabled: true
secretName: intra-tls-data-0-secret
certSecretPath: tls.crt
keySecretPath: tls.key
caSecretPath: ca.crt
- id: "1"
tls:
intraCluster:
enabled: true
secretName: intra-tls-data-1-secret
certSecretPath: tls.crt
keySecretPath: tls.key
caSecretPath: ca.crt
coordinators:
- id: "1"
tls:
intraCluster:
enabled: true
secretName: intra-tls-coord-1-secret
- id: "2"
tls:
intraCluster:
enabled: true
secretName: intra-tls-coord-2-secret
- id: "3"
tls:
intraCluster:
enabled: true
secretName: intra-tls-coord-3-secret
```
`certSecretPath`, `keySecretPath` and `caSecretPath` are the keys inside the
Secret holding the certificate, private key and CA bundle respectively
(default `tls.crt`, `tls.key` and `ca.crt`). The chart fails the install if
`tls.intraCluster.enabled` is `true` but `tls.intraCluster.secretName` is
empty. Enable it on every instance that participates in encrypted intra-cluster
communication.
## Setting up the cluster
Although many configuration options exist, especially for networking, the workflow for creating a Memgraph HA cluster follows these steps:
1. Provision the Kubernetes cluster. Ensure your nodes, storage, and networking are ready.
2. Label nodes according to your chosen affinity strategy (optional). For example, when using `nodeSelection`, label nodes as `data-node` or `coordinator-node`.
3. Create the `memgraph-secrets` Kubernetes secret holding `MEMGRAPH_ENTERPRISE_LICENSE` and `MEMGRAPH_ORGANIZATION_NAME` (required — the chart reads these via `secretKeyRef`).
4. Install the Memgraph HA Helm chart using `helm install`. This creates a fully connected cluster.
5. Install auxiliary components for external access, such as `ingress-nginx` (optional).
6. Update Bolt server addresses if clients will connect from outside the cluster (optional).
### Customize the cluster-setup Job
After `helm install`, the chart runs a `cluster-setup` Job as a Helm
`post-install` and `post-upgrade` hook. This Job registers all coordinators and
data instances via `mgconsole`, producing a fully connected cluster, and re-runs
on every `helm upgrade` to reconcile cluster membership (see [Scale the cluster
up and down](#scale-the-cluster-up-and-down)).
You can customize the labels and security context applied to this Job through
the `clusterSetup` block:
- `clusterSetup.labels` - custom labels applied to both the Job and its pod
template. Useful for matching network policies, cost-allocation, or
observability selectors that target the setup Job.
- `clusterSetup.podSecurityContext` - pod-level security context for the
`cluster-setup` pod.
- `clusterSetup.containerSecurityContext` - container-level security context for
the `cluster-setup` container.
By default the Job runs hardened: as the non-root `memgraph` user
(`runAsUser: 101`, `runAsGroup: 103`), with a read-only root filesystem, all
Linux capabilities dropped, no privilege escalation, and the `RuntimeDefault`
seccomp profile. Override these values when deploying to a restricted
environment such as Red Hat OpenShift, where the allowed UID/GID range differs:
```yaml
clusterSetup:
labels:
app.kubernetes.io/component: cluster-setup
podSecurityContext:
runAsUser: 1000680000
runAsGroup: 1000680000
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
containerSecurityContext:
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
readOnlyRootFilesystem: true
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
```
### Scale the cluster up and down
The `cluster-setup` Job runs as both a `post-install` and a `post-upgrade` hook,
so it re-runs on every `helm upgrade` and reconciles the registered cluster with
your chart values. All operations are idempotent — existing coordinators and data
instances are left untouched:
- **Scale out**: add an entry to `coordinators[]` or `data[]` and run `helm
upgrade`. The new coordinator is registered with `ADD COORDINATOR` and the new
data instance with `REGISTER INSTANCE`.
- **Scale down**: remove an entry from `coordinators[]` or `data[]` and run `helm
upgrade`. Members that are registered in the cluster but no longer present in
the values are removed with `REMOVE COORDINATOR` / `UNREGISTER INSTANCE`.
Instance names are keyed by their stable `id`, not by their position in the
array, so removal is position-independent: dropping a middle entry unregisters
exactly that member and leaves the others' names and endpoints intact. Data
instance names are derived as `instance_` (so ids `0, 1, 2` map to
`instance_1, instance_2, instance_3`), and coordinator names as
`coordinator_`.
> **Info**
>
> **Scale-down safety**: the Job never removes the current MAIN data instance or
> the leader coordinator, because unregistering either can corrupt cluster state.
> When you scale one of those down, its pod is deleted and the cluster fails over /
> re-elects; on a later `helm upgrade` the removed member is no longer MAIN/leader
> and is cleaned up then. To remove it immediately, first promote/elect a different
> member, then upgrade — or run `UNREGISTER INSTANCE` / `REMOVE COORDINATOR`
> manually on the leader coordinator.
> **Warning**
>
> **Breaking change for non-default data instance IDs**: data instance names are
> now derived from each entry's `id` (`instance_`) instead of its position in
> the `data` array. For the shipped default ids (`0, 1, 2`) the names are unchanged
> (`instance_1, instance_2, instance_3`), so those deployments upgrade safely.
>
> If you use **non-sequential or non-zero-based ids** (e.g. `5, 6, 7`), the
> registered instance names change on upgrade: the `cluster-setup` Job will
> `REGISTER` the new names and `UNREGISTER` the old ones during reconciliation. The
> current MAIN is protected, but replicas are torn down and re-registered under
> their new names. Review your `data[].id` values before upgrading, and expect a
> brief replica re-registration if they don't follow the default `0`-based
> sequential scheme. In that case you may need to run `UNREGISTER INSTANCE` and
> `REMOVE COORDINATOR` manually to consolidate the cluster state.
>
> Each `data[].id` must be a unique, non-negative integer given as a string (e.g.
> `"0"`, `"1"`, `"2"`). Non-numeric ids are not supported, as they would all
> collapse to the same instance name.
### Update bolt server
This step is required only when:
- Clients access the database from outside the cluster, and
- You're using bolt+routing for client-side routing
> **Info**
>
> When you expose the cluster through **IngressNginx**, **Gateway API**, the
> coordinator **CommonLoadBalancer**, or per-instance **LoadBalancer** Services
> and set an `external-dns` hostname via annotations, the `cluster-setup` Job
> sets each instance's routing `bolt_server` to the external hostname and port
> automatically, and reconciles it on every `helm upgrade` (including reverting
> to the internal address when external access is turned off). In that case you
> don't need to run the queries below manually. For `LoadBalancer`, set the
> hostname **per instance** via `data[].externalAccessAnnotations` /
> `coordinators[].externalAccessAnnotations` — each instance has its own
> LoadBalancer Service, so a tier-level hostname would register the same
> `bolt_server` for every instance. This does not apply to `NodePort` (or to
> instances without a hostname annotation), where the internal `bolt_server` is
> kept and you update it manually.
Each instance must know its external address for routing to work correctly. Run the following queries on the leader coordinator:
```cypher
UPDATE CONFIG FOR COORDINATOR 1 WITH CONFIG {"bolt_server": ""};
UPDATE CONFIG FOR COORDINATOR 2 WITH CONFIG {"bolt_server": ""};
UPDATE CONFIG FOR COORDINATOR 3 WITH CONFIG {"bolt_server": ""};
UPDATE CONFIG FOR INSTANCE instance_0 WITH CONFIG {"bolt_server": ""};
UPDATE CONFIG FOR INSTANCE instance_1 WITH CONFIG {"bolt_server": ""};
```
Note that the only the `bolt_server` values are provided. The correct
value depends on the type of external access you configured (LoadBalancer IP,
Ingress host/port, NodePort, etc.).
Refer to the Memgraph HA [User API docs](https://memgraph.com/docs/clustering/high-availability/ha-commands-reference)
for the full set of commands and usage patterns.
### Use Memgraph HA chart with Gateway API
The Memgraph HA Helm chart has native support for the [Kubernetes Gateway API](https://gateway-api.sigs.k8s.io/). When enabled, the chart automatically creates a TCPRoute for each data instance and a single TCPRoute shared by all coordinators. You can either let the chart create its own Gateway or attach routes to a pre-existing one.
> **Warning**
>
> **Breaking change in HA chart versions after 1.3.3**: coordinators are no
> longer exposed on one Gateway listener each. They now share a single listener
> and a single TCPRoute on `ports.boltPort`, so
> `externalAccessConfig.gateway.coordinatorPortBase` no longer exists and chart
> rendering fails if you still set it. Remove the value from your `values.yaml`,
> point clients at `:` instead of the old per-coordinator
> ports, and if you use an existing Gateway, replace the `coordinator-{id}-bolt`
> listeners with a single `coordinators-bolt` listener on `ports.boltPort`. Change
> `ports.boltPort` if you need the coordinators on a different port.
> **Info**
>
> The Gateway only exposes tiers whose `serviceType` is **empty**: a tier with
> `externalAccessConfig.dataInstance.serviceType` or
> `externalAccessConfig.coordinator.serviceType` set gets its external access
> from that service type instead and is excluded from the Gateway (no listeners
> or TCPRoutes are created for it). This lets you mix methods per tier — for
> example, coordinators via `CommonLoadBalancer` and data instances via the
> Gateway, which is our [recommended setup](#recommended-external-access-setup).
> Enabling the gateway while **both** tiers have a `serviceType` set fails chart
> rendering, since the Gateway would then apply to nothing.
#### Prerequisites
Before enabling Gateway API in the chart, you need:
1. **The Gateway API CRDs** installed in your cluster. Some controllers ship them
with their own Helm chart (Envoy Gateway does), while others expect you to
install them yourself (Traefik does). The controller-specific guides below
cover both cases.
2. **A Gateway API controller** installed in your cluster. Examples include
[Envoy Gateway](https://gateway.envoyproxy.io/), [Istio](https://istio.io/)
(see also: [Use Istio as the Gateway API controller](#use-istio-as-the-gateway-api-controller)),
[Cilium](https://cilium.io/), [Traefik](https://traefik.io/), and
[Kong](https://konghq.com/). Step-by-step guides are available for two of
them: [Envoy Gateway](#deploy-with-envoy-gateway) and
[Traefik](#deploy-with-traefik).
3. **A GatewayClass resource** that references your controller. A GatewayClass is
a cluster-scoped resource that defines which controller manages Gateways —
each Gateway references a GatewayClass by name. The Memgraph Helm chart does
**not** create a GatewayClass; you must create one yourself or use one
provided by your controller installation (the Envoy Gateway chart does not
create one, the Traefik chart does).
> **Warning**
>
> The chart exposes Bolt through `TCPRoute`, which is part of the Gateway API
> **experimental** channel. A standard-channel CRD bundle is not enough — whatever
> you install must include `tcproutes.gateway.networking.k8s.io`, and the
> controller itself may need experimental support turned on (Traefik does, see
> [Deploy with Traefik](#deploy-with-traefik)).
> **Warning**
>
> You must ensure the GatewayClass exists before enabling the gateway feature in the chart. If you create your own Gateway (Option 1 below), the chart requires `gatewayClassName` to reference an existing GatewayClass, and will fail with an error if it is not set.
#### Option 1: Chart-managed Gateway
When you want the chart to create its own Gateway along with TCPRoute resources, set `externalAccessConfig.gateway.enabled` to `true` and provide the `gatewayClassName`:
```yaml
externalAccessConfig:
gateway:
enabled: true
gatewayClassName: "eg"
```
The chart will create, for every tier whose `serviceType` is empty:
- A **Gateway** (`gateway.networking.k8s.io/v1`) with one TCP listener per data instance, plus a single shared listener for all coordinators.
- A **TCPRoute** (`gateway.networking.k8s.io/v1alpha2`) per data instance, plus one TCPRoute for the whole coordinator tier that lists every coordinator Service as a backend.
Each data instance gets its own port, assigned as `dataPortBase + data instance
id` (default: 9000, 9001, ...), because clients must be able to reach a
specific data instance. Coordinators instead share **one** listener on
`ports.boltPort` (default 7687): the single coordinator TCPRoute has every
coordinator Service as an equally weighted backend, so the Gateway distributes
new connections round-robin across them. Clients therefore need only one
coordinator address, connections keep working when the coordinator they landed
on goes away, and scaling the coordinator tier does not add a listener or a
port. When an `external-dns` hostname is set on the Gateway, the
`cluster-setup` Job registers that shared `:` as the routing
`bolt_server` of every coordinator, the same way it does for a coordinator
`CommonLoadBalancer` (see [Update bolt server](#update-bolt-server)).
You can customize the data instance base port:
```yaml
externalAccessConfig:
gateway:
enabled: true
gatewayClassName: "eg"
dataPortBase: 9000
```
Keep the resulting data instance ports away from `ports.boltPort`: when both
tiers are exposed through the Gateway and `dataPortBase + id` of some data
instance equals `ports.boltPort`, that listener would collide with the shared
coordinator listener and chart rendering fails.
You can also set annotations and labels on the Gateway resource:
```yaml
externalAccessConfig:
gateway:
enabled: true
gatewayClassName: "eg"
annotations:
example.io/owner: "memgraph"
labels:
app: memgraph-ha
```
To install with a chart-managed Gateway (assuming the `memgraph-secrets`
Secret with the license and organization name already exists, see [Install
Memgraph HA](#install-memgraph-ha)):
```bash
helm install memgraph-ha memgraph/memgraph-high-availability \
--set externalAccessConfig.gateway.enabled=true \
--set externalAccessConfig.gateway.gatewayClassName=eg
```
#### Option 2: Existing (external) Gateway
When you already have a Gateway resource in your cluster (for example, a shared Gateway serving multiple services including Memgraph Lab), you can have the chart create only TCPRoute resources that attach to it:
```yaml
externalAccessConfig:
gateway:
enabled: true
existingGatewayName: "memgraph-gateway"
```
In this mode, the chart skips Gateway creation and only creates TCPRoute resources. The `gatewayClassName` is not required.
If the existing Gateway is in a different namespace, specify it:
```yaml
externalAccessConfig:
gateway:
enabled: true
existingGatewayName: "memgraph-gateway"
existingGatewayNamespace: "gateway-system"
```
To install with an existing Gateway (assuming the `memgraph-secrets` Secret
with the license and organization name already exists, see [Install Memgraph
HA](#install-memgraph-ha)):
```bash
helm install memgraph-ha memgraph/memgraph-high-availability \
--set externalAccessConfig.gateway.enabled=true \
--set externalAccessConfig.gateway.existingGatewayName=memgraph-gateway
```
> **Warning**
>
> When using an existing Gateway, ensure it has listeners whose **names** match the
> TCPRoute `sectionName` references. The chart expects listener names in the format
> `data-{id}-bolt` for data instances and `coordinator-{id}-bolt`
> for coordinators. The listener ports are your choice in this mode,
> but keeping them aligned with the chart defaults avoids surprises. For example,
> the default HA setup (2 data instances, 3 coordinators) needs these listeners:
>
> - `data-0-bolt` on port 9000
> - `data-1-bolt` on port 9001
> - `coordinator-1-bolt` on port 10001
> - `coordinator-2-bolt` on port 10002
> - `coordinator-3-bolt` on port 10003
>
> A standalone Gateway manifest with these pre-configured listeners is available in the [Helm charts repository](https://github.com/memgraph/helm-charts/blob/main/examples/gateway/gateway.yaml).
> **Info**
>
> **TCPRoute API version**: TCPRoute uses `v1alpha2`, which is the latest available API version. It is supported by Envoy Gateway and other major implementations but is not yet GA. Gateway and HTTPRoute are both GA (`v1`).
#### Deploy with Envoy Gateway
[Envoy Gateway](https://gateway.envoyproxy.io/) is the controller used in the
examples above. Its Helm chart bundles the Gateway API CRDs, so no separate CRD
installation is needed, and it programs listener ports dynamically — the Bolt
ports only have to be configured in the Memgraph chart.
#### Install Envoy Gateway
```bash
helm install eg oci://docker.io/envoyproxy/gateway-helm --version v1.2.4 \
-n envoy-gateway-system --create-namespace
```
Chart version `v1.2.4` ships Gateway API bundle version `v1.2.1`, which includes
the `TCPRoute` CRD.
#### Create the GatewayClass
The Envoy Gateway chart does not create a GatewayClass, so create one that points
at its controller:
```yaml
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: eg
spec:
controllerName: gateway.envoyproxy.io/gatewayclass-controller
```
```bash
kubectl apply -f gatewayclass.yaml
```
#### Install the Memgraph HA chart
Assuming the `memgraph-secrets` Secret with the license and organization name
already exists (see [Install Memgraph HA](#install-memgraph-ha)):
```bash
helm install memgraph-ha memgraph/memgraph-high-availability \
--set externalAccessConfig.gateway.enabled=true \
--set externalAccessConfig.gateway.gatewayClassName=eg
```
#### Verify the setup
```bash
kubectl get gatewayclass eg
kubectl get gateway -o wide
kubectl get tcproute
```
The GatewayClass should be `ACCEPTED=True` and the Gateway `PROGRAMMED=True` with
an external address. Envoy Gateway provisions a separate Envoy deployment and
LoadBalancer Service for the Gateway.
#### Deploy with Traefik
[Traefik](https://traefik.io/) can serve the same setup, but its Gateway API
support differs from Envoy Gateway in two ways that affect the Memgraph chart:
- Its Helm chart ships **no Gateway API CRDs**, so you install them yourself, and
`TCPRoute` support has to be explicitly enabled.
- Traefik does **not** open arbitrary listener ports. Every port used by a Gateway
listener must also be declared as a Traefik entryPoint, otherwise the listener
is silently ignored.
#### Install the Gateway API CRDs
Install the experimental channel, which is where `TCPRoute` lives:
```bash
kubectl apply --server-side --force-conflicts \
-f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.5.1/experimental-install.yaml
```
> **Info**
>
> Use `--server-side`. A plain `kubectl apply` stores the whole manifest in the
> `kubectl.kubernetes.io/last-applied-configuration` annotation and the HTTPRoute
> CRD exceeds the 256 KB annotation limit, failing with
> `metadata.annotations: Too long: may not be more than 262144 bytes`.
> `--force-conflicts` is needed only when the CRDs were previously created with a
> client-side apply.
#### Configure Traefik
Create a `traefik-values.yaml` file. The `ports` entries are the entryPoints
backing the Gateway listeners the Memgraph chart creates — data instances on
`dataPortBase + data instance id`, coordinators on
`coordinatorPortBase + coordinator id`:
```yaml
providers:
kubernetesGateway:
enabled: true
# Required for TCPRoute support.
experimentalChannel: true
statusAddress:
service:
# Copies the Traefik LoadBalancer address into Gateway .status.addresses,
# which is what external-dns reads.
enabled: true
# The Memgraph chart creates its own Gateway, so Traefik's default one is not needed.
gateway:
enabled: false
# Creates the GatewayClass "traefik" (controller traefik.io/gateway-controller).
gatewayClass:
enabled: true
service:
type: LoadBalancer
ports:
data-0:
port: 9000
exposedPort: 9000
protocol: TCP
expose:
default: true
data-1:
port: 9001
exposedPort: 9001
protocol: TCP
expose:
default: true
```
A ready-to-use version of this file is available in the [Helm charts
repository](https://github.com/memgraph/helm-charts/blob/main/examples/gateway/traefik-values.yaml).
> **Warning**
>
> Declare one entryPoint per Gateway listener, and remember to add new ones when
> you [scale the cluster](#scale-the-cluster-up-and-down) — a third data instance
> needs port 9002, a fourth coordinator needs port 10004. If a tier is exposed
> through a `serviceType` instead of the Gateway, it has no listeners and needs no
> entryPoints; with the [recommended setup](#recommended-external-access-setup)
> (coordinators via `CommonLoadBalancer`, data instances via the Gateway) only the
> `data-*` entryPoints are required.
#### Install Traefik
```bash
helm repo add traefik https://traefik.github.io/charts
helm repo update
helm install traefik traefik/traefik -n traefik --create-namespace \
-f traefik-values.yaml
```
The values above were verified against Traefik chart `41.0.2` (Traefik `v3.7.6`).
The `providers.kubernetesGateway.*` options require Traefik 3.x.
#### Install the Memgraph HA chart
Point `gatewayClassName` at the GatewayClass created by the Traefik chart.
Assuming the `memgraph-secrets` Secret with the license and organization name
already exists (see [Install Memgraph HA](#install-memgraph-ha)):
```bash
helm install memgraph-ha memgraph/memgraph-high-availability \
--set externalAccessConfig.gateway.enabled=true \
--set externalAccessConfig.gateway.gatewayClassName=traefik
```
#### Verify the setup
```bash
kubectl get gatewayclass traefik
kubectl get gateway -o wide
kubectl get tcproute
# The chart names the Gateway -memgraph-high-availability-gateway
kubectl describe gateway memgraph-ha-memgraph-high-availability-gateway
```
If a listener reports `Detached` or `UnsupportedProtocol`, the matching entryPoint
is missing from the Traefik values. Unlike Envoy Gateway, Traefik routes all
Gateways through the single Traefik Service, so the external address comes from
the `traefik` Service in the `traefik` namespace.
#### Use Istio as the Gateway API controller
[Istio](https://istio.io/) implements the Gateway API and works with the HA
chart out of the box — only Istio's `base` and `istiod` Helm charts are needed,
because istiod deploys a gateway for every Gateway resource automatically (the
`istio/gateway` chart is not used), and the installation creates the `istio`
GatewayClass for you.
Since the chart exposes Bolt through TCPRoute, which is an **alpha** Gateway
API resource, two extra steps are required compared to a plain Istio
installation: the Gateway API CRDs must come from the **experimental** channel,
and istiod must run with `PILOT_ENABLE_ALPHA_GATEWAY_API=true`.
Install the experimental Gateway API CRDs first — istiod detects the alpha CRDs
at startup, so if it is already running, restart it afterwards with `kubectl
rollout restart deploy/istiod -n istio-system`:
```bash
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.5.1/experimental-install.yaml
```
Then install Istio with the alpha Gateway API flag enabled:
```bash
helm repo add istio https://istio-release.storage.googleapis.com/charts
helm repo update
helm install istio-base istio/base -n istio-system --create-namespace
helm install istiod istio/istiod -n istio-system --wait \
--set pilot.env.PILOT_ENABLE_ALPHA_GATEWAY_API=true
```
> **Warning**
>
> Without `PILOT_ENABLE_ALPHA_GATEWAY_API=true`, istiod rejects the chart's TCP
> listeners (`protocol "TCP" is supported, but only when
> PILOT_ENABLE_ALPHA_GATEWAY_API=true is configured`) and the failure is easy to
> misread: the Gateway still shows `Programmed=True` and its LoadBalancer Service
> still lists all the ports, but the TCPRoutes get **no status at all** and every
> connection times out. If you hit this, set the flag with `helm upgrade istiod
> istio/istiod -n istio-system --reuse-values --set
> pilot.env.PILOT_ENABLE_ALPHA_GATEWAY_API=true --wait` and check that the
> TCPRoutes report `Accepted`.
Finally, install the HA chart with the `istio` GatewayClass:
```bash
helm install memgraph-ha memgraph/memgraph-high-availability \
--set externalAccessConfig.gateway.enabled=true \
--set externalAccessConfig.gateway.gatewayClassName=istio
```
Istio provisions a Deployment and a LoadBalancer Service named
`-istio` carrying every listener port: clients reach all
coordinators through the shared listener at `:` (default
7687) and each data instance at `:` (default
9000, 9001, ...). Unlike some controllers (for example Traefik, which requires
every listener port to be declared as an entryPoint), Istio programs listener
ports automatically, so scaling the data tier needs no controller
configuration.
Drivers should connect with `bolt://` directly to the port they want, or, when
using `neo4j://` routing mode, set an `external-dns` hostname on the Gateway so
the coordinators register the external `:` as their routing
`bolt_server` (see [Update bolt server](#update-bolt-server)):
```yaml
externalAccessConfig:
gateway:
enabled: true
gatewayClassName: "istio"
annotations:
external-dns.alpha.kubernetes.io/hostname: "memgraph.example.com"
```
Istio copies the LoadBalancer address into the Gateway's `.status.addresses`,
which is where `external-dns` reads it from, so no extra Istio configuration is
needed.
> **Info**
>
> Using Istio as the Gateway API controller does **not** require adding the
> Memgraph pods to the Istio service mesh. If you do sidecar-inject them
> (namespace label `istio-injection=enabled`), use native sidecars (Kubernetes >=
> 1.28 and `--set pilot.env.ENABLE_NATIVE_SIDECARS=true` on istiod) — a classic
> `istio-proxy` sidecar keeps the `cluster-setup` Job pod running forever, which
> hangs the chart's post-install hook, and native sidecars also start the proxy
> before Memgraph so coordinators don't fail their initial connections to each
> other. On older clusters, exclude the Job from the mesh instead with `--set
> clusterSetup.labels."sidecar\.istio\.io/inject"=false`.
A ready-made istiod values file with these settings is available in the [Helm
charts repository](https://github.com/memgraph/helm-charts/blob/main/examples/gateway/istio-values.yaml).
### Use Memgraph HA chart with IngressNginx
One of the most cost-efficient ways to expose a Memgraph HA cluster is by using
**IngressNginx**. The controller supports TCP routing (including the Bolt
protocol), allowing all Memgraph instances to share:
* a **single LoadBalancer**, and
* a **single external IP address**.
Clients connect to any coordinator or data instance by using **different Bolt
ports**.
To install Memgraph HA with IngressNginx enabled (assuming the
`memgraph-secrets` Secret with the license and organization name already
exists, see [Install Memgraph HA](#install-memgraph-ha)):
```bash
helm install mem-ha-test ./charts/memgraph-high-availability --set \
affinity.nodeSelection=true,\
externalAccessConfig.dataInstance.serviceType=IngressNginx,\
externalAccessConfig.coordinator.serviceType=IngressNginx
```
When using these settings, the chart will automatically install and configure
**IngressNginx**, including all required TCP routing setup for Memgraph.
> **Warning**
>
> The `IngressNginx` service type can be combined with the Gateway API
> (`externalAccessConfig.gateway.enabled`) as long as at least one tier keeps an
> empty `serviceType` — the Gateway only exposes tiers whose `serviceType` is
> empty, so a tier using `IngressNginx` is simply excluded from the Gateway.
> Enabling the gateway while **both** tiers have a `serviceType` set fails chart
> rendering.
All `IngressNginx` tiers share a single ingress-nginx controller Service (one
LoadBalancer, one external IP). When you set an `external-dns` hostname via
annotations, the data instance and coordinator annotations are merged onto that
shared Service (the coordinator value wins on a key collision), so set the **same
hostname** on both tiers. Instances behind the shared Service are told apart by
port, not host: data instances are reachable at `ingress.dataPortBase + data
instance id` and coordinators at `ingress.coordinatorPortBase + coordinator id`.
## Probes
Memgraph HA uses standard Kubernetes **startup**, **readiness**, and
**liveness** probes to ensure correct container operation.
- **Startup probe**
Determines when Memgraph has fully started.
It succeeds **only after database recovery completes**.
Liveness and readiness probes do not run until startup succeeds.
- **Readiness probe**
Indicates when the instance is ready to accept client traffic.
- **Liveness probe**
Determines when the container should be restarted if it becomes unresponsive.
---
**Default timing**
- On **data instances**, the startup probe must succeed within **2 hours**.
If recovery (e.g., from backup) may take longer, increase the timeout.
- Liveness and readiness probes must succeed **at least once every 5 minutes**
for the pod to be considered healthy.
---
**Probe endpoints**
- **Coordinators**: probed on the **NuRaft server**
- **Data instances**: probed on the **Bolt server**
> **Warning**
>
> **Breaking change (HA Helm chart 1.0.0)**: The probe target port is no longer configurable through
> `container.data.{readinessProbe,livenessProbe,startupProbe}.tcpSocket.port` or
> `container.coordinators.{readinessProbe,livenessProbe,startupProbe}.tcpSocket.port`.
> Probes are now hard-coded to a `tcpSocket` check against `ports.boltPort` for data
> instances and `ports.coordinatorPort` for coordinators. Only the probe timings
> (`failureThreshold`, `timeoutSeconds`, `periodSeconds`) remain configurable.
> Remove any `tcpSocket` overrides from your `values.yaml` and change the bolt or
> coordinator port via `ports.boltPort` / `ports.coordinatorPort` instead.
## Debugging
There are different ways in which you can debug Memgraph's HA cluster in
production. One way is to send us logs from all instances if you notice some
issue. That's why we advise users to set the log level to `TRACE` if possible.
Note however that running `TRACE` log level has some performance costs,
especially when logging to stderr in addition to files. If performance is your
concern, first set `commonArgs.data.logging.also_log_to_stderr` and
`commonArgs.coordinators.logging.also_log_to_stderr` to `false` since logging
to files is cheaper. If you're still unhappy with the performance overhead of
logging, set `commonArgs.{data,coordinators}.logging.log_level` to `DEBUG`
(higher log levels like `INFO` or `CRITICAL` are also fine) and keep
`also_log_to_stderr: true`. These settings replace the `--log-level`,
`--also-log-to-stderr`, `--log-file` and `--log-retention-days` flags that the
chart now appends to instance args automatically — setting them directly in
`data[].args` or `coordinators[].args` is rejected. Configure log retention via
`commonArgs.{data,coordinators}.logging.log_retention_days` (defaults to `35`).
By default, the chart provisions a dedicated log PVC for every data and
coordinator pod. If you only log to stderr and don't need a persistent log
volume, you can disable the log PVC by setting
`storage.data.createLogStorageClaim` and/or
`storage.coordinators.createLogStorageClaim` to `false`. When you do this you
must also set the corresponding `commonArgs.{data,coordinators}.logging.log_file`
to `""` to disable file logging — installing the chart with file logging
enabled but no log volume is rejected.
If you notice your application is crashing, you will be able to collect core
dumps by setting `storage.data.createCoreDumpsClaim` and
`storage.coordinators.createCoreDumpsClaim` to `true`. That will trigger the
creation of an init container which will be run in privileged mode as the root user
to set up all the necessary things on your nodes to be able to collect core
dumps. You can then create the debug pod and attach the PVC containing core dumps to
that pod to be able to extract core dumps outside of the K8s nodes. The example
of such a debug pod is the following YAML file:
```yaml
apiVersion: v1
kind: Pod
metadata:
name: debug-coredump
spec:
containers:
- name: debug
image: ubuntu:22.04
command: ["sleep", "infinity"]
volumeMounts:
- name: coredumps
mountPath: /var/core/memgraph
volumes:
- name: coredumps
persistentVolumeClaim:
claimName: memgraph-data-0-core-dumps-storage-memgraph-data-0-0
restartPolicy: Never
```
There is also a possibility of automatically uploading core dumps to S3. To do that, set `coreDumpUploader.enabled` to `true` and configure the S3 bucket,
AWS region, and credentials secret in the `coreDumpUploader` section. Note that the `createCoreDumpsClaim` flag for the relevant role (data/coordinators)
must also be set to `true`, as the uploader sidecar mounts the same PVC used for core dump storage. Core dumps are uploaded to
`s3://///`.
## Graceful termination
When a pod is stopped (e.g., during upgrades, rescheduling, or scale-down),
Kubernetes sends `SIGTERM` and waits up to `terminationGracePeriodSeconds`
for the container to exit cleanly before forcefully killing it with `SIGKILL`.
The HA chart defaults this value to **30 seconds** for both data
and coordinator pods, configurable via:
- `container.data.terminationGracePeriodSeconds`
- `container.coordinators.terminationGracePeriodSeconds`
since `--storage-snapshot-on-exit` is explicitly set to false by default.
### Using `--storage-snapshot-on-exit` with HA
If you enable the [`--storage-snapshot-on-exit`](https://memgraph.com/docs/database-management/configuration)
flag on data instances, Memgraph will attempt to create a full snapshot of the
database during shutdown. Snapshot creation time scales with dataset size and
can easily exceed the default grace period on larger deployments.
> **Warning**
>
> If `terminationGracePeriodSeconds` is shorter than the time needed to write the
> on-exit snapshot, Kubernetes will `SIGKILL` the Memgraph process mid-write,
> leaving the snapshot incomplete and defeating the purpose of the flag.
>
> When enabling `--storage-snapshot-on-exit`, set
> `container.data.terminationGracePeriodSeconds` to a value that comfortably
> covers the expected snapshot duration for your dataset. Benchmark the snapshot
> time on a representative dataset and add a safety margin.
## Monitoring
Memgraph HA integrates with Kubernetes monitoring tools through:
- The [kube-prometheus-stack](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack) Helm chart
- [Memgraph’s Prometheus exporter](https://github.com/memgraph/prometheus-exporter)
The chart `kube-prometheus-stack` should be installed independently from the HA
chart. Add the `prometheus-community` Helm repository and install (or upgrade) the
release into the `monitoring` namespace:
```bash
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm upgrade --install kube-prometheus-stack prometheus-community/kube-prometheus-stack \
--namespace monitoring --create-namespace
```
Remember the release name (`kube-prometheus-stack`) and the namespace (`monitoring`):
the HA chart needs both to wire its `ServiceMonitor` and Grafana dashboard to this
stack (see [`prometheus.serviceMonitor.kubePrometheusStackReleaseName`](#configuration-options)
and `prometheus.namespace` below).
To customize the stack, pass your own values file with `-f kube_prometheus_stack_values.yaml`.
A template is available in the upstream [chart’s
repository](https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/values.yaml).
If you scrape through `mg-exporter` (see the next section) you can additionally pass
`-f memgraph_prometheus.yaml`, a generic dashboard which shows the metrics
that the exporter exposes for both standalone and HA deployments. The dashboard file can be downloaded from
[here](https://github.com/memgraph/prometheus-exporter/blob/main/dashboards/memgraph_prometheus.yaml).
When scraping Memgraph directly, use the bundled **Memgraph OpenMetrics** dashboard
instead (see [Grafana dashboard for direct scraping](#grafana-dashboard-for-direct-scraping)).
If you install the `kube-prometheus-stack` in a **non-default namespace**, allow
cross-namespace scraping. You can allow this by adding the following
configuration to your `kube_prometheus_stack_values.yaml` file:
```yaml
prometheus:
prometheusSpec:
serviceMonitorSelectorNilUsesHelmValues: false
```
### Enable monitoring in the Memgraph HA chart
To enable the Memgraph Prometheus exporter and ServiceMonitor:
```yaml
prometheus:
enabled: true
namespace: monitoring
memgraphExporter:
port: 9115
pullFrequencySeconds: 5
repository: memgraph/mg-exporter
tag: 0.2.3
serviceMonitor:
enabled: true
kubePrometheusStackReleaseName: kube-prometheus-stack
interval: 15s
```
If you set `prometheus.enabled` to `false`, resources from
`charts/memgraph-high-availability/templates/mg-exporter.yaml` will still be
installed into the `monitoring` namespace.
`prometheus.serviceMonitor.enabled` defaults to `false`; set it to `true` only
when you have `kube-prometheus-stack` (or another Prometheus Operator) in the
cluster to consume the `ServiceMonitor` resource.
Refer to the configuration table later in the document for details on all
parameters.
#### mg-exporter TLS
When any data instance or coordinator has `tls.bolt.enabled: true`, the
chart automatically configures the mg-exporter to scrape that instance over
`https://` instead of `http://`. Each instance entry in the exporter config
also gets `skip_tls_verify` and (optionally) `ca_file` derived from
`prometheus.memgraphExporter.tls`:
```yaml
prometheus:
memgraphExporter:
tls:
skipVerify: true
caSecretName: ""
caSecretKey: ca.crt
```
- `skipVerify` — when `true` (default), the exporter does not verify the
Memgraph server certificate. Convenient for self-signed certs but not
suitable for production.
- `caSecretName` — name of a pre-created Secret holding the CA bundle that
signed Memgraph's certificate. When set and `skipVerify` is `false`, the
chart mounts the Secret at `/etc/mg-exporter/ssl` and passes
`ca_file=/etc/mg-exporter/ssl/` to the exporter.
- `caSecretKey` — key inside the Secret holding the CA certificate
(default `ca.crt`).
Example with strict CA verification:
```yaml
prometheus:
memgraphExporter:
tls:
skipVerify: false
caSecretName: bolt-ca-bundle
caSecretKey: ca.crt
```
### Scrape Memgraph directly (without mg-exporter)
From Memgraph 3.11, Memgraph can serve metrics in Prometheus
[OpenMetrics](https://prometheus.io/docs/specs/om/open_metrics_spec/) format on its
HTTP monitoring port, so the cluster can be scraped directly and `mg-exporter` can be
skipped entirely.
To start the HA chart with direct scraping you need:
1. **A Prometheus Operator in the cluster** – for example `kube-prometheus-stack`
installed as shown [above](#monitoring). The HA chart only creates a
`ServiceMonitor`; something must consume it.
2. **`scrapeMemgraphDirectly: true`** – the chart appends `--metrics-format=OpenMetrics`
to every coordinator and data instance and adds a `tcp-metrics-port` to each
instance `Service` so it can be discovered.
3. **`prometheus.enabled: false`** – no `mg-exporter` is deployed.
4. **`prometheus.serviceMonitor.enabled: true`** – provisions the `ServiceMonitor`
that targets each instance `Service` directly.
5. **Matching release name and namespace** – the `ServiceMonitor` is labelled
`release: `, which is how
`kube-prometheus-stack` selects it, and it is created in `prometheus.namespace`
(falls back to the release namespace). Both must match the `kube-prometheus-stack`
Helm release name and namespace you used when installing the stack.
A minimal values file for the default installation of `kube-prometheus-stack`
(release `kube-prometheus-stack` in namespace `monitoring`) looks like this:
```yaml
# Every coordinator and data instance serves OpenMetrics; the chart exposes their
# monitoring ports automatically.
scrapeMemgraphDirectly: true
prometheus:
enabled: false # no mg-exporter is deployed
namespace: monitoring # where kube-prometheus-stack lives; the ServiceMonitor is created here
serviceMonitor:
enabled: true # in-cluster: kube-prometheus-stack scrapes each instance directly
kubePrometheusStackReleaseName: kube-prometheus-stack # must equal the Helm release name
interval: 15s
grafanaDashboard:
enabled: true # optional: ship the "Memgraph OpenMetrics" dashboard
namespace: monitoring # a namespace the Grafana sidecar watches
```
The `ServiceMonitor` selects the instance `Services` in the HA chart's release
namespace through its `namespaceSelector`, so the HA chart and `kube-prometheus-stack`
can live in different namespaces without any extra configuration.
`scrapeMemgraphDirectly` only selects the metrics *format and target* — on its own it
does not scrape anything. You still enable a scraper:
- **In-cluster**: `prometheus.serviceMonitor.enabled: true` provisions a `ServiceMonitor`
that `kube-prometheus-stack` uses to scrape each Memgraph instance directly.
- **Remote**: `vmagentRemote.enabled: true` makes vmagent scrape the instances directly
and remote-write the metrics (see [Remote metrics and logs](#remote-metrics-and-logs)).
The same switch governs both paths. When it is `false`, those scrapers read the
`mg-exporter` instead, which requires `prometheus.enabled: true`.
> **Info**
>
> Direct scraping requires Memgraph >= 3.11. When `scrapeMemgraphDirectly: true`, the
> chart runs every coordinator and data instance with `--metrics-format=OpenMetrics` and
> exposes their monitoring ports automatically. The metric names differ from the legacy
> JSON exporter, so use the bundled **Memgraph OpenMetrics** dashboard.
After installing the HA chart, verify that the `ServiceMonitor` exists and that
Prometheus picked up the targets:
```bash
kubectl get servicemonitor -n monitoring
kubectl port-forward -n monitoring svc/kube-prometheus-stack-prometheus 9090:9090
```
Open `http://localhost:9090/targets` and look for the
`serviceMonitor/monitoring/-memgraph/0` job; every coordinator and data
instance should be listed as `UP`. If the job is missing, the `release` label or
namespace does not match your `kube-prometheus-stack` installation.
#### Grafana dashboard for direct scraping
Set `prometheus.grafanaDashboard.enabled: true` to ship the bundled "Memgraph
OpenMetrics" Grafana dashboard as a `ConfigMap` (labelled `grafana_dashboard`) for a
Grafana sidecar (e.g. the one bundled with `kube-prometheus-stack`) to auto-load. The
dashboard uses a datasource template variable, so it binds to Grafana's default
Prometheus. The `ConfigMap` must live in a namespace the sidecar watches — set
`prometheus.grafanaDashboard.namespace` to Grafana's namespace (or run the sidecar with
`searchNamespace: ALL`).
```yaml
prometheus:
grafanaDashboard:
enabled: true
namespace: monitoring
label: grafana_dashboard
```
A second dashboard, **Memgraph Logs**, covers the logs shipped by `vectorRemote` and is
enabled with `vectorRemote.grafanaDashboard.enabled: true` — see
[Grafana dashboards](#grafana-dashboards).
### Uninstall kube-prometheus-stack
```bash
helm uninstall kube-prometheus-stack --namespace monitoring
```
**Note:**
The stack’s CRDs are **not** deleted automatically and must be removed manually:
```bash
kubectl delete crd alertmanagerconfigs.monitoring.coreos.com
kubectl delete crd alertmanagers.monitoring.coreos.com
kubectl delete crd podmonitors.monitoring.coreos.com
kubectl delete crd probes.monitoring.coreos.com
kubectl delete crd prometheusagents.monitoring.coreos.com
kubectl delete crd prometheuses.monitoring.coreos.com
kubectl delete crd prometheusrules.monitoring.coreos.com
kubectl delete crd scrapeconfigs.monitoring.coreos.com
kubectl delete crd servicemonitors.monitoring.coreos.com
kubectl delete crd thanosrulers.monitoring.coreos.com
```
### Remote metrics and logs
The HA chart supports optional remote observability:
- `vmagentRemote` for shipping metrics with Prometheus `remote_write`
- `vectorRemote` sidecars for shipping Memgraph logs to Loki-compatible endpoints
Prerequisites:
- keep `prometheus.enabled: true` so `mg-exporter` is deployed — or set `scrapeMemgraphDirectly: true` with `prometheus.enabled: false` to have vmagent scrape Memgraph's OpenMetrics endpoint directly (no exporter; see [Scrape Memgraph directly](#scrape-memgraph-directly-without-mg-exporter))
- if you only need remote shipping and not local scraping, set `prometheus.serviceMonitor.enabled: false` to avoid duplicate scraping
- configure `vectorRemote.data` and/or `vectorRemote.coordinators` depending on which pod roles should ship logs
- when `vectorRemote.enabled: true`, add `--monitoring-port=` and `--monitoring-address=0.0.0.0` to each instance `args`
Example `values.yaml`:
```yaml
prometheus:
enabled: true
namespace: monitoring
serviceMonitor:
enabled: false
vmagentRemote:
enabled: true
namespace: monitoring
remoteWrite:
url: "https:///api/v1/write"
# Optional: only set basicAuth when your remote_write endpoint requires basic auth.
basicAuth:
secretName: monitoring-basic-auth
usernameKey: username
passwordKey: password
externalLabels:
cluster_id: "memgraph-testing-cluster-53"
service_name: "memgraph-ha"
cluster_env: "self-hosted-large-01"
vectorRemote:
enabled: true
data: true
coordinators: true
websocketPort: 7444
logsEndpoint: "https://"
# Optional: only set auth when your endpoint requires basic auth.
auth:
secretName: monitoring-basic-auth
usernameKey: username
passwordKey: password
extraLabels:
cluster_id: "memgraph-testing-cluster-53"
service_name: "memgraph-ha"
cluster_env: "self-hosted-large-01"
data:
- id: "0"
args:
- "--monitoring-port=7444"
- "--monitoring-address=0.0.0.0"
- id: "1"
args:
- "--monitoring-port=7444"
- "--monitoring-address=0.0.0.0"
coordinators:
- id: "1"
args:
- "--monitoring-port=7444"
- "--monitoring-address=0.0.0.0"
- id: "2"
args:
- "--monitoring-port=7444"
- "--monitoring-address=0.0.0.0"
- id: "3"
args:
- "--monitoring-port=7444"
- "--monitoring-address=0.0.0.0"
```
> **Info**
>
> The chart auto-appends `--bolt-port`, `--management-port`, `--coordinator-port`,
> `--coordinator-id`, `--coordinator-hostname`, `--data-directory`, `--log-level`,
> `--also-log-to-stderr`, `--log-file`, `--bolt-cert-file`, `--bolt-key-file`,
> `--cluster-cert-file`, `--cluster-key-file` and `--cluster-ca-file`
> from `ports.*`, `commonArgs.{data,coordinators}.logging.*` and the per-instance
> `tls.bolt.*` / `tls.intraCluster.*` blocks. Setting any of these in `data[].args` or
> `coordinators[].args` causes `helm install` to fail with a template error.
Create credentials secret in the namespace where vmagent runs (usually `monitoring`):
```bash
kubectl create secret generic monitoring-basic-auth -n monitoring \
--from-literal=username='' \
--from-literal=password=''
```
For HA Vector sidecars, create the same secret in the Memgraph release namespace as well:
```bash
kubectl create secret generic monitoring-basic-auth -n \
--from-literal=username='' \
--from-literal=password=''
```
#### Kubernetes infrastructure metrics
`vmagentRemote` can additionally scrape Kubernetes infrastructure metrics
(`kube-state-metrics`, `node-exporter`, `kubelet`) required by
`kube-prometheus-stack` Kubernetes and Node dashboards, and remote-write them
to your centralized monitoring cluster.
Enable Kubernetes scraping by extending your existing `vmagentRemote` values:
```yaml
vmagentRemote:
# ... existing fields (enabled, remoteWrite, externalLabels) ...
kubernetes:
enabled: true
kubeStateMetrics:
enabled: true
jobName: kube-state-metrics
targets:
- kube-prometheus-stack-kube-state-metrics.monitoring.svc.cluster.local:8080
nodeExporter:
enabled: true
jobName: node-exporter
targets:
- kube-prometheus-stack-prometheus-node-exporter.monitoring.svc.cluster.local:9100
kubelet:
enabled: true
jobName: kubelet
metricsPath: /metrics/cadvisor
apiServerAddress: kubernetes.default.svc:443
insecureSkipVerify: false
```
Notes:
- RBAC and `ServiceAccount` resources are created only when an enabled scrape
job requires Kubernetes API access (for example `kubelet.enabled=true` or
`nodeExporter.useKubernetesDiscovery=true`).
- Keep `jobName` values aligned with dashboard and recording-rule expectations
unless you also update those queries.
- Dashboards that rely on precomputed recording-rule series still require
rule evaluation in your monitoring stack.
A ready-to-use example values file is available in the Helm charts repository:
[`examples/remote-monitoring/values-ha-k8s-metrics.yaml`](https://github.com/memgraph/helm-charts/blob/main/examples/remote-monitoring/values-ha-k8s-metrics.yaml).
#### Grafana dashboards
The chart bundles two dashboards, each shipped as its own `ConfigMap` (labelled
`grafana_dashboard`) for a Grafana sidecar to auto-load, and each behind its own flag:
| Dashboard | Enable with | Shows |
| --------- | ----------- | ----- |
| **Memgraph OpenMetrics** | `prometheus.grafanaDashboard.enabled: true` | The metrics `vmagentRemote` scrapes and remote-writes |
| **Memgraph Logs** | `vectorRemote.grafanaDashboard.enabled: true` | The logs `vectorRemote` ships to VictoriaLogs |
```yaml
prometheus:
grafanaDashboard:
enabled: true
namespace: monitoring # must be a namespace the Grafana sidecar watches
label: grafana_dashboard
vectorRemote:
grafanaDashboard:
enabled: true
namespace: monitoring
label: grafana_dashboard
```
Each `ConfigMap` must live in a namespace the sidecar watches — set `namespace` to
Grafana's namespace (or run the sidecar with `searchNamespace: ALL`). Both default to
`prometheus.namespace`, else the release namespace. The metrics dashboard is also covered
under [Grafana dashboard for direct scraping](#grafana-dashboard-for-direct-scraping).
Both use datasource template variables rather than hardcoded UIDs, so they bind to the
datasources you select. The OpenMetrics dashboard needs a Prometheus-compatible one
("Data source"); the logs dashboard needs a VictoriaLogs one ("Logs data source") for its
panels plus a Prometheus-compatible one ("Metrics data source") that populates its filter
variables. Both filter on the `cluster_id`, `cluster_env` and `service_name` labels you set
in `vmagentRemote.externalLabels` and `vectorRemote.extraLabels`, and break metrics and logs
down per instance, so you can compare coordinators and data instances in the same view.
If you do not run a Grafana sidecar, both dashboards can be imported by hand via
*Dashboards* → *New* → *Import* in Grafana — they live in the Helm charts repository under
[`charts/memgraph-high-availability/dashboards/`](https://github.com/memgraph/helm-charts/tree/main/charts/memgraph-high-availability/dashboards).
## Configuration options
The following table lists the configurable parameters of the Memgraph HA chart
and their default values.
| Parameter | Description | Default |
| ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | ------------------------------ |
| `image.repository` | Memgraph Docker image repository | `docker.io/memgraph/memgraph` |
| `image.tag` | Specific tag for the Memgraph Docker image. Overrides the image tag whose default is chart version. | `3.1.0` |
| `image.pullPolicy` | Image pull policy | `IfNotPresent` |
| `memgraphUserId` | The user id that is hardcoded in Memgraph and Mage images | `101` |
| `memgraphGroupId` | The group id that is hardcoded in Memgraph and Mage images | `103` |
| `clusterDomain` | Kubernetes cluster domain used to build the internal FQDN service addresses (`..svc.`) of data instances and coordinators. Override if your cluster uses a custom domain. | `cluster.local` |
| `storage.data.libPVCSize` | Size of the lib storage PVC for data instances | `1Gi` |
| `storage.data.libStorageAccessMode` | Access mode used for lib storage on data instances | `ReadWriteOnce` |
| `storage.data.libStorageClassName` | The name of the storage class used for storing data on data instances | `""` |
| `storage.data.createLogStorageClaim` | Create a PVC for logs on data instances. When `false`, `commonArgs.data.logging.log_file` must be `""`. | `true` |
| `storage.data.logPVCSize` | Size of the log PVC for data instances | `1Gi` |
| `storage.data.logStorageAccessMode` | Access mode used for log storage on data instances | `ReadWriteOnce` |
| `storage.data.logStorageClassName` | The name of the storage class used for storing logs on data instances | `""` |
| `storage.data.createCoreDumpsClaim` | Create a PVC for core dumps on data instances | `false` |
| `storage.data.coreDumpsStorageClassName` | Storage class name for core dumps PVC on data instances | `""` |
| `storage.data.coreDumpsStorageSize` | Size of the core dumps PVC on data instances | `10Gi` |
| `storage.data.coreDumpsMountPath` | Mount path for core dumps on data instances | `/var/core/memgraph` |
| `storage.data.coreDumpsImage.repository` | Image repository for the data instance core-dumps init container. | `docker.io/library/busybox` |
| `storage.data.coreDumpsImage.tag` | Image tag for the data instance core-dumps init container. | `1.37.0` |
| `storage.data.coreDumpsImage.pullPolicy` | Image pull policy for the data instance core-dumps init container. | `IfNotPresent` |
| `storage.data.extraVolumes` | Additional volumes to add to data instance pods | `[]` |
| `storage.data.extraVolumeMounts` | Additional volume mounts to add to data instance containers | `[]` |
| `storage.coordinators.libPVCSize` | Size of the lib storage PVC for coordinators | `1Gi` |
| `storage.coordinators.libStorageAccessMode` | Access mode used for lib storage on coordinators | `ReadWriteOnce` |
| `storage.coordinators.libStorageClassName` | The name of the storage class used for storing data on coordinators | `""` |
| `storage.coordinators.createLogStorageClaim` | Create a PVC for logs on coordinators. When `false`, `commonArgs.coordinators.logging.log_file` must be `""`.| `true` |
| `storage.coordinators.logPVCSize` | Size of the log PVC for coordinators | `1Gi` |
| `storage.coordinators.logStorageAccessMode` | Access mode used for log storage on coordinators | `ReadWriteOnce` |
| `storage.coordinators.logStorageClassName` | The name of the storage class used for storing logs on coordinators | `""` |
| `storage.coordinators.createCoreDumpsClaim` | Create a PVC for core dumps on coordinators | `false` |
| `storage.coordinators.coreDumpsStorageClassName` | Storage class name for core dumps PVC on coordinators | `""` |
| `storage.coordinators.coreDumpsStorageSize` | Size of the core dumps PVC on coordinators | `10Gi` |
| `storage.coordinators.coreDumpsMountPath` | Mount path for core dumps on coordinators | `/var/core/memgraph` |
| `storage.coordinators.coreDumpsImage.repository` | Image repository for the coordinator core-dumps init container. | `docker.io/library/busybox` |
| `storage.coordinators.coreDumpsImage.tag` | Image tag for the coordinator core-dumps init container. | `1.37.0` |
| `storage.coordinators.coreDumpsImage.pullPolicy` | Image pull policy for the coordinator core-dumps init container. | `IfNotPresent` |
| `storage.coordinators.extraVolumes` | Additional volumes to add to coordinator pods | `[]` |
| `storage.coordinators.extraVolumeMounts` | Additional volume mounts to add to coordinator containers | `[]` |
| `externalAccessConfig.coordinator.serviceType` | IngressNginx, NodePort, CommonLoadBalancer or LoadBalancer. By default, no external service will be created. | `""` |
| `externalAccessConfig.coordinator.annotations` | Annotations for external services attached to coordinators. | `{}` |
| `externalAccessConfig.dataInstance.serviceType` | IngressNginx, NodePort or LoadBalancer. By default, no external service will be created. | `""` |
| `externalAccessConfig.dataInstance.annotations` | Annotations for external services attached to data instances. | `{}` |
| `externalAccessConfig.ingress.dataPortBase` | Base port for data instance ports on the IngressNginx controller (`dataPortBase + data instance id`). | `9000` |
| `externalAccessConfig.ingress.coordinatorPortBase` | Base port for coordinator ports on the IngressNginx controller (`coordinatorPortBase + coordinator id`). Kept well above `dataPortBase` so the data and coordinator port ranges never overlap. | `10000` |
| `externalAccessConfig.gateway.enabled` | Enable Gateway API external access. Only exposes tiers whose `serviceType` is empty; fails rendering if both tiers have a `serviceType` set. | `false` |
| `externalAccessConfig.gateway.gatewayClassName` | Name of a pre-existing GatewayClass. Required when creating a new Gateway. | `""` |
| `externalAccessConfig.gateway.existingGatewayName` | Name of an existing Gateway to attach routes to. Skips Gateway creation. | `""` |
| `externalAccessConfig.gateway.existingGatewayNamespace` | Namespace of the existing Gateway. Defaults to release namespace. | `""` |
| `externalAccessConfig.gateway.annotations` | Annotations for the Gateway resource. | `{}` |
| `externalAccessConfig.gateway.labels` | Labels for the Gateway resource. | `{}` |
| `externalAccessConfig.gateway.dataPortBase` | Base port for data instance Gateway listeners (`dataPortBase + data instance id`). Must not resolve to `ports.boltPort`, which the shared coordinators listener uses. | `9000` |
| `headlessService.enabled` | Specifies whether headless services will be used inside K8s network on all instances. | `false` |
| `ports.boltPort` | Bolt port used on coordinator and data instances. Also the port of the shared coordinators Gateway listener. | `7687` |
| `ports.managementPort` | Management port used on coordinator and data instances. | `10000` |
| `ports.replicationPort` | Replication port used on data instances. | `20000` |
| `ports.coordinatorPort` | Coordinator port used on coordinators. | `12000` |
| `ports.metricsPort` | Metrics port for coordinators and data instances. Opened only if `prometheus.enabled` is set to `true`. | `9091` |
| `affinity.topologyKey` | Topology domain used by all pod affinity and anti-affinity rules (for example a node or an availability zone) | `kubernetes.io/hostname` |
| `affinity.unique` | Schedule pods on different nodes in the cluster | `false` |
| `affinity.parity` | Schedule pods on the same node with maximum one coordinator and one data node | `false` |
| `affinity.nodeSelection` | Schedule pods on nodes with specific labels | `false` |
| `affinity.roleLabelKey` | Label key for node selection | `role` |
| `affinity.dataNodeLabelValue` | Label value for data nodes | `data-node` |
| `affinity.coordinatorNodeLabelValue` | Label value for coordinator nodes | `coordinator-node` |
| `container.data.terminationGracePeriodSeconds` | Grace period for data instance pod termination | `1800` |
| `container.data.livenessProbe.failureThreshold` | Failure threshold for liveness probe | `20` |
| `container.data.livenessProbe.timeoutSeconds` | Timeout for liveness probe | `10` |
| `container.data.livenessProbe.periodSeconds` | Period seconds for liveness probe | `5` |
| `container.data.readinessProbe.failureThreshold` | Failure threshold for readiness probe | `20` |
| `container.data.readinessProbe.timeoutSeconds` | Timeout for readiness probe | `10` |
| `container.data.readinessProbe.periodSeconds` | Period seconds for readiness probe | `5` |
| `container.data.startupProbe.failureThreshold` | Failure threshold for startup probe | `1440` |
| `container.data.startupProbe.timeoutSeconds` | Timeout for probe | `10` |
| `container.data.startupProbe.periodSeconds` | Period seconds for startup probe | `10` |
| `container.data.terminationGracePeriodSeconds` | Grace period for data pod termination. Increase when `--storage-snapshot-on-exit` is enabled so the snapshot has time to finish. | `30` |
| `container.coordinators.livenessProbe.failureThreshold` | Failure threshold for liveness probe | `20` |
| `container.coordinators.livenessProbe.timeoutSeconds` | Timeout for liveness probe | `10` |
| `container.coordinators.livenessProbe.periodSeconds` | Period seconds for liveness probe | `5` |
| `container.coordinators.readinessProbe.failureThreshold` | Failure threshold for readiness probe | `20` |
| `container.coordinators.readinessProbe.timeoutSeconds` | Timeout for readiness probe | `10` |
| `container.coordinators.readinessProbe.periodSeconds` | Period seconds for readiness probe | `5` |
| `container.coordinators.startupProbe.failureThreshold` | Failure threshold for startup probe | `20` |
| `container.coordinators.startupProbe.timeoutSeconds` | Timeout for probe | `10` |
| `container.coordinators.startupProbe.periodSeconds` | Period seconds for startup probe | `10` |
| `container.coordinators.terminationGracePeriodSeconds` | Grace period for coordinators pod termination. | `30`. |
| `data` | Configuration for data instances | See `data` section |
| `coordinators` | Configuration for coordinator instances | See `coordinators` section |
| `sysctlInitContainer.enabled` | Enable the init container to set sysctl parameters | `true` |
| `sysctlInitContainer.maxMapCount` | Value for `vm.max_map_count` to be set by the init container | `524288` |
| `sysctlInitContainer.image.repository` | Image repository for the sysctl init container | `library/busybox` |
| `sysctlInitContainer.image.tag` | Image tag for the sysctl init container | `1.37.0` |
| `sysctlInitContainer.image.pullPolicy` | Image pull policy for the sysctl init container | `IfNotPresent` |
| `fixOwnershipInitContainer.enabled` | Enable the init container that `chown`s lib/log/core-dump mounts to `memgraphUserId:memgraphGroupId` before Memgraph starts. Use when the storage driver does not honor `fsGroup`. | `false` |
| `fixOwnershipInitContainer.image.repository` | Image repository for the fix-ownership init container. | `docker.io/library/busybox` |
| `fixOwnershipInitContainer.image.tag` | Image tag for the fix-ownership init container. | `1.37.0` |
| `fixOwnershipInitContainer.image.pullPolicy` | Image pull policy for the fix-ownership init container. | `IfNotPresent` |
| `secrets.name` | Name of the Kubernetes Secret holding the Memgraph Enterprise license and organization name. Must exist before `helm install`. | `memgraph-secrets` |
| `secrets.licenseKey` | Key in the Secret whose value is exposed as `MEMGRAPH_ENTERPRISE_LICENSE` to data and coordinator pods. | `MEMGRAPH_ENTERPRISE_LICENSE` |
| `secrets.organizationKey` | Key in the Secret whose value is exposed as `MEMGRAPH_ORGANIZATION_NAME` to data and coordinator pods. | `MEMGRAPH_ORGANIZATION_NAME` |
| `resources.coordinators` | CPU/Memory resource requests/limits for coordinators. Left empty by default. | `{}` |
| `resources.data` | CPU/Memory resource requests/limits for data instances. Left empty by default. | `{}` |
| `scrapeMemgraphDirectly` | Serve metrics in OpenMetrics format and scrape Memgraph directly instead of `mg-exporter` (requires Memgraph >= 3.11). Only selects the format/target — still enable a scraper via `prometheus.serviceMonitor.enabled` and/or `vmagentRemote.enabled`. | `false` |
| `prometheus.enabled` | If set to `true`, K8s resources representing Memgraph's Prometheus exporter will be deployed. | `false` |
| `prometheus.namespace` | Namespace in which `kube-prometheus-stack` is installed. The `ServiceMonitor`, the Grafana dashboard `ConfigMap` and (when `prometheus.enabled` is `true`) the exporter resources are created here. When empty, the release namespace is used. | `""` |
| `prometheus.memgraphExporter.port` | The port on which Memgraph's Prometheus exporter is available. | `9115` |
| `prometheus.memgraphExporter.pullFrequencySeconds` | How often will Memgraph's Prometheus exporter pull data from Memgraph instances. | `5` |
| `prometheus.memgraphExporter.repository` | The repository where Memgraph's Prometheus exporter image is available. | `docker.io/memgraph/prometheus-exporter` |
| `prometheus.memgraphExporter.tag` | The tag of Memgraph's Prometheus exporter image. | `0.2.3` |
| `prometheus.memgraphExporter.tls.skipVerify` | When `true`, mg-exporter does not verify Memgraph's server certificate. Only applied when scraping instances with `tls.bolt.enabled=true`. | `true` |
| `prometheus.memgraphExporter.tls.caSecretName` | Name of a pre-created Secret containing the CA bundle. When set (and `skipVerify=false`), the chart mounts it at `/etc/mg-exporter/ssl`. | `""` |
| `prometheus.memgraphExporter.tls.caSecretKey` | Key inside the Secret holding the CA certificate. | `ca.crt` |
| `prometheus.memgraphExporter.extraVolumes` | Additional volumes mounted on the `mg-exporter` Deployment (e.g. ConfigMaps with custom exporter configs). | `[]` |
| `prometheus.memgraphExporter.extraVolumeMounts` | Additional volume mounts for the `mg-exporter` container. | `[]` |
| `prometheus.serviceMonitor.enabled` | If enabled, a `ServiceMonitor` object will be deployed. | `false` |
| `prometheus.serviceMonitor.kubePrometheusStackReleaseName` | The release name under which `kube-prometheus-stack` chart is installed. Rendered as the `release` label on the `ServiceMonitor`, which is how the stack discovers it. | `kube-prometheus-stack` |
| `prometheus.serviceMonitor.interval` | How often will Prometheus pull data from Memgraph's Prometheus exporter. | `15s` |
| `prometheus.grafanaDashboard.enabled` | Ship the bundled "Memgraph OpenMetrics" Grafana dashboard as a `ConfigMap` for a Grafana sidecar to auto-load. | `false` |
| `prometheus.grafanaDashboard.namespace` | Namespace for the dashboard `ConfigMap`; must be one the Grafana sidecar watches. Defaults to `prometheus.namespace`, else release namespace. | `""` |
| `prometheus.grafanaDashboard.label` | Label the Grafana sidecar selects dashboards by. | `grafana_dashboard` |
| `vmagentRemote.enabled` | Deploy a vmagent Deployment that scrapes mg-exporter (or Memgraph directly when `scrapeMemgraphDirectly=true`) and remote-writes to a Prometheus-compatible endpoint. | `false` |
| `vmagentRemote.namespace` | Namespace for the vmagent Deployment and its resources. Defaults to `prometheus.namespace` when empty. | `""` |
| `vmagentRemote.image.repository` | vmagent image repository. | `victoriametrics/vmagent` |
| `vmagentRemote.image.tag` | vmagent image tag. | `v1.139.0` |
| `vmagentRemote.image.pullPolicy` | vmagent image pull policy. | `IfNotPresent` |
| `vmagentRemote.remoteWrite.url` | Prometheus `remote_write` URL. Required when `vmagentRemote.enabled=true`. | `""` |
| `vmagentRemote.remoteWrite.basicAuth.secretName` | Kubernetes Secret holding basic-auth credentials for `remote_write`. When empty, basic auth is not configured. | `""` |
| `vmagentRemote.remoteWrite.basicAuth.usernameKey` | Key in the basic-auth Secret holding the username. | `username` |
| `vmagentRemote.remoteWrite.basicAuth.passwordKey` | Key in the basic-auth Secret holding the password. | `password` |
| `vmagentRemote.scrapeInterval` | Global `scrape_interval` applied to vmagent scrape jobs. | `15s` |
| `vmagentRemote.externalLabels` | External labels attached to every scraped sample before remote-write. | `{}` |
| `vmagentRemote.resources` | Resource requests/limits for the vmagent container. | `{}` |
| `vmagentRemote.httpPort` | vmagent local HTTP listen port for metrics/debug (the remote-write target is `remoteWrite.url`). | `8429` |
| `vmagentRemote.kubernetes.enabled` | Enable scraping of Kubernetes infrastructure metrics used by kube-prometheus dashboards. | `false` |
| `vmagentRemote.kubernetes.kubeStateMetrics.enabled` | Scrape `kube-state-metrics`. | `true` |
| `vmagentRemote.kubernetes.kubeStateMetrics.jobName` | Prometheus `job` label for kube-state-metrics. Keep aligned with dashboard/recording-rule expectations. | `kube-state-metrics` |
| `vmagentRemote.kubernetes.kubeStateMetrics.targets` | Static scrape targets for kube-state-metrics. | `[kube-prometheus-stack-kube-state-metrics.monitoring.svc.cluster.local:8080]` |
| `vmagentRemote.kubernetes.nodeExporter.enabled` | Scrape `node-exporter`. | `true` |
| `vmagentRemote.kubernetes.nodeExporter.jobName` | Prometheus `job` label for node-exporter. | `node-exporter` |
| `vmagentRemote.kubernetes.nodeExporter.useKubernetesDiscovery` | Discover node-exporter pods via Kubernetes SD so namespace/pod/node labels are present for recording rules. | `false` |
| `vmagentRemote.kubernetes.nodeExporter.podMetricsPort` | Pod port used by Kubernetes SD to match node-exporter pods. | `"9100"` |
| `vmagentRemote.kubernetes.nodeExporter.appNameLabel` | Expected value of `app.kubernetes.io/name` on node-exporter pods. | `prometheus-node-exporter` |
| `vmagentRemote.kubernetes.nodeExporter.appInstanceLabel` | Expected value of `app.kubernetes.io/instance` on node-exporter pods. | `kube-prometheus-stack-prometheus-node-exporter` |
| `vmagentRemote.kubernetes.nodeExporter.targets` | Static fallback targets for node-exporter when `useKubernetesDiscovery=false`. | `[kube-prometheus-stack-prometheus-node-exporter.monitoring.svc.cluster.local:9100]` |
| `vmagentRemote.kubernetes.kubelet.enabled` | Scrape kubelet metrics via the Kubernetes API server node proxy. | `true` |
| `vmagentRemote.kubernetes.kubelet.jobName` | Prometheus `job` label for kubelet. Keep as `kubelet` so kube-prometheus dashboards and rules still match. | `kubelet` |
| `vmagentRemote.kubernetes.kubelet.metricsPath` | Metrics path for the primary kubelet scrape (cAdvisor). | `/metrics/cadvisor` |
| `vmagentRemote.kubernetes.kubelet.additionalMetricsEnabled` | Enable a second kubelet scrape job for `/metrics` alongside the cAdvisor job. | `true` |
| `vmagentRemote.kubernetes.kubelet.additionalJobName` | Prometheus `job` label for the additional kubelet scrape. | `kubelet-metrics` |
| `vmagentRemote.kubernetes.kubelet.additionalMetricsPath` | Metrics path for the additional kubelet scrape. | `/metrics` |
| `vmagentRemote.kubernetes.kubelet.apiServerAddress` | Kubernetes API server address used to proxy kubelet scrapes. | `kubernetes.default.svc:443` |
| `vmagentRemote.kubernetes.kubelet.insecureSkipVerify` | Skip TLS verification of the kube-apiserver serving cert when scraping kubelet. | `false` |
| `vectorRemote.enabled` | Add a Vector sidecar that ships Memgraph logs from its monitoring websocket to a Loki-compatible endpoint. | `false` |
| `vectorRemote.data` | Add the Vector sidecar to data instance pods. | `true` |
| `vectorRemote.coordinators` | Add the Vector sidecar to coordinator pods. | `true` |
| `vectorRemote.websocketPort` | Memgraph monitoring websocket port used by both the chart args and the Vector sidecars. | `7444` |
| `vectorRemote.image.repository` | Vector image repository. | `timberio/vector` |
| `vectorRemote.image.tag` | Vector image tag. | `0.49.0-debian` |
| `vectorRemote.image.pullPolicy` | Vector image pull policy. | `IfNotPresent` |
| `vectorRemote.logsEndpoint` | Loki-compatible logs endpoint base URL; Vector's loki sink appends `/loki/api/v1/push`. Required when `vectorRemote.enabled=true`. | `""` |
| `vectorRemote.auth.secretName` | Kubernetes Secret holding basic-auth credentials for the logs endpoint. When empty, basic auth is not configured. | `""` |
| `vectorRemote.auth.usernameKey` | Key in the basic-auth Secret holding the username. | `username` |
| `vectorRemote.auth.passwordKey` | Key in the basic-auth Secret holding the password. | `password` |
| `vectorRemote.extraLabels` | Extra labels attached to every log event shipped by Vector. | `{}` |
| `vectorRemote.grafanaDashboard.enabled` | Ship the bundled "Memgraph Logs" Grafana dashboard as a `ConfigMap` for a Grafana sidecar to auto-load. | `false` |
| `vectorRemote.grafanaDashboard.namespace` | Namespace for the dashboard `ConfigMap`; must be one the Grafana sidecar watches. Defaults to `prometheus.namespace`, else release namespace. | `""` |
| `vectorRemote.grafanaDashboard.label` | Label the Grafana sidecar selects dashboards by. | `grafana_dashboard` |
| `vectorRemote.resources` | Resource requests/limits for the Vector sidecar container. | `{}` |
| `labels.coordinators.podLabels` | Enables you to set labels on a pod level for coordinators. | `{}` |
| `labels.coordinators.statefulSetLabels` | Enables you to set labels on a stateful set level for coordinators. | `{}` |
| `labels.coordinators.serviceLabels` | Enables you to set labels on a service level for coordinators. | `{}` |
| `labels.data.podLabels` | Enables you to set labels on a pod level for data instances. | `{}` |
| `labels.data.statefulSetLabels` | Enables you to set labels on a stateful set level for data instances. | `{}` |
| `labels.data.serviceLabels` | Enables you to set labels on a service level for data instances. | `{}` |
| `clusterSetup.labels` | Custom labels applied to both the post-install `cluster-setup` Job and its pod template. | `{}` |
| `clusterSetup.podSecurityContext` | Pod-level security context for the `cluster-setup` Job pod. See [Customize the cluster-setup Job](#customize-the-cluster-setup-job). | `runAsUser: 101`, `runAsGroup: 103`, `runAsNonRoot: true`, `seccompProfile.type: RuntimeDefault` |
| `clusterSetup.containerSecurityContext` | Container-level security context for the `cluster-setup` container. See [Customize the cluster-setup Job](#customize-the-cluster-setup-job). | `allowPrivilegeEscalation: false`, `capabilities.drop: ["ALL"]`, `readOnlyRootFilesystem: true`, `runAsNonRoot: true`, `seccompProfile.type: RuntimeDefault` |
| `updateStrategy.type` | Update strategy for StatefulSets. Possible values are `RollingUpdate` and `OnDelete` | `RollingUpdate` |
| `extraEnv.data` | Env variables that users can define and are applied to data instances | `[]` |
| `extraEnv.coordinators` | Env variables that users can define and are applied to coordinators | `[]` |
| `commonArgs.data.logging.log_level` | Log level applied to every data instance via `--log-level`. Must not be empty. | `TRACE` |
| `commonArgs.data.logging.also_log_to_stderr` | When `true`, appends `--also-log-to-stderr` to every data instance. Must be a boolean. | `true` |
| `commonArgs.data.logging.log_file` | Log-file path applied to every data instance via `--log-file`. Empty disables file logging. | `/var/log/memgraph/memgraph.log` |
| `commonArgs.data.logging.log_retention_days` | Number of days to retain log files on every data instance via `--log-retention-days`. | `35` |
| `commonArgs.coordinators.logging.log_level` | Log level applied to every coordinator via `--log-level`. Must not be empty. | `TRACE` |
| `commonArgs.coordinators.logging.also_log_to_stderr` | When `true`, appends `--also-log-to-stderr` to every coordinator. Must be a boolean. | `true` |
| `commonArgs.coordinators.logging.log_file` | Log-file path applied to every coordinator via `--log-file`. Empty disables file logging. | `/var/log/memgraph/memgraph.log` |
| `commonArgs.coordinators.logging.log_retention_days` | Number of days to retain log files on every coordinator via `--log-retention-days`. | `35` |
| `userContainers.data` | Additional sidecar containers for data instance pods | `[]` |
| `userContainers.coordinators` | Additional sidecar containers for coordinator pods | `[]` |
| `tolerations.data` | Tolerations for data instance pods | `[]` |
| `tolerations.coordinators` | Tolerations for coordinator pods | `[]` |
| `initContainers.data` | Init containers that users can define that will be applied to data instances. | `[]` |
| `initContainers.coordinators` | Init containers that users can define that will be applied to coordinators. | `[]` |
| `coreDumpUploader.enabled` | Enable the core dump S3 uploader sidecar. Requires `storage..createCoreDumpsClaim` to be `true`. | `false` |
| `coreDumpUploader.image.repository` | Docker image repository for the uploader sidecar | `amazon/aws-cli` |
| `coreDumpUploader.image.tag` | Docker image tag for the uploader sidecar | `2.33.28` |
| `coreDumpUploader.image.pullPolicy` | Image pull policy for the uploader sidecar | `IfNotPresent` |
| `coreDumpUploader.s3BucketName` | S3 bucket name where core dumps will be uploaded | `""` |
| `coreDumpUploader.s3Prefix` | S3 key prefix (folder) for uploaded core dumps | `core-dumps` |
| `coreDumpUploader.awsRegion` | AWS region of the S3 bucket | `us-east-1` |
| `coreDumpUploader.pollIntervalSeconds` | How often (in seconds) the sidecar checks for new core dump files | `30` |
| `coreDumpUploader.secretName` | Name of the K8s Secret containing AWS credentials | `aws-s3-credentials` |
| `coreDumpUploader.accessKeySecretKey` | Key in the K8s Secret for `AWS_ACCESS_KEY_ID` | `AWS_ACCESS_KEY_ID` |
| `coreDumpUploader.secretAccessKeySecretKey` | Key in the K8s Secret for `AWS_SECRET_ACCESS_KEY` | `AWS_SECRET_ACCESS_KEY` |
For the `data` and `coordinators` sections, each item in the list has the
following parameters:
| Parameter | Description | Default |
|---------------------------------------------|-----------------------------------------------------------------------------------------------------|-----------------------------------------|
| `id` | ID of the instance | `0` for data, `1` for coordinators |
| `internalAccessAnnotations` | Per-instance annotations for the internal ClusterIP Service. | `{}` |
| `externalAccessAnnotations` | Per-instance annotations for the external access Service, merged with global annotations. | `{}` |
| `extraVolumes` | Additional volumes available only to this instance's pod, added on top of `storage..extraVolumes`. Rendered through `tpl`, so Helm templating is supported.| `[]` |
| `extraVolumeMounts` | Additional volume mounts for this instance's Memgraph container. Each `name` must match an entry in this instance's `extraVolumes` or in `storage..extraVolumes`.| `[]` |
| `tls.bolt.enabled` | Enable Bolt TLS termination on this instance. The chart auto-appends `--bolt-cert-file` / `--bolt-key-file` and mounts the certificate Secret at `/etc/memgraph/ssl`. | `false` |
| `tls.bolt.secretName` | Name of a pre-existing Kubernetes Secret holding the Bolt TLS certificate and private key. Required when `tls.bolt.enabled=true`. | `bolt-tls-secret` |
| `tls.bolt.certSecretPath` | Key inside the Secret holding the TLS certificate. | `tls.crt` |
| `tls.bolt.keySecretPath` | Key inside the Secret holding the TLS private key. | `tls.key` |
| `tls.intraCluster.enabled` | Enable TLS on internal cluster communication for this instance. The chart auto-appends `--cluster-cert-file` / `--cluster-key-file` / `--cluster-ca-file` and mounts the certificate Secret at `/etc/memgraph/intra_cluster_tls`. | `false` |
| `tls.intraCluster.secretName` | Name of a pre-existing Kubernetes Secret holding the intra-cluster TLS certificate, private key and CA bundle. Required when `tls.intraCluster.enabled=true`. | `intra-tls--secret` |
| `tls.intraCluster.certSecretPath` | Key inside the Secret holding the TLS certificate. | `tls.crt` |
| `tls.intraCluster.keySecretPath` | Key inside the Secret holding the TLS private key. | `tls.key` |
| `tls.intraCluster.caSecretPath` | Key inside the Secret holding the CA bundle. | `ca.crt` |
| `args` | Per-instance Memgraph CLI flags. Append-only — see the note below for flags the chart manages. | `["--storage-snapshot-on-exit=false"]` for data, `[]` for coordinators |
The `args` field accepts any Memgraph CLI flag **except** the following, which
the chart appends automatically and rejects when set per-instance:
`--bolt-port`, `--management-port`, `--coordinator-port`, `--coordinator-id`,
`--coordinator-hostname`, `--data-directory`, `--log-level`,
`--also-log-to-stderr`, `--log-file`, `--bolt-cert-file`, `--bolt-key-file`,
`--cluster-cert-file`, `--cluster-key-file` and `--cluster-ca-file`.
Configure those through `ports.*`, `commonArgs.{data,coordinators}.logging.*`
and the per-instance `tls.bolt.*` / `tls.intraCluster.*` blocks instead.
For all available database settings, refer to the [configuration settings
docs](https://memgraph.com/docs/database-management/configuration).
## In-Service Software Upgrade (ISSU)
Memgraph’s **High Availability** supports in-service software upgrades (ISSU).
This guide explains the process when using [HA Helm
charts](https://memgraph.com/docs/getting-started/install-memgraph/kubernetes#memgraph-high-availability-helm-chart).
The procedure is very similar for native deployments.
> **Info**
>
> Some Memgraph versions require additional upgrade steps beyond the standard ISSU
> procedure. Check the [Migrating to v3.9 HA](https://memgraph.com/docs/clustering/high-availability/migrating-to-v3-9-ha) page for
> version-specific instructions before proceeding.
> **Warning**
>
> **Important**: Although the upgrade process is designed to complete
> successfully, unexpected issues may occur. We strongly recommend doing a backup
> of your `lib` directory on all of your `StatefulSets` or native instances
> depending on the deployment type.
### Prerequisites
If you are using **HA Helm charts**, set the following configuration before
doing any upgrade.
```yaml
updateStrategy.type: OnDelete
```
Depending on the infrastructure on which you have your Memgraph cluster, the
details will differ a bit, but the backbone is the same.
Prepare a backup of all data from all instances. This ensures you can safely
downgrade cluster to the last stable version you had.
- For **native deployments**, tools like `cp` or `rsync` are sufficient.
- For **Kubernetes**, create a `VolumeSnapshotClass`with the yaml file fimilar
to this:
```yaml
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
name: csi-azure-disk-snapclass
driver: disk.csi.azure.com
deletionPolicy: Delete
```
Apply it:
```bash
kubectl apply -f azure_class.yaml
```
- On **Google Kubernetes Engine**, the default CSI driver is
`pd.csi.storage.gke.io` so make sure to change the field `driver`.
- On **AWS EKS**, refer to the [AWS snapshot controller
docs](https://docs.aws.amazon.com/eks/latest/userguide/csi-snapshot-controller.html).
### Create snapshots
Now you can create a `VolumeSnapshot` of the lib directory using the yaml file:
```yaml
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
name: coord-3-snap # Use a unique name for each instance
namespace: default
spec:
volumeSnapshotClassName: csi-azure-disk-snapclass
source:
persistentVolumeClaimName: memgraph-coordinator-3-lib-storage-memgraph-coordinator-3-0
```
Apply it:
```bash
kubectl apply -f azure_snapshot.yaml
```
Repeat for every instance in the cluster.
### Update configuration
Next you should update `image.tag` field in the `values.yaml` configuration file
to the version to which you want to upgrade your cluster.
1. In your `values.yaml`, update the image version:
```yaml
image:
tag:
```
2. Apply the upgrade:
```bash
helm upgrade -f
```
Since we are using `updateStrategy.type=OnDelete`, this step will not restart
any pod, rather it will just prepare pods for running the new version.
- For **native deployments**, ensure the new binary is available.
### Upgrade procedure (zero downtime)
Our procedure for achieving zero-downtime upgrades consists of restarting one
instance at a time. Memgraph uses **primary–secondary replication**. To avoid
downtime:
1. Upgrade **replicas** first.
2. Upgrade the **main** instance.
3. Upgrade **coordinator followers**
4. Upgrade the **coordinator leader**.
In order to find out on which pod/server the current main and the current
cluster leader sits, run:
```cypher
SHOW INSTANCES;
```
#### Query behavior during the upgrade
Your applications can stay connected to the cluster throughout the upgrade.
What they will observe depends on the query type and the replication mode:
- **Read queries always work**, regardless of the replication mode used. There
is no specific exception you need to catch — Neo4j drivers are preconfigured
with automatic retries, so reads hitting an instance that is restarting are
transparently retried.
One exception applies to minimal clusters: with **1 main and 1 replica**, you
need to enable the
[`enabled_reads_on_main`](https://memgraph.com/docs/clustering/high-availability/ha-commands-reference#enabled_reads_on_main)
coordinator setting so that, while the only replica is being upgraded, read
queries are routed to the main instance:
```cypher
SET COORDINATOR SETTING 'enabled_reads_on_main' TO 'true';
```
- **Write queries depend on the replication mode:**
- **ASYNC** — writes keep working without errors.
- **SYNC** — writes keep working. A commit that couldn't be replicated to the
instance being restarted (e.g. the old main while it is coming back up) is
still committed on the MAIN, and the failure is reported as a
`SyncReplicationFailure` warning
[notification](https://memgraph.com/docs/database-management/query-metadata#notifications) in the
query summary rather than as an error. Your application doesn't need to
handle anything to keep working; read the notification if you want to log
which replica fell behind, for example: `Failed to replicate to SYNC replica
'instance_1': replica is not reachable or not in sync with the main`. Before
Memgraph 3.13, the same situation raised a `ReplicationException` that
applications had to catch.
- **STRICT_SYNC** — writes won't work while a data instance is being
restarted. Because of the two-phase commit protocol, a transaction cannot
be committed unless every STRICT_SYNC replica confirms it, so writes are
aborted until the restarted instance rejoins the cluster.
If these write-side effects are not acceptable for your workload, you can pause
writes across the whole cluster for the duration of the upgrade. The
recommended way is the
[`global_read_only`](https://memgraph.com/docs/clustering/high-availability/ha-commands-reference#global_read_only)
coordinator setting. Connect to the coordinator leader and run:
```cypher
SET COORDINATOR SETTING 'global_read_only' TO 'true';
```
When enabled, the current MAIN stops accepting write queries while it keeps
serving reads and replicating existing data to REPLICAs. The setting blocks
**all write sources** on the MAIN — user Cypher writes,
[TTL](https://memgraph.com/docs/querying/time-to-live) background expiry, and [stream-](https://memgraph.com/docs/data-streams) and
[trigger-](https://memgraph.com/docs/fundamentals/triggers)driven writes — so the dataset stays frozen
while replicas catch up and you swap binaries. The value is persisted in the
coordinator's Raft-replicated cluster state, so it survives coordinator restarts
and leader re-elections, and is honored across failovers: a newly promoted MAIN
comes up read-only instead of silently accepting writes.
> **Warning**
>
> While the cluster runs mixed versions during the rolling upgrade, read-only mode
> is honored best-effort: older instances behave as before until every node is
> upgraded, after which the cluster self-heals to the requested state.
#### Upgrade replicas
Before restarting any replica, check whether replicas are caught up with the
MAIN:
```cypher
SHOW REPLICATION LAG;
```
If the replica you are about to restart is lagging behind, wait until it
catches up before deleting its pod.
If you are using K8s, the upgrade can be performed by deleting the pod. Start by
deleting the replica pod (in this example replica is running on the pod
`memgraph-data-1-0`):
```bash
kubectl delete pod memgraph-data-1-0
```
**Native deployment:** stop the old binary and start the new one.
Before starting the upgrade of the next pod, it is important to wait until all
pods are ready. Otherwise, you may end up with a data loss. On K8s you can
easily achieve that by running:
```bash
kubectl wait --for=condition=ready pod --all
```
For the native deployment, check if all your instances are alived manually.
This step should be repeated for all of your replicas in the cluster.
#### Upgrade the main
When the main pod is deleted, the coordinator detects that the main is down and
performs an **automatic failover**, promoting one of the already upgraded
replicas to be the new main. Until the failover completes, the cluster does not
accept write queries. The length of this time window depends on how quickly the
failover happens, which is governed by two coordinator settings: the
coordinator pings data instances every
[`instance_health_check_frequency_sec`](https://memgraph.com/docs/clustering/high-availability/ha-commands-reference#instance_health_check_frequency_sec)
seconds and declares an instance down after it hasn't responded for
[`instance_down_timeout_sec`](https://memgraph.com/docs/clustering/high-availability/ha-commands-reference#instance_down_timeout_sec)
seconds. Lowering these values shortens the write-unavailability window.
Upgrade the main pod:
```bash
kubectl delete pod memgraph-data-0-0
kubectl wait --for=condition=ready pod --all
```
#### Upgrade coordinators
The upgrade of coordinators is done in exactly the same way. Start by upgrading
followers and finish with deleting the leader pod:
```bash
kubectl delete pod memgraph-coordinator-3-0
kubectl wait --for=condition=ready pod --all
kubectl delete pod memgraph-coordinator-2-0
kubectl wait --for=condition=ready pod --all
kubectl delete pod memgraph-coordinator-1-0
kubectl wait --for=condition=ready pod --all
```
### Verify upgrade
Your upgrade should be finished now, to check that everything works, run:
```cypher
SHOW VERSION;
```
It should show you the new Memgraph version.
If you paused writes with `global_read_only` before the upgrade, return the
cluster to normal read/write operation once all nodes are upgraded and healthy:
```cypher
SET COORDINATOR SETTING 'global_read_only' TO 'false';
```
### Rollback
If during the upgrade, you figured out that an error happened or even after
upgrading all of your pods something doesn't work (e.g. write queries don't
pass), you can safely downgrade your cluster to the previous version using
`VolumeSnapshots` you took on K8s or file backups for native deployments.
- **Kubernetes:**
```bash
helm uninstall
```
In `values.yaml`, for all instances set:
```yaml
restoreDataFromSnapshot: true
```
Make sure to set correct name of the snapshot you will use to recover your
instances.
- **Native deployments:** restore from your file backups.
> **Info**
>
> If you're doing an upgrade on `minikube`, it is important to make sure that the
> snapshot resides on the same node on which the `StatefulSet` is installed.
> Otherwise, it won't be able to restore `StatefulSet's` attached
> PersistentVolumeClaim from the `VolumeSnapshot`.