# Data durability Memgraph uses two mechanisms to ensure the durability of stored data and make disaster recovery possible: * write-ahead logging (WAL) * periodic snapshot creation These mechanisms generate **durability files** and save them in the respective `wal` and `snapshots` folders in the **data directory**. Data directory stores permanent data on disk. Memgraph **cannot be used with only WAL files enabled**. You can either have only snapshots or snapshots and WAL files. The default data directory path is `/var/lib/memgraph` but the path can be changed by modifying the `--data-directory` configuration flag. To learn how to modify configuration flags, head over to the [Configuration](https://memgraph.com/docs/database-management/configuration#changing-configuration-settings) page. With Memgraph Enterprise, the `data_directory` holds `databases` directory which splits durability files by database name. The reason for that is the multi-tenant architecture in Memgraph Enterprise, where the durability files for each database are stored under `/data_directory/databases/`. The `databases` directory will exist even if you're not using the multi-tenancy feature. Durability files are deleted when certain events are triggered, for example, exceeding the maximum number of snapshots, defined by the `--storage-snapshot-retention-count=3` flag. To prevent the deletion of durability files, you need to lock the `data directory`, and enable it again by unlocking the directory. To manage this behavior, use the following queries: ```cypher LOCK DATA DIRECTORY; UNLOCK DATA DIRECTORY; ``` To show the status of the data directory, run: ```cypher DATA DIRECTORY LOCK STATUS; ``` To encrypt the data directory, use [LUKS](https://gitlab.com/cryptsetup/cryptsetup/) as it works with Memgraph out of the box and is undetectable from the application perspective so it shouldn't break any existing applications. ## Durability mechanisms To configure the durability mechanisms, check their respective [configuration flags](https://memgraph.com/docs/database-management/configuration#storage). ### Write-ahead logging Write-ahead logging (WAL) is a technique applied in providing **atomicity** and **durability** to database systems. In the default IN_MEMORY_TRANSACTIONAL [storage mode](https://memgraph.com/docs/fundamentals/storage-memory-usage#storage-modes), Memgraph creates a `Delta` object each time data is changed. By using Deltas, Memgraph creates write-ahead logs. Each database modification is therefore recorded in a log file before being written to the DB, and in the end the log file contains all steps needed to reconstruct the DB’s most recent state. Memgraph has WAL enabled by default. To switch it on and off, use the boolean `--storage-wal-enabled` flag. For other WAL-related flags check the [configuration reference guide](https://memgraph.com/docs/database-management/configuration#storage). By default, WAL files are located at `/var/lib/memgraph/wal`. #### WAL file lifecycle **Older WAL files are deleted automatically after a snapshot is created** since the snapshot contains the full database state up to that point. Only WAL files containing changes after the latest snapshot are retained. To control WAL file cleanup indirectly, you can limit the number of snapshots via `--storage-snapshot-retention-count`. **It is not possible to use WAL files exclusively** without snapshots. Memgraph enforces periodic snapshots when WAL is enabled and will fail to start if WAL is enabled with snapshot interval set to zero. #### WAL data integrity > **Info** > > Per-transaction WAL checksums were introduced in Memgraph v3.12. > > WAL files written by older versions do not contain checksums and are recovered > without integrity verification. Checksum verification applies only to WAL files > written by Memgraph v3.12 or newer. > > Before Memgraph v3.13, a checksum mismatch anywhere in the WAL chain recovered > the readable prefix of that file and then continued with the files after it, > which could bring the database up with an incomplete dataset without reporting > it. Since v3.13, damage in a completed WAL file fails recovery instead. To guard against silent on-disk corruption, Memgraph protects WAL files with [CRC32](https://en.wikipedia.org/wiki/Cyclic_redundancy_check) checksums: - The **WAL file header** (offsets and metadata such as the UUID, epoch ID and sequence number) is protected by its own checksum. - **Each transaction** is protected by a 4-byte checksum covering the transaction's bytes (transaction start, deltas and transaction end). Checksums are verified automatically during recovery, so corrupted data is never applied to the database. A mismatch in the WAL header causes recovery from that file to fail. What happens on a transaction checksum mismatch depends on whether the file was completed: - A **completed (finalized) WAL file** — one Memgraph finished writing and rotated away from — was flushed to disk before it was closed, so every transaction in it is durable and was already acknowledged. A mismatch there means the bytes on disk rotted, and recovery fails rather than continuing with part of the file. Recovering only a prefix would be unsound, because the WAL files that follow were written on top of the transactions that went missing. - A **WAL file that was still being written** when the process stopped may legitimately have a torn tail. Recovery applies its transactions up to the last whole one and stops there, which is how an interrupted write degrades gracefully. Files that the newest snapshot already covers are skipped without being read, so corruption in those is harmless. If a WAL file written after the newest snapshot is damaged, the database fails to recover; use [`--storage-allow-recovery-failure`](https://memgraph.com/docs/database-management/configuration) together with `RECOVER SNAPSHOT` (see [recovery failure handling](#recovery-failure-handling)), or restore from a backup. The same checksums protect WAL files that are buffered on disk on a replica before being applied, so corruption introduced between the main and the replica is detected before the data is committed. When a replica hits damage in a finalized file it reports the failure and stops applying the rest of the batch instead of skipping ahead; the in-flight transaction is aborted, main does not advance its view of the replica, and the transfer is retried later. Deltas streamed during the commit (`PrepareCommitRpc`) are not checksummed because the TCP transport already provides integrity guarantees. > **Info** > > Snapshots are not yet protected by checksums. ### Snapshots Snapshots provide a faster way to restore the states of your database. Snapshots are created periodically based on the value defined with the `--storage-snapshot-interval` configuration flags, as well as upon exit based on the value of the `--storage-snapshot-on-exit` configuration flag. When a snapshot creation is triggered, the entire data storage is written to the drive. Nodes and relationships are divided into groups called batches. > **Info** > > If both flags `--storage-snapshot-interval` and > `--storage-snapshot-interval-sec` are defined, the flag > `--storage-snapshot-interval` will be used. Snapshot creation can be made faster by using **multiple threads**. See [Parallelized execution](#parallelized-execution) for more information. On startup, the database state is recovered from the most recent snapshot file. Memgraph can read the data and build the indexes on multiple threads, using batches as a parallelization unit: each thread will recover one batch at a time until there are no unhandled batches. This means the same batch size might not be suitable for every dataset. A smaller dataset might require a smaller batch size to utilize a multi-threaded processor, while bigger datasets might use bigger batches to minimize the synchronization between the worker threads. Therefore, the size of batches and the number of used threads [are configurable](https://memgraph.com/docs/database-management/configuration#storage) similarly to other durability-related settings. The timestamp of the snapshot is compared with the latest update recorded in the WAL file and, if the snapshot is less recent, the state of the DB will be recovered using the WAL file. Memgraph has snapshot creation enabled by default. You can configure the exact snapshot creation behavior by [defining the relevant flags](https://memgraph.com/docs/database-management/configuration#storage). Alternatively, you can make one directly by running the following query: ```opencypher CREATE SNAPSHOT; ``` > **Info** > > If another snapshot is already being created or no committed writes to the > database have been made since the last snapshot, this query will fail with an > error. By default, snapshot files are saved inside the `var/lib/memgraph/snapshots` directory. The `CREATE SNAPSHOT` query will return the path of the newly created snapshot file. To query which snapshots currently exist in the data directory, execute: ```opencypher SHOW SNAPSHOTS; ``` #### Snapshot and WAL recovery logic During recovery, Memgraph always attempts to use the fastest and most efficient method to restore the database state: - If the snapshot has a **more recent** timeline than the WAL, the database is fully recovered from the latest snapshot. - If the snapshot has a **less recent** timeline than the WAL, Memgraph first recovers from the snapshot, and then replays WAL files containing changes made after the snapshot was taken. This ensures recovery to the most recent state. - Snapshot recovery is **typically faster** than recovery from WAL because snapshots store the complete state of the database in a single file, while WAL files store incremental changes and need to be replayed sequentially. ### Periodic snapshots `IN_MEMORY_TRANSACTIONAL` mode supports periodic snapshot creation. The interval can be set at startup via the `--storage-snapshot-interval` flag or at run-time via the database settings: ```opencypher SET DATABASE SETTING "storage.snapshot.interval" TO "1200"; SET DATABASE SETTING "storage.snapshot.interval" TO "* * 12 * * *"; SET DATABASE SETTING "storage.snapshot.interval" TO ""; ``` > **Info** > > Changing the configuration settings depends on the way you are using Memgraph, > so please refer to the [configuration > docs](https://memgraph.com/docs/database-management/configuration#changing-configuration) for more > information. If the interval string is an integer, then it's treated as the execution period in seconds. Interval can also be defined as a 6-field CRON expression (seconds, minute, hour, day of month, month, day of week). Standard 5-field cron references (e.g. [crontab.guru](https://crontab.guru/)) describe the last five fields; Memgraph adds seconds as the first field. By setting the value to an empty string, the background process is paused and any currently active snapshot creation will finish. Please note that defining the interval via a CRON expression is an [Enterprise feature](https://memgraph.com/docs/database-management/enabling-memgraph-enterprise). If the database is started in or migrated into `IN_MEMORY_ANALYTICAL` mode, the background thread will pause and no snapshots will be created as long as that mode is active. The job will continue with the last defined interval when the storage mode is changed to `IN_MEMORY_TRANSACTIONAL` storage mode. > **Info** > > The periodic snapshot will be skipped if another snapshot is in progress or no > new writes have been committed since the last snapshot. If the periodic snapshot > is skipped it will be logged on INFO level. > **Warning** > > Snapshots and WAL files are presently not compatible between Memgraph versions. ### Parallelized execution Snapshot creation in Memgraph can be optimized using multiple threads, which significantly reduces the time required to create snapshots for large datasets. This behavior can be controlled using the following flags: - `--storage-parallel-snapshot-creation`: This flag determines whether snapshot creation is performed in a multi-threaded fashion. By default, it is set to `false`. To enable parallelized execution, set this flag to `true`. - `--storage-snapshot-thread-count`: This flag specifies the number of threads to be used for snapshot creation. By default, Memgraph uses the system's maximum thread count. You can override this value to fine-tune performance based on your system's resources. When parallelized execution is enabled, Memgraph divides the data into batches, where the batch size is defined via `--storage-items-per-batch`. The optimal batch size and thread count may vary depending on the dataset size and system configuration. #### When parallelization helps Parallel execution is especially beneficial when CPU-bound operations dominate the snapshot creation process, such as serialization or compression of in-memory structures. As a general guideline, parallel snapshot creation provides the most significant performance improvement when disk I/O constitutes 25% or less of the total snapshot creation time. To take full advantage of parallelization, it’s also important to set the `--storage-items-per-batch` flag appropriately. This value determines how the dataset is split into work units for threads. A good rule of thumb is: Total number of items (vertices + edges) ≈ 4 × number of threads × --storage-items-per-batch This ensures that each thread has enough batches to work on without idling, helping maximize CPU utilization during snapshot creation. When using multi-threaded snapshot creation with the correct batch size, the disk will once again become the bottleneck. At that point, more threads will not necessarily yield better performance. #### Measuring disk write speed on Linux To determine how fast your disk can handle writes (which influences the I/O bottleneck), you can use the dd command: ```bash dd if=/dev/zero of=testfile bs=1G count=1 oflag=direct ``` This writes a 1 GB file directly to disk and reports the write speed. After the test, remove the file. You can also monitor real-time disk utilization during snapshot creation using tools like `iostat`, `iotop`, or `dstat`. ### Limiting the impact of snapshots on queries A snapshot writes the whole dataset to disk, and on larger datasets that can slow down the queries running at the same time. Memgraph therefore writes snapshots out in small chunks, so a snapshot in progress doesn't take memory away from your workload. The chunk size is set with `--storage-snapshot-writeback-window-mib` and defaults to 32 MiB. Each snapshot thread uses its own chunk, so with `--storage-parallel-snapshot-creation=true` the total is the chunk size times `--storage-snapshot-thread-count`. The default suits most deployments. Tuning it trades off snapshot duration against the performance of queries running at the same time, but the balance depends on your hardware, your operating system and your workload, so treat any change as something to measure on your own deployment. Setting it to `0` turns the behavior off and lets the operating system decide. ## Recovery failure handling By default, if a database fails durability recovery on startup — because of a truncated WAL, a corrupt snapshot, a missing prefix WAL file, or any other malformed durability file — the **entire Memgraph process refuses to boot**. Recovery hits a fatal error deep in the durability layer and the process exits. In a [multi-tenant](https://memgraph.com/docs/database-management/multi-tenancy) deployment, a single corrupt tenant therefore takes down every other (healthy) tenant on the instance, and in a [clustered](https://memgraph.com/docs/clustering/high-availability) deployment it takes down that instance's role in the cluster. The only recourse is to manually restore the whole data directory from a backup before the process will start at all. Setting the [`--storage-allow-recovery-failure`](https://memgraph.com/docs/database-management/configuration) flag to `true` changes this: a database that fails recovery no longer crashes the process. Instead it comes up in a **broken** state — an empty, inert in-memory placeholder whose on-disk durability files are left byte-for-byte untouched. The instance boots, all healthy databases work normally, and the operator can inspect which database is broken and recover it online. The flag defaults to `false`, so upgrading Memgraph does not silently change how corruption is handled — you keep the existing fail-stop-on-corruption guarantee unless you deliberately opt into availability over it. > **Info** > > Recovery failure handling applies to **in-memory** storage > (`IN_MEMORY_TRANSACTIONAL` and `IN_MEMORY_ANALYTICAL`) only. On-disk > (`ON_DISK_TRANSACTIONAL`) storage keeps the fatal crash-on-recovery-failure > behavior; the flag has no effect there. ### Broken database behavior A broken database: - comes up **empty** — it holds no data, and its on-disk `snapshots/` and `wal/` files are left exactly as they were, so no recovery option is lost; - **rejects every query that touches its data** with a clear, actionable error (see below), so you are never served silently wrong (empty) results from a database that failed to load; - suppresses background durability while broken — periodic snapshots and the on-exit snapshot are skipped and no WAL is written, so an empty placeholder can never overwrite your corrupt durability files with an empty snapshot; - is re-derived on every startup. The broken state is **never persisted** — if you fix the underlying files out-of-band, the database simply recovers normally on the next restart. Any query that operates on a broken database's data (Cypher reads and writes, DDL, `CREATE SNAPSHOT`, `SHOW INDEX INFO` / `SHOW CONSTRAINT INFO` and other graph-metadata `SHOW ... INFO` variants) is rejected with: ``` Database is in the broken state because the recovery process failed. Please recover your database using the RECOVER SNAPSHOT query. If you have a backup of the whole data directory, please replace the current data directory with the backup one and restart the process. ``` Meta and admin queries that operate on instance-level or system state rather than the database graph remain available so you can diagnose and recover — for example `USE DATABASE`, `SHOW DATABASES`, `SHOW STORAGE INFO`, `SHOW CONFIG`, `RECOVER SNAPSHOT`, authentication and replication queries, `LOCK`/`UNLOCK DATA DIRECTORY`, and `FREE MEMORY`. ### Identifying a broken database Two operator-facing queries report each database's health as `ready` or `broken`: - [`SHOW DATABASES`](https://memgraph.com/docs/database-management/multi-tenancy) has a `Health` column (alongside `Name` and `State`), so you can immediately see which database did not load: ``` +------------+-------+---------+ | Name | State | Health | +------------+-------+---------+ | memgraph | HOT | ready | | mydb | HOT | broken | +------------+-------+---------+ ``` - [`SHOW STORAGE INFO ON CURRENT DATABASE`](https://memgraph.com/docs/database-management/server-stats#per-database-storage-information) (and `ON DATABASE `) includes a `health` row with the value `ready` or `broken`. ### Restoration paths Once a database is broken, you have several ways to restore it. Choose based on whether you have a good snapshot, whether you are running in a cluster, and whether you want to keep the database's data at all. #### Recover in place with RECOVER SNAPSHOT The primary cure is [`RECOVER SNAPSHOT`](https://memgraph.com/docs/database-management/backup-and-restore#restore-data), which loads a known-good snapshot into the broken placeholder. On a broken current database (switch to it first with `USE DATABASE ` in Enterprise multi-tenant deployments; in Community the default `memgraph` database is recovered directly): ```cypher RECOVER SNAPSHOT "/path/to/good.snapshot"; ``` On success, Memgraph clears the broken flag and the database resumes normal operation and background durability. The existing `RECOVER SNAPSHOT` behavior moves all prior/corrupt snapshots and WAL files to the `.old` directory (or deletes them when [`--storage-backup-dir-enabled`](https://memgraph.com/docs/database-management/configuration) is off), leaving a clean single-snapshot directory. As a result, the database **recovers cleanly on the next restart and does not re-enter the broken state**. `RECOVER SNAPSHOT` is available in both Community and Enterprise editions, so single-tenant deployments can recover a corrupt default database. #### Restore the whole data directory from a backup If you do not have a usable snapshot to load with `RECOVER SNAPSHOT`, replace the instance's data directory with a backup copy of the whole data directory and restart the process. Because the broken placeholder never wrote to the corrupt files, the original durability files are still intact if you prefer to recover them out-of-band before restarting. #### Replica self-heal (High Availability) In a [clustered (HA)](https://memgraph.com/docs/clustering/high-availability) deployment, a broken **replica** database heals automatically. Because the broken placeholder comes up empty with a fresh epoch, the main detects that the replica is not caught up, drives it into recovery, and sends a full snapshot (plus WAL). Loading that snapshot clears the broken state, after which the database resumes normal replication and serves reads. No operator intervention is required on replicas, and a broken database on one instance does not disrupt replication of the other healthy databases. #### Broken main and failover (High Availability) A **main** that is broken for one database but otherwise healthy **remains main** — the cluster does **not** automatically fail over based on per-database broken state, so it never thrashes over a single corrupt tenant. Meanwhile, replicas keep serving reads of their healthy copy of that database, so read availability survives main-side corruption. Recovery of a broken main is operator-driven, and you have two options: - **Cure it in place** with `RECOVER SNAPSHOT` (as above), after which the recovered database participates in replication normally and the cluster returns to a fully healthy state. - **Fail over to a healthy replica.** If a replica holds a good copy of the broken database, promote it with a [manual failover](https://memgraph.com/docs/clustering/high-availability/ha-commands-reference) — `DEMOTE INSTANCE` the old main, then `SET INSTANCE ... TO MAIN` on the replica. The demoted old main rejoins as a replica, and its broken database then self-heals from the new main through the replica self-heal path described above. #### Drop and recreate the database If you want to abandon a broken database rather than recover its data, you can [`DROP DATABASE`](https://memgraph.com/docs/database-management/multi-tenancy#drop-database-with-force) it and create it again: ```cypher DROP DATABASE mydb; CREATE DATABASE mydb; ``` `DROP DATABASE` works on a broken database like any other and also removes its corrupt on-disk files, reclaiming the disk. It does not affect other databases. > **Warning** > > Dropping and recreating a database **loses the fine-grained access privileges** > that were granted on it. User-role-to-database mappings and grants are not > preserved across a drop, so after recreating the database you must re-grant > access to the users and roles that need it. See > [user privileges and database access](https://memgraph.com/docs/database-management/multi-tenancy#user-privileges-and-database-access). ## Storage modes Memgraph has the option to work in `IN_MEMORY_ANALYTICAL`, `IN_MEMORY_TRANSACTIONAL` or `ON_DISK_TRANSACTIONAL` [storage modes](https://memgraph.com/docs/fundamentals/storage-memory-usage.md). Memgraph always starts in the `IN_MEMORY_TRANSACTIONAL` mode in which it creates periodic snapshots and write-ahead logging as durability mechanisms, and also enables creating manual snapshots. In the `IN_MEMORY_ANALYTICAL` mode, Memgraph offers no periodic snapshots and write-ahead logging. Users can create a snapshot with the `CREATE SNAPSHOT;` Cypher query. During the process of snapshot creation, other transactions will be prevented from starting until the snapshot creation is completed. In the `ON_DISK_TRANSACTIONAL` mode, durability is supported by RocksDB since it keeps its own [WAL](https://github.com/facebook/rocksdb/wiki/Write-Ahead-Log-%28WAL%29) files. Memgraph persists the metadata used in the implementation of the on-disk storage.