Custom query modules

Custom query modules

Many graph algorithms and uitility procedures have already been developed and are available as part of the MAGE library you can add to your Memgraph installation. The library is already included if you are using Memgraph Platform or Memgraph MAGE Docker images to run Memgraph.

But, you can also implement custom query modules. Memgraph supports extending the query language with user-written procedures in C, **C++*, Python, and Rust. These procedures are grouped into modules - query module files (either *.so or *.py files).

Every single Memgraph installation comes with the example.so and py_example.py query modules located in the /usr/lib/memgraph/query_modules directory. They were provided as examples of query modules for you to examine and learn from.

Each query module file corresponds to one query module, and file names are mapped as query module names. For example, example.so will be mapped as example module, and py_example.py will be mapped as py_example module. If each module file has a procedure called procedure defined, those procedures would be mapped in the Cypher query language as example.procedure() and py_example.procedure() respectively.

You can develop custom procedures from Memgraph itself.

Develop custom query modules

If you want to develop Python procedures you can do so from Memgraph Lab, the visual user interface. The files will be saved at /var/lib/memgraph/internal_modules. Just navigate to Query Modules and select + New Module.

If you don't won't to develop from Memgraph Lab, you can:

  • install MAGE development environment from Docker Hub
  • build MAGE using the docker build command
  • build MAGE from source

Then, select a language you want to develop in.

Install MAGE and import query modules

The following steps will guide you through installing MAGE development environment from Docker Hub and importing develooped query modules.

Download the MAGE image

Run the following command to get the Memgraph and MAGE development Docker image:

docker run -p 7687:7687 memgraph/memgraph-mage:<version>-dev

By running this command, you will get an image with the following tools installed: Python3, Rust, Clang, Make, and CMake.

Develop a query

Develop query modules in Python, C, C++ or Rust (opens in a new tab).

Start the MAGE container

Use the following command to start the MAGE container:

docker run --rm -p 7687:7687 --name mage memgraph-mage:version-dev

Be sure to replace the version with the specific version, for example:

docker run --rm -p 7687:7687 --name mage memgraph-mage:1.4-dev

Copy the files to the container

Copy the files to the container named mage:

docker cp . mage:/mage/

Enter the container

Position yourself inside the container as root:

docker exec -u root -it mage /bin/bash
💡

If you performed the build locally, make sure to delete the cpp/build directory because you might be dealing with different architectures or problems with CMakeCache.txt.

To delete it, run:

rm -rf cpp/build

Build MAGE

Build MAGE with the option to copy executables from mage/dist to /usr/lib/memgraph/query_modules:

python3 setup build -p /usr/lib/memgraph/query_modules/

Exit the container

Everything should be ready to exit the container and load the query modules:

exit

Querying

Query modules are loaded into Memgraph on startup, so if your instance was already running, you need to execute the following query inside one of the querying platforms (mgonsole, Memgraph Lab or a client library) to load them:

CALL mg.load_all();

Run a query and test your module. All procedures are called using the CALL clause:

MERGE (start:Node {id: 0})-[:RELATION]->(:Node {id: 1})-[:RELATION]->(:Node {id: 2})
CALL random_walk.get(start, 2) YIELD path
RETURN path

Since every query module is run as one transaction in Memgraph, the user can stop the query module by terminating the corresponding transaction, but only if crucial parts of the code, such as while and until loops, or similar points where the procedure might become costly, are preceded with CheckMustAbort() function. The user can then find out the transaction ID using SHOW TRANSACTIONS command and terminate it using the TERMINATE TRANSACTIONS <tid> command.

Testing

Test decoupled parts of your code that don't depend on Memgraph like you would in any other setting. E2e (end to end) tests, on the other hand, depend on internal Memgraph data structures, like nodes and edges. After running Memgraph, we need to prepare the testing environment on the host machine. Position yourself in the mage directory you cloned from GitHub. The expected folder structure for each module is the following:

mage
└── e2e
    └── random_walk_test
        └── test_base
            ├── input.cyp
            └── test.yml

input.cyp represents a Cypher script for entering the data into the database. To simplify this tutorial, we'll leave the database empty. test.yml specifies which test query should be run by the database and what should be the result or exception. Create the files following the aforementioned directory structure.

input.cyp

MATCH (n) DETACH DELETE n;

test.yml

query: >
  MATCH (start:Node {id: 0})
    CALL random_walk.get(start, 2) YIELD path
    RETURN path
 
output: []

Lastly, run the e2e tests with python:

python test_e2e