If you prefer to use GUI, you can now connect to data streams by using a wizard in the Memgraph Lab.
If you need a stream to play around with, we've provided some at Awesome Data Stream!
1. Add a stream
To add a stream in Memgraph Lab:
- Switch to Streams and Add New Stream.
- Choose Kafka stream type, enter a stream name, server address, and topics you want to subscribe to.
- Go to the Next Step.
- Click on Edit (pencil icon) to modify the Consumer Group, Batch Interval or Batch Size.
If you are trying to connect to MovieLens data stream from the Awesome Data streams, the stream configuration should look similar to this:
Once the basic configuration is finished, you need to define a transformation module and attach it to the stream.
2. Add a transformation module
A transformation module is a set of user-defined transformation procedures written in C or Python that act on data received from a streaming engine. Transformation procedures instruct Memgraph on how to transform the incoming messages to consume them correctly.
To add the transformation module:
- Click on Add Transformation Module.
- Click on Choose Transformation Module.
- Select an existing transformation module or + Create new transformation.
- Review an existing module or clear the screen and write a new transformation procedure.
- Save the transformation module.
- Check if the necessary transformation procedure is visible under Detected transformation functions on the right.
- Select a transformation procedure and Attach to Stream.
Check the transformation module for MovieLens on Awesome Data Stream.
3. Set Kafka configuration parameters
If necessary, add the Kafka configuration parameters to customize the stream further:
- In the Kafka Configuration Parameters + Add parameter field.
- Insert the parameter name and value.
- To add another parameter, Add parameter filed.
- Save Configuration once you have set all parameters.
To connect to the Awesome Data Stream you need to set the following Kafka configuration parameters:
- sasl.username | public
- sasl.password | public
- security.protocol | SASL_PLAINTEXT
- sasl.mechanism | PLAIN
4. Connect Memgraph to the stream and start ingesting the data
Once the stream is configured, you can Connect to Stream.
Memgraph will do a series of checks, ensuring that defined topics and transformation procedures are correctly configured. If all checks pass successfully, you can Start the stream. Once you start the stream, you will no longer be able to change any of the configuration settings, except the stream offset and the transformation module.
The stream status changes to Running, and data is ingested into Memgraph. You can see the number of nodes and relationships rising as the data keeps coming in. If your nodes and relationships numbers stay at zero, check the transformation module, as there might be a flaw in the logic that needs to be resolved.
Switch to Query Execution and run a query to visualize the data coming in:
RETURN p LIMIT 100;