How We Integrated 15+ Data Sources with Memgraph in a Day Using ChatGPT
Data migration is one of the biggest implementation challenges faced by engineering teams all over the world. Different formats, protocols, query languages, and processing models can turn even small migrations into a time-consuming task. Especially when it involves graph data.
But what if you could integrate 15+ sources in just one day?
Thanks to Memgraph’s External Procedures, which support Python, Rust, C, and other programming languages, you can programmatically build data loaders for virtually any system. Pair that with the code-generation capabilities of large language models (LLMs), and you can build and test migration scripts at unprecedented speed.
That’s exactly how we managed to create connectors for DuckDB, Spark, Iceberg, S3, and many more. All in just one day!
Here’s how it works.
Behind the Speed: How These Migrations Were Built in a Single Day
The speed was achieved by combining two powerful technologies:
- Memgraph Query Modules: These allow you to write custom data integration logic using Python, Rust, C++ or other programming languages. This makes it easy to fetch, parse, and stream data directly into Memgraph using an API. The initial use case for the query modules was to be more expressive if Cypher expressiveness was holding users down. But with Python, there is the whole ecosystem which enables users to use the data source Python clients in order to integrate a data source with Memgraph
- Large Language Models (ChatGPT in this case): Using LLMs, our team was able to auto-generate chunks of very similar Python code needed for these data source integrations. Instead of manually writing 10–15 integrations, we have manually written only 1 data source integration, and used LLMs to generate other integrations in just a day.
The result? In under 24 hours, we successfully built integrations for over 15 data sources.
What’s New in Memgraph’s Migration Toolkit
With our latest set of migration tools and connectors, we can now confidently say: You can migrate to Memgraph from anywhere!
Whether your data lives in a relational database, a streaming platform, a data lake, or another graph database, there’s now likely a clear path to getting that data into Memgraph.
Here’s what’s new:
1. DuckDB: One Connector, 15+ Data Sources
The new DuckDB integration with Memgraph allows Memgraph users to benefit from its extensive list of supported data sources. With a single connector, you can now access over 15 different file formats and database systems, including:
- Common file formats: Parquet, CSV, JSON, and Excel files
- Cloud storage services (AWS S3, Azure Blob Storage, Cloudflare R2)
- Open-source table formats (Delta Lake, Apache Iceberg)
- Relational databases (PostgreSQL, MySQL, SQLite)
📄 Full list of DuckDB supported data sources
2. Migrate from Neo4j with a Single Cypher Query
Until now, migrating from Neo4j required exporting data into CSV or JSON files, followed by custom import logic.
With this new update, a single Cypher query is all it takes. Memgraph’s built-in migrate module can directly connect to your Neo4j instance, fetch your data, and load it into your Memgraph database.
3. Migrate Apache Iceberg Tables from Data Lakes using Dremio
Apache Iceberg is becoming a popular format for managing large analytical datasets in modern data lake environments. Memgraph now connects to data lakes via Dremio using the Arrow Flight RPC protocol.
For organizations that have invested in modern data lake architectures, this creates a frictionless path to graph analysis while maintaining your existing data infrastructure.
4. Migrate using Apache Spark
If your data is already part of a Spark-powered pipeline, you can now prepare and transform it using the Neo4j Spark Connector and load it into Memgraph.
This setup gives you fine-grained control over how data is structured and optimized before ingestion. This is especially useful when dealing with large-scale or deeply nested datasets.
5. ServiceNow Connector
We've introduced a dedicated migration module for ServiceNow, enabling you to pull data directly from this widely used enterprise service management platform via its REST API. This simplifies integrating operational data from ServiceNow with Memgraph for graph querying and analysis.
📄 Migration module from ServiceNow
6. Amazon S3 and S3-Compatible Storage
In addition to the access provided via DuckDB, we've added dedicated support for migrating data directly from Amazon S3 and other S3-compatible storage services. This allows for straightforward ingestion of structured data files stored in cloud object storage.
Connecting Memgraph to Your Data Ecosystem
These new additions are part of our ongoing effort to ensure Memgraph can integrate effectively into any modern data environment. They add greater variety to Memgraph's existing list of supported data sources. Our focus is on reducing technical hurdles so you can concentrate on product development and graph modeling.
📄 Full list of supported sources
Built to Fit Existing Workflows
The goal behind expanding migration support is simple: to reduce the time and effort needed to bring graph analytics into your data stack. Whether you're working with local files, cloud storage, or live APIs, Memgraph’s connectors are designed to work alongside your existing systems, not replace them.
If you're testing out Memgraph or preparing to scale your graph workloads, you can now plug it in more easily without building custom loaders or restructuring your data pipelines.
What’s Next
We're committed to making Memgraph the most accessible graph database on the market. These new migration pathways represent a major step towards our goal of enabling you to bring your data to Memgraph from anywhere.
If you're using a data source we don't yet support, or if you have feedback on our migration tools, we'd love to hear from you. Drop a request in our GitHub discussions or reach out to us via our community Discord channel.
