Streaming Operational Graphs
Keep a live service dependency graph in Memgraph, fed from Kafka, for impact analysis and root-cause investigation.
Architecture
Memgraph consumes Kafka topics directly through native streams and transformation modules. Applications and operators query the live graph.
Sources
Services, network devices
Kafka topic
Retained event log
Memgraph Kafka stream
Transformation module maps messages to Cypher updates executed by Memgraph.
Operational graph
Services and dependencies
Applications and operators
Impact queries, incident tooling
Event semantics
At-least-once delivery
A batch can be applied twice if the Kafka offset commit fails, so every write must be safe to repeat.
Idempotent writes
Use stable IDs and deterministic updates so replays are safe. MERGE alone is not enough.
Per-partition ordering
Kafka orders within a partition only. Key related events together; reject stale versions.
Explicit deletes
The transformation must convert delete or tombstone events into graph deletions; otherwise obsolete services and edges remain.
Scaling levers
Storage mode
IN_MEMORY_TRANSACTIONAL for a live graph. Use analytical mode only when its lack of ACID, WAL and replication is acceptable like when there's controlled bulk imports or read-only analysis.
Kafka
Partitions, partition key, retention
Memgraph
Batch interval and size, indexes, contention, memory
Replicas
ASYNC replicas can lag and return older data
Graph model
Impact
Follow incoming DEPENDS_ON edges: what breaks if this fails?
Upstream candidates
Follow outgoing edges to inspect dependencies that may explain the symptom. Confirm with operational evidence.
Observability
Stream state · status and configuration
Consumer lag · how far the graph trails Kafka
Transform failures · events not written
Write conflicts · retries from contention
Memory · headroom to the limit
Query latency · key impact queries
Replication lag · replicas behind main
Freshness · event to queryable graph
Implementation path
Write transform
Map Kafka messages to Cypher queries and parameters.
Configure stream
Set the topic, consumer group, batch interval and batch size.
Validate transform and replay
Check generated queries; confirm replay leaves no duplicates or corrupted state.
Start and monitor
Watch lag, failures, conflicts and freshness.
Validate queries
Run impact queries against known past incidents.
