DataLineage Doctor
LLM-powered RCA engine for data quality incidents, with OpenMetadata.
Problem
Data quality incidents in large pipelines have no automated root cause analysis. Engineers manually trace lineage graphs for hours.
Built
LLM-powered engine that ingests OpenMetadata lineage graphs, computes blast radius, identifies ownership, and generates structured RCA reports with remediation recommendations.
lineage
blast radius analysis
automated
RCA
ownership
mapping
The build
[Idea]
DataLineage Doctor started from a real Monday morning. Your revenue dashboard shows $0 at 9 AM. The CEO is already asking why. The data steward is somewhere in a Slack thread trying to figure out which upstream pipeline broke. The fix is automated RCA. A webhook from OpenMetadata kicks off a structured agent that pulls lineage, DQ history, pipeline status, ownership, and prior incident memory, then writes a structured report with timeline, blast radius, confidence, and remediation.
[Framing]
The architecture is FastAPI, Celery, Redis, Postgres with SQLAlchemy 2.0 async, Alembic, and an httpx-based OpenMetadata client. The interesting piece is the agent. It runs an iterative tool-calling loop. It calls tools (lineage API, DQ API, pipeline status API, blast radius calculator, incident history), parses strict JSON responses, and decides whether to call another tool or stop. Every tool call is logged. Every failure is normalized into a structured error payload.
[Build]
We parse the final response into a strict RCAReport Pydantic model and recompute the confidence label from the confidence score. LLMs tend to over-confess, so a re-map is healthier than trusting the raw output. The typed OM client has retry and compatibility guards. OpenMetadata's API has been pretty stable, but breaking changes happen between minor versions, and we'd rather fail loudly than silently skip a step.
[Deploy]
The dashboard is Jinja2 plus React Flow via CDN (no SPA build), with auto-refreshing high-severity cases. Subdomain architecture in prod (dldoctor.app, prometheus.dldoctor.app, grafana.dldoctor.app, om.dldoctor.app) behind one Nginx. Six Prometheus metrics feed a pre-provisioned Grafana dashboard. Open loop is writing the RCA back into OpenMetadata via its native Incident API, version-aware and best-effort.
[What's next]
What's next. HMAC webhook signature verification. Real auth on the dashboard. Multi-tenant routing. The repo has 25 stars and 11 forks on GitHub, which I'm genuinely proud of for a niche tool. The whole thing runs with make demo against a local OpenMetadata stack, and that's how I test changes before pushing.
Stack
1 techs · grouped by layerBackend
- FastAPI
Also used