DATA PLATFORM / 01
Metadata-driven ingestion framework
A reusable ingestion contract for OLTP systems, file feeds, REST APIs, and streaming sources — replacing bespoke pipeline builds with configuration.
20+pipelines standardised
minutesfreshness target
zerobespoke source paths
01 / THE CHALLENGEMake the repeatable path
Make the repeatable path
the default path.
Every new source arrived with a new implementation. That multiplied operational paths, made schema changes risky, and kept the team in a permanent handover loop.
02 / THE ARCHITECTURESystems that stay
Systems that stay
legible under load.
The framework separates what changes from what should stay stable. Source metadata describes connection, schema, cadence, and destination. Shared PySpark and Auto Loader components handle ingestion, schema evolution, checkpointed recovery, retries, and dead-letter paths. Delta Lake medallion layers provide the contract after landing.
New sources onboard through metadata entries rather than new pipeline code.