Rethinking Data Ingestion as a DAG
falconer.com
Rethinking Data Ingestion as a DAG
1–3 of 3 posts
Re: Rethinking Data Ingestion as a DAG
#2Once you decomposed the pipeline into stages, did observability become significantly easier? I imagine stage-level latency and queue depth give you a much clearer signal than profiling a single async service.
Re: Rethinking Data Ingestion as a DAG
#3Once you decomposed the pipeline into stages, did observability become significantly easier? I imagine stage-level latency and queue depth give you a much clearer signal than profiling a single async service.
Exactly. Once each stage in the pipeline lived as its own job in the queue, observability got dramatically better. Stage-level latency, queue depth, and failure rates made bottlenecks obvious in a way that was hard to see inside a single async service. It also made it much easier to tune concurrency per stage instead of guessing globally.