Viewing profile — ryan_green

exactly!!!

apologies for not getting into more detail--wanted to start by covering things at a high level. There are a few key concepts that might be helpful. * data state - this is contents …

sort of. the main issues we've found with each developer having their own DB on a complex data pipeline are 1) if that DB contains petabytes of data, creating it one for each devel…

saltcured, find these comments super insightful! > Yeah, there's a lot of hidden magic/assumptions in having a "writable snapshot of a specific version" of production data. That's …

NBJack, your point about the difficult of managing external dependencies is well taken. That said, our data pipeline uses cloud storage and multiple external services and the scena…

totally agree. and creating ephemeral environments for data pipelines is quite a bit more challenging than systems with a less complicated data state. nonetheless, this has paid of…