Dev / Stage / Prod is the wrong pattern for data pipelines
1–10 of 19 posts
Re: Dev / Stage / Prod is the wrong pattern for data pipelines
#2Yet another discovery that ephemeral environments are the more robust way to design things. Yes they are harder to setup and require some proper thought and engineering but they end up paying themselves off down the line many times over.
Re: Dev / Stage / Prod is the wrong pattern for data pipelines
#3Yet another discovery that ephemeral environments are the more robust way to design things. Yes they are harder to setup and require some proper thought and engineering but they end up paying themselves off down the line many times over.
What does this mean exactly? Containerized everything? Cloud IDEs?
Re: Dev / Stage / Prod is the wrong pattern for data pipelines
#4Yet another discovery that ephemeral environments are the more robust way to design things. Yes they are harder to setup and require some proper thought and engineering but they end up paying themselves off down the line many times over.
What does this mean exactly? Containerized everything? Cloud IDEs?
However you do it, the ability to spin up a fresh env with one click or so
Re: Dev / Stage / Prod is the wrong pattern for data pipelines
#5Earlier quoted context omitted.
What does this mean exactly? Containerized everything? Cloud IDEs?
However you do it, the ability to spin up a fresh env with one click or so
Yep, basically “cattle, not pets” in code. There should only ever be “what runs and production” and “the best imitation of what runs in production” dynamically spun up. Anything else is going to hurt you in the long run
Re: Dev / Stage / Prod is the wrong pattern for data pipelines
#6Re: Dev / Stage / Prod is the wrong pattern for data pipelines
#7Is there some more examples or blog posts that talk about this? I find the idea interesting and possibly very applicable in my work but just from this post alone I don’t feel like I have grasped it well enough to implement this.
Re: Dev / Stage / Prod is the wrong pattern for data pipelines
#8Sounds like a complicated way of saying every developer should have their own DB instead of a shared dev instance.
Re: Dev / Stage / Prod is the wrong pattern for data pipelines
#9This doesn't scale well. It is a perfectly fine approach for smaller systems with a few dependencies, but you are going to have serious headaches whenever you (1) start to see more complex internal system dependencies, and/or (2) start taking on deeper integration with external systems like cloud infrastructure, other services, etc. Once you hit this inflection point, you either start getting very robust with your integration boundaries (and likely developing more complex 'stubs'), or go the dev/stg/prod route.
You start with a database? Great. But wait, you need bulk storage now, so you start sticking it in a cloud bucket (and ensure you use a separate namespace for it). But then Team 2 introduced a new service you now need to spin up in a separate container, so you pull their repo. Then there's a production issue that could have been solved by proper AB testing, so you decide to go with a third party solution that offers that. The party continues, and soon your simple one-click setup ends up so complicated you end up with a full time person just keeping it alive. Whoops! Someone got the cloud namespace wrong on their desktop instance, and production data got hosed. Etc.
Re: Dev / Stage / Prod is the wrong pattern for data pipelines
#10Earlier quoted context omitted.
However you do it, the ability to spin up a fresh env with one click or so
So, like start virtual machine snapshot
Beware, recreating a VM snapshot can be very difficult, and its lineage may become an unreproduceable "pet".
Something you are able to construct from scratch on demand, automatically, is preferable.