Disclaimer: I am building a no-code, SQL-focused data pipeline platform to improve the experience around data pipelines, see my profile for more. The idea is to replace all of Fivetran, DBT, Airflow and more with a single platform that can handle all with no code.
I like the fact that the articles the author has are walking the reader through their own journey, along with the learnings and opinions after the trial periods. I am curious to hear how Dagster will evolve for their usage.
One of the things that has been a big discussion point among data folks that I have been talking to is that people that haven't done ops before underestimate the amount of operations that go into managing an Airflow instance, there are still quite a lot of stuff to be figured out:
- how do I get my pipeline code there?
- where do I execute them?
- how do I setup my development environment?
- how do I make sure the platform is up and running?
- how do I know if a task fails?
- where do I store my logs?
- how do I scale my setup?
If you are using a managed solution like MWAA or Cloud Composer, some of these questions might go away, but some aren't. As it stands today, Airflow is a powerful but hard-to-use technology; in my opinion, it is less of a tool that is supposed to be used directly by data engineers / analysts, and more of a platform that should enable easier-to-use platforms for its internal users.
In that sense, I believe Dagster is hitting the right chord: they focus on the pain points in DX for Airflow and similar solutions, they have figured out how to do development branches, they are focusing on assets rather than tasks, and they are constantly improving their product as far as I can tell from the outside. However, it is still a platform that you have to have engineers writing code for it: asking a data analyst to write python code to schedule a few SQL queries still adds a huge barrier to entry.
I am excited to see all the innovation that is happening in this space. I find the point about people getting used to the baggage around Airflow to be a quite real problem, and I am very happy to see solutions like Dagster are gaining speed. All in all, it is a very large space and there are many different problems that need to be solved to add the data abilities larger organizations have to smaller companies with limited budgets.