Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dabbling with Dagster vs. Airflow

davidsj.substack.com

Post reply on HN
Print view View original

51–56 of 56 posts

Re: Dabbling with Dagster vs. Airflow

#51

Post by SOLAR_FIELDS »

karakanb wrote: post #48

Disclaimer: I am building a no-code, SQL-focused data pipeline platform to improve the experience around data pipelines, see my profile for more. The idea is to replace all of Fivetran, DBT, Airflow and more with a single platform that can handle all with no code. I like the fact that the articles the author has are walking the reader through their own journey, along with the learnings and opinions after the trial pe…

All I can say about your aspirations is: good luck. I’m working for a startup in the space currently, and this is a billion dollar valued flush with cash startup that is only aiming to capture one of the several segments you mentioned.

If you can solve all of these problems, you deserve to be rich. If you are a little unsure, perhaps take one of the segments you mentioned and focus on that.

Re: Dabbling with Dagster vs. Airflow

#52

Post by nerdponx »

geodel wrote: post #37
stuckinhell wrote: post #33

I hear grumbling from a friend on the data engineering side of my company. It takes an enormous amount of effort to stay on top of their data pipelines, and they still have lots of failures in cleaning, transformation, orchestration, and reporting. One product wasn't updated in 5 months ! They've tried everything airflow, informatica, alteryx, etc. They've even built their own custom data flow etl in python. I often…

Well it all starts with seductive but mostly bullshit idea that once all the data is transformed and put in some common data lake some amazing, deep insights can be extracted which were not possible before. And there is large budget to be spent on developers, hardware, software, consulting and so on. Who will say no to this?

I mean, I have actually used a data lake and it worked pretty well for its intended purpose. Fuck having to email 5 different people (and have a 50 minute meeting) and wait 3 days (or 3 weeks) for them all to send me fucked up CSVs and Zip files full of XML and whatever the hell else, with no clear provenance and absolutely no documentation.

Re: Dabbling with Dagster vs. Airflow

#53

Post by samhita-alla »

valenterry wrote: post #7

Where is a service that doesn't focus specifically on 1.) data pipelines / data-science 2.) cicd / build pipelines 3.) ... you name it I mean, just a service that gives me all the groundwork to build one of the above myself. Is there something like that?

Have you seen https://github.com/flyteorg/flyte? It's a data, ML and infrastructure orchestrator. (btw, I'm a Flyte engineer)

Re: Dabbling with Dagster vs. Airflow

#54

Post by swyx »

SOLAR_FIELDS wrote: post #51
karakanb wrote: post #48

Disclaimer: I am building a no-code, SQL-focused data pipeline platform to improve the experience around data pipelines, see my profile for more. The idea is to replace all of Fivetran, DBT, Airflow and more with a single platform that can handle all with no code. I like the fact that the articles the author has are walking the reader through their own journey, along with the learnings and opinions after the trial pe…

All I can say about your aspirations is: good luck. I’m working for a startup in the space currently, and this is a billion dollar valued flush with cash startup that is only aiming to capture one of the several segments you mentioned. If you can solve all of these problems, you deserve to be rich. If you are a little unsure, perhaps take one of the segments you mentioned and focus on that.

2 ways to make money - bundling and unbundling. unbundling works until a competent enough founder actually makes a good enough bundle. its an enormous undertaking to rebundle an unbundled industry, which data eng basically is

Re: Dabbling with Dagster vs. Airflow

#56

Post by valenterry »

samhita-alla wrote: post #53
valenterry wrote: post #7

Where is a service that doesn't focus specifically on 1.) data pipelines / data-science 2.) cicd / build pipelines 3.) ... you name it I mean, just a service that gives me all the groundwork to build one of the above myself. Is there something like that?

Have you seen https://github.com/flyteorg/flyte ? It's a data, ML and infrastructure orchestrator. (btw, I'm a Flyte engineer)

Bonus points for having a Scala SDK!
Post reply on HN