11–20 of 56 posts
Where is a service that doesn't focus specifically on 1.) data pipelines / data-science 2.) cicd / build pipelines 3.) ... you name it I mean, just a service that gives me all the groundwork to build one of the above myself. Is there something like that?
Also, as other's have mentioned the comparison to MWAA is unfair, the true own of Airflow is Astronomer as they have over 50% of the commits to Airflow, and Astronomer is a much better product the MWAA.
Dabbling with Dagster vs. Angling with Airflow?
Where is a service that doesn't focus specifically on 1.) data pipelines / data-science 2.) cicd / build pipelines 3.) ... you name it I mean, just a service that gives me all the groundwork to build one of the above myself. Is there something like that?
Argo Workflows
The author seems to think that Dagster or Prefect will take over Airflow, I don't think this is true. All of them being open source means that if one has a good idea or better way of doing something the other can quickly implement the feature and even use the some of same code. We saw it with Airflow implementing the TaskFlow API as a response to Dagster, and in a few weeks Airflow 2.4 is going to have dataset schedu…
sounds like a very positive experience all around, but not going too deep into dagster itself yet. i feel like comparing MWAA to Dagster is not an even footing, i'd be interested in seeing how Astronomer has also improved the Airflow experience.
The author seems to think that Dagster or Prefect will take over Airflow, I don't think this is true. All of them being open source means that if one has a good idea or better way of doing something the other can quickly implement the feature and even use the some of same code. We saw it with Airflow implementing the TaskFlow API as a response to Dagster, and in a few weeks Airflow 2.4 is going to have dataset schedu…
Where is a service that doesn't focus specifically on 1.) data pipelines / data-science 2.) cicd / build pipelines 3.) ... you name it I mean, just a service that gives me all the groundwork to build one of the above myself. Is there something like that?
If you want to go even lower level, a framework like DTFx lets you define long-running, distributed and resilient orchestrations in code:
Prefect is generally very easy to use. Essentially, you: (a) write a Python-based flow, which defines some job to run (with subtasks), (b) turn on an orchestrator on a server somewhere, (c) turn on an agent on a server somewhere (to run the flow when instructed by the orchestrator), and (d) connect to the orchestrator, build & apply a deployment, and run it.
I find the docs a little half baked right now. One example is that cron jobs, which one would think are essential to something like Prefect, basically can't be done (as of a month ago) without touching the Prefect UI. This is extremely odd.
I also found it fairly confusing which components were supposed to be checked into source control, and which weren't. I blame this on Python deployment generally being very odd and confusing, but Prefect docs don't make it any more clear. Prefect assumes that there's an S3-like storage that both the submitting computer (my laptop) and the orchestrator (the server) can access.
Overall I find it quite handy, and probably won't switch. It feels more lightweight than say using full Docker containers, which we probably don't need right now. The UI is nicer than Airflow's, and the orchestrator & agent are much easier on resources. It feels more reproducible. I haven't tried Prefect Cloud, and we're unlikely to (security & cost are the main reasons).
Where is a service that doesn't focus specifically on 1.) data pipelines / data-science 2.) cicd / build pipelines 3.) ... you name it I mean, just a service that gives me all the groundwork to build one of the above myself. Is there something like that?