21–30 of 46 posts
Extremely bearish on existing tools solving agentic workflows well. If anyone, it will be temporal. Airflow and the like simply were not designed for high dynamic execution, and so have all sorts of annoyances that will make them lose.
Truthfully have been a little skeptical of how many workloads will actually need “agents” vs doing something totally deterministic with a little LLM augmentation. Seems like I’m not the only one that thinks the latter works a lot of the time!
this is really cool! That said, my impression is that Airflow is a really dated choice for a greenfield project. There isn't a clear successor though. I looked into this recently, and was quickly overwhelmed by Prefect, Dagster, Temporal, and even newer ones like Hatchet and Hamilton Most of these frameworks now have docs / plugins / sister libraries geared around AI agents It would be really helpful to read a good t…
https://www.dbos.dev/blog/durable-execution-crashproof-ai-ag...
Earlier quoted context omitted.
Dated doesn’t mean bad (usually the opposite in my experience!) What issues do you have with Airflow?
Here's my problems with MWAA (amazon hosted airflow.) I have about 100 dags which maxes out the scheduler thread. Airflow parses all the files every minute so it's always parsing around 94% cpu. I could run a second scheduler thread if I coordinate with my SRE team and get the terraform deployed...it's really tedious. Related possibly, my dags get kill -9 for no apparent reason. The RAM usage is not that high, maybe…
this is really cool! That said, my impression is that Airflow is a really dated choice for a greenfield project. There isn't a clear successor though. I looked into this recently, and was quickly overwhelmed by Prefect, Dagster, Temporal, and even newer ones like Hatchet and Hamilton Most of these frameworks now have docs / plugins / sister libraries geared around AI agents It would be really helpful to read a good t…
Dated doesn’t mean bad (usually the opposite in my experience!) What issues do you have with Airflow?
Their operational perspective is catastrophic; how does one view the logs for a dag through the UI[1]? Why can't it store the python in the database they have attached to their deployment, versus making me jump through 80,000 hoops to put the files in the right magic directory on disk of every worker[2]?
1: no, not https://airflow.apache.org/docs/apache-airflow/stable/ui.htm...> I mean the log, you know, like in the old days of $(tail -f /var/log/the.thing). I'm open to the answer hiding somewhere in this gobbledygook https://airflow.apache.org/docs/apache-airflow/stable/admini...> but who is the target audience for having such a fancy UI and omitting log viewing from it, doubly so if there's some alleged http just for viewing logs
2: https://airflow.apache.org/docs/apache-airflow/stable/core-c... and double-plus-good anytime python software mentions PYTHONPATH -- that's how you know you're in for a hot good time https://airflow.apache.org/docs/apache-airflow/stable/admini...
Earlier quoted context omitted.
Dated doesn’t mean bad (usually the opposite in my experience!) What issues do you have with Airflow?
> What issues do you have with Airflow? Their operational perspective is catastrophic; how does one view the logs for a dag through the UI[1]? Why can't it store the python in the database they have attached to their deployment, versus making me jump through 80,000 hoops to put the files in the right magic directory on disk of every worker[2]? 1: no, not https://airflow.apache.org/docs/apache-airflow/stable/ui.htm...…
WRT your 1, above any DAG output to stdout/err is available via the logs tab from the graph view of the individual tasks. Almost all our DAGs leverage on the PythonOperator though, not sure if that standardises this for us and your experience is muddied by more complexity than we currently have?
WRT 2. we generate an uber requirements.txt running pyreqs from the pipeline and install everything in the container automatically. Again no issues currently - although we do need to manually add the installation of test libraries to the pipeline job as for some reason auto-discovery is flakier for unit-tests frameworks.
Extremely bearish on existing tools solving agentic workflows well. If anyone, it will be temporal. Airflow and the like simply were not designed for high dynamic execution, and so have all sorts of annoyances that will make them lose.
Temporal’s great! That being said, there is something about being able to orchestrate LLMs and agents using what many already use to orchestrate their data workflows because there’s already proven out reliability, scalability, observability, etc. I’m sure there are boundary conditions for really advanced agentic workflows though…
this is really cool! That said, my impression is that Airflow is a really dated choice for a greenfield project. There isn't a clear successor though. I looked into this recently, and was quickly overwhelmed by Prefect, Dagster, Temporal, and even newer ones like Hatchet and Hamilton Most of these frameworks now have docs / plugins / sister libraries geared around AI agents It would be really helpful to read a good t…
I'd be curious if this scratches your itch: https://www.dbos.dev/blog/durable-execution-crashproof-ai-ag...
I'm looking into using LLM calls inside SQL Triggers to make agents / 'agentic' workflows. Having LLM powered workflows can get you powerful results and are basically the equivalent of 'spinning up' an agent.