Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLM Workflows then Agents: Getting Started with Apache Airflow

github.com

Post reply on HN
Print view View original

11–20 of 46 posts

Re: LLM Workflows then Agents: Getting Started with Apache Airflow

#11

Post by bashfulpup »

mushufasa wrote: post #5

this is really cool! That said, my impression is that Airflow is a really dated choice for a greenfield project. There isn't a clear successor though. I looked into this recently, and was quickly overwhelmed by Prefect, Dagster, Temporal, and even newer ones like Hatchet and Hamilton Most of these frameworks now have docs / plugins / sister libraries geared around AI agents It would be really helpful to read a good t…

This space is honestly a mess. I did an in depth survey around 1.5 yrs ago and my eventual conclusion was just to build with airflow.

You either get simplicity with the caveate that your systems need to perfectly align.

Or you get complexity but will work with basically anything (airflow).

Re: LLM Workflows then Agents: Getting Started with Apache Airflow

#13

Post by febed »

bashfulpup wrote: post #11
mushufasa wrote: post #5

this is really cool! That said, my impression is that Airflow is a really dated choice for a greenfield project. There isn't a clear successor though. I looked into this recently, and was quickly overwhelmed by Prefect, Dagster, Temporal, and even newer ones like Hatchet and Hamilton Most of these frameworks now have docs / plugins / sister libraries geared around AI agents It would be really helpful to read a good t…

This space is honestly a mess. I did an in depth survey around 1.5 yrs ago and my eventual conclusion was just to build with airflow. You either get simplicity with the caveate that your systems need to perfectly align. Or you get complexity but will work with basically anything (airflow).

Would be interested to know what drawbacks you found with Dagster or Prefect.

Re: LLM Workflows then Agents: Getting Started with Apache Airflow

#14

Post by gre »

Hasz wrote: post #7
mushufasa wrote: post #5

this is really cool! That said, my impression is that Airflow is a really dated choice for a greenfield project. There isn't a clear successor though. I looked into this recently, and was quickly overwhelmed by Prefect, Dagster, Temporal, and even newer ones like Hatchet and Hamilton Most of these frameworks now have docs / plugins / sister libraries geared around AI agents It would be really helpful to read a good t…

Dated doesn’t mean bad (usually the opposite in my experience!) What issues do you have with Airflow?

Here's my problems with MWAA (amazon hosted airflow.) I have about 100 dags which maxes out the scheduler thread. Airflow parses all the files every minute so it's always parsing around 94% cpu. I could run a second scheduler thread if I coordinate with my SRE team and get the terraform deployed...it's really tedious.

Related possibly, my dags get kill -9 for no apparent reason. The RAM usage is not that high, maybe 2gb out of 8gb system RAM in use. No reason is given in the logs.

I am trying to switch to dagster, not because it's awesome, but because it hasn't crashed randomly on me.

Re: LLM Workflows then Agents: Getting Started with Apache Airflow

#15

Post by alittletooraph2 »

ldjkfkdsjnv wrote: post #12

Extremely bearish on existing tools solving agentic workflows well. If anyone, it will be temporal. Airflow and the like simply were not designed for high dynamic execution, and so have all sorts of annoyances that will make them lose.

Temporal’s great! That being said, there is something about being able to orchestrate LLMs and agents using what many already use to orchestrate their data workflows because there’s already proven out reliability, scalability, observability, etc. I’m sure there are boundary conditions for really advanced agentic workflows though…

Re: LLM Workflows then Agents: Getting Started with Apache Airflow

#16

Post by jt_b »

febed wrote: post #13
bashfulpup wrote: post #11

Earlier quoted context omitted.

This space is honestly a mess. I did an in depth survey around 1.5 yrs ago and my eventual conclusion was just to build with airflow. You either get simplicity with the caveate that your systems need to perfectly align. Or you get complexity but will work with basically anything (airflow).

Would be interested to know what drawbacks you found with Dagster or Prefect.

Prefect is amazing. Built out an ETL pipeline system with it at last job and would love to get it incorporated in the current one, but unfortunately have a lot of legacy stuff in Airflow. Being able to debug stuff locally was amazing and super clean integration with K8S.

Re: LLM Workflows then Agents: Getting Started with Apache Airflow

#17

Post by drdaeman »

I'm sorry, I don't really know Airflow, but what's the point of `@task.agent`, as compared to plain old `return my_agent.run_sync(...)`? To me it feels like a more restrictive[1], and possibly less intuitive[2] API.

[1]: Limited to what decorator arguments can do. I suspect it could become an issue with `@task.branch` if some post-processing would be needed to adjust for smaller models' finickinesses.

[2]: As the final step is described at the top of the function.

Re: LLM Workflows then Agents: Getting Started with Apache Airflow

#19

Post by nikolayasdf123 »

bashfulpup wrote: post #11
mushufasa wrote: post #5

this is really cool! That said, my impression is that Airflow is a really dated choice for a greenfield project. There isn't a clear successor though. I looked into this recently, and was quickly overwhelmed by Prefect, Dagster, Temporal, and even newer ones like Hatchet and Hamilton Most of these frameworks now have docs / plugins / sister libraries geared around AI agents It would be really helpful to read a good t…

This space is honestly a mess. I did an in depth survey around 1.5 yrs ago and my eventual conclusion was just to build with airflow. You either get simplicity with the caveate that your systems need to perfectly align. Or you get complexity but will work with basically anything (airflow).

+1 to this. other solutions over-promise, under-deliver, poor developer relations and communication, "open-source, but pay us" style open-source, and is indeed a mess

Re: LLM Workflows then Agents: Getting Started with Apache Airflow

#20

Post by jedberg »

ldjkfkdsjnv wrote: post #12

Extremely bearish on existing tools solving agentic workflows well. If anyone, it will be temporal. Airflow and the like simply were not designed for high dynamic execution, and so have all sorts of annoyances that will make them lose.

Have you checked out DBOS Transact[0]? DBOS is designed for high dynamic execution, and doesn't have the overhead or complexity of Temporal [1].

Disclosure, I'm the CEO of DBOS.

[0] https://github.com/dbos-inc/dbos-transact-py

[1] https://www.dbos.dev/blog/durable-execution-coding-compariso...

Post reply on HN