Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLM Workflows then Agents: Getting Started with Apache Airflow

github.com

Post reply on HN
Print view View original

41–46 of 46 posts

Re: LLM Workflows then Agents: Getting Started with Apache Airflow

#41

Post by jlaneve »

greatgib wrote: post #31

Decorators in the usage example looks useless, and more to show off than being a real convenience. In real life program, I don't think that you will have hundreds of calls to LLM or agent in your app so much that you have any code gains to decorator but at the opposite the decorator will make it very hard to have parametric values or values not hard coded but from config that you don't set up upfront at application s…

Disclaimer: author of the SDK here.

Airflow actually uses decorators to indicate something is an explicit task in a data pipeline vs just a utility function, so this follows that pattern!

It also uses an "operator" under the hood (Airflow's term for a pre-built, parameterized task) which can be subclassed and customized if you want to do any customization.

Re: LLM Workflows then Agents: Getting Started with Apache Airflow

#42

Post by jedberg »

ldjkfkdsjnv wrote: post #37
jedberg wrote: post #20

Earlier quoted context omitted.

Have you checked out DBOS Transact[0]? DBOS is designed for high dynamic execution, and doesn't have the overhead or complexity of Temporal [1]. Disclosure, I'm the CEO of DBOS. [0] https://github.com/dbos-inc/dbos-transact-py [1] https://www.dbos.dev/blog/durable-execution-coding-compariso...

I have seen it! And appreciate your response, just haven't had the time to dive in. If you want a product hint from me, I think that adding integrations natively into the platform that would allow vibe coders to build asynchronous agents easier would really boost revenue. Like email, text, etc. Probably not your vision, just a suggestion

> that would allow vibe coders

We've experimented with that actually! Six months ago it was terrible, but the new models are getting pretty good.

And it's definitely easier for an AI to generate DBOS code to make a fully formed distributed system than a fully formed distributed system somewhere else.

Re: LLM Workflows then Agents: Getting Started with Apache Airflow

#43

Post by jedberg »

hbarka wrote: post #29
jedberg wrote: post #23

Earlier quoted context omitted.

I'd be curious if this scratches your itch: https://www.dbos.dev/blog/durable-execution-crashproof-ai-ag...

Pleasantly surprised to see the name Mike Stonebraker in the About Us.

About to jump into an eng meeting with him right now!

Re: LLM Workflows then Agents: Getting Started with Apache Airflow

#44

Post by ldjkfkdsjnv »

jedberg wrote: post #42
ldjkfkdsjnv wrote: post #37

Earlier quoted context omitted.

I have seen it! And appreciate your response, just haven't had the time to dive in. If you want a product hint from me, I think that adding integrations natively into the platform that would allow vibe coders to build asynchronous agents easier would really boost revenue. Like email, text, etc. Probably not your vision, just a suggestion

> that would allow vibe coders We've experimented with that actually! Six months ago it was terrible, but the new models are getting pretty good. And it's definitely easier for an AI to generate DBOS code to make a fully formed distributed system than a fully formed distributed system somewhere else.

Making an asynchronous ai agent is still hard, there is a disconnect between the agentic LLM code (langgraph, openai agents, etc) and asynchronous distributed systems/message passing. True AI agents will have a cohesive joining of the two.

Re: LLM Workflows then Agents: Getting Started with Apache Airflow

#45

Post by bashfulpup »

febed wrote: post #13
bashfulpup wrote: post #11

Earlier quoted context omitted.

This space is honestly a mess. I did an in depth survey around 1.5 yrs ago and my eventual conclusion was just to build with airflow. You either get simplicity with the caveate that your systems need to perfectly align. Or you get complexity but will work with basically anything (airflow).

Would be interested to know what drawbacks you found with Dagster or Prefect.

Other guy said it right. These work and are fine but you lose the legacy stuff. If you know your limits and where the eventual system will end up it's great and probably better.

If you are building a expandable long term system and you want all the goodies baked in choose airflow.

Pretty much the same as any architecture choice. Ugly/hard often means control and features, pretty/easy means less of both.

On the surface the differences are not very noticable other than the learning curve of getting started.

Re: LLM Workflows then Agents: Getting Started with Apache Airflow

#46

Post by tomwheeler »

ldjkfkdsjnv wrote: post #36
acchow wrote: post #28

Earlier quoted context omitted.

Temporal is for a static graph with idempotent nodes. Powerful LLM workflows don’t fit this model.

Temporal is absolutely not for a static graph, idempotent nodes yes. Please explain your argument more

> Temporal is absolutely not for a static graph

I'd clarify this to say "Temporal is absolutely not limited to a static graph." It can certainly handle a static graph, but it can also handle a dynamic one. Here is an example in Go (https://github.com/temporalio/samples-go/tree/main/choice-mu...), there are similar ones for other languages.

I think the confusion might stem from the determinism requirement in Temporal (and other replay-based Durable Execution platforms). It's not the Workflow Definition (i.e., the code) that must be deterministic, it's the Workflow Execution (i.e., a specific running instance of that code) that must be deterministic. Each running instance is allowed to take a different path through that code, so long as it does so consistently when executed with the same input.

Post reply on HN