LLM Workflows then Agents: Getting Started with Apache Airflow
1–10 of 46 posts
Re: LLM Workflows then Agents: Getting Started with Apache Airflow
#2[dead]
Re: LLM Workflows then Agents: Getting Started with Apache Airflow
#3I'm looking into using LLM calls inside SQL Triggers to make agents / 'agentic' workflows. Having LLM powered workflows can get you powerful results and are basically the equivalent of 'spinning up' an agent.
Re: LLM Workflows then Agents: Getting Started with Apache Airflow
#4Truthfully have been a little skeptical of how many workloads will actually need “agents” vs doing something totally deterministic with a little LLM augmentation. Seems like I’m not the only one that thinks the latter works a lot of the time!
Re: LLM Workflows then Agents: Getting Started with Apache Airflow
#5this is really cool!
That said, my impression is that Airflow is a really dated choice for a greenfield project. There isn't a clear successor though. I looked into this recently, and was quickly overwhelmed by Prefect, Dagster, Temporal, and even newer ones like Hatchet and Hamilton
Most of these frameworks now have docs / plugins / sister libraries geared around AI agents
It would be really helpful to read a good technical blog doing a landscape of design patterns in these different approaches, and thoughts on how to fit things together well into a pipeline given various quirks of LLMs (e.g. nondeterminism).
This page is a good start, even if it is written as an airflow-specific how-to!
Re: LLM Workflows then Agents: Getting Started with Apache Airflow
#6[deleted]
Re: LLM Workflows then Agents: Getting Started with Apache Airflow
#7this is really cool! That said, my impression is that Airflow is a really dated choice for a greenfield project. There isn't a clear successor though. I looked into this recently, and was quickly overwhelmed by Prefect, Dagster, Temporal, and even newer ones like Hatchet and Hamilton Most of these frameworks now have docs / plugins / sister libraries geared around AI agents It would be really helpful to read a good t…
Dated doesn’t mean bad (usually the opposite in my experience!) What issues do you have with Airflow?
Re: LLM Workflows then Agents: Getting Started with Apache Airflow
#8this is really cool! That said, my impression is that Airflow is a really dated choice for a greenfield project. There isn't a clear successor though. I looked into this recently, and was quickly overwhelmed by Prefect, Dagster, Temporal, and even newer ones like Hatchet and Hamilton Most of these frameworks now have docs / plugins / sister libraries geared around AI agents It would be really helpful to read a good t…
[dead]
Re: LLM Workflows then Agents: Getting Started with Apache Airflow
#9I'm looking into using LLM calls inside SQL Triggers to make agents / 'agentic' workflows. Having LLM powered workflows can get you powerful results and are basically the equivalent of 'spinning up' an agent.
Know of any online examples of the same?
Re: LLM Workflows then Agents: Getting Started with Apache Airflow
#10This is about workflows that use AI, but it lead me to actually think of the inverse - has anyone experimented with AI agents defining and iterating upon long-running workflows?