31–40 of 46 posts
In real life program, I don't think that you will have hundreds of calls to LLM or agent in your app so much that you have any code gains to decorator but at the opposite the decorator will make it very hard to have parametric values or values not hard coded but from config that you don't set up upfront at application startup like globals. That is a bad practice...
I'm looking into using LLM calls inside SQL Triggers to make agents / 'agentic' workflows. Having LLM powered workflows can get you powerful results and are basically the equivalent of 'spinning up' an agent.
It just exposes set of functions to propagate the DAG through states and queues tasks for a task worker to perform the actual work and acknowledge completion or failure back to the sql orchestrator.
I was working on it for last few months and it will be ready in upcoming weeks, first version is dedicated to work on Supabase but I plan to make it agnostic.
If you want to learn more, check out the SQL Core readme which explains the whole concept (https://github.com/pgflow-dev/pgflow/tree/main/pkgs/core#rea...) or my Twitter for updates and some demos (https://x.com/pgflow_dev).
I'm currently building pgflow, which is a simple, postgres-first engine that uses task queues to perform real work.
Have explicit DAG approach, strong typesafety, nice DSL in TypeScript and a dedicated task queue worker that allows it to run solely on Supabase without any external tools.
I'm super close to the alpha release, if you guys want more info, check out the readme for SQL core (https://github.com/pgflow-dev/pgflow/tree/main/pkgs/core#rea...) or my Twitter (https://x.com/pgflow_dev).
Hope that grabs someone attention :-) Cheers
Nice to see some workflow engine action on Hacker News! :-) I'm currently building pgflow, which is a simple, postgres-first engine that uses task queues to perform real work. Have explicit DAG approach, strong typesafety, nice DSL in TypeScript and a dedicated task queue worker that allows it to run solely on Supabase without any external tools. I'm super close to the alpha release, if you guys want more info, check…
Nice to see some workflow engine action on Hacker News! :-) I'm currently building pgflow, which is a simple, postgres-first engine that uses task queues to perform real work. Have explicit DAG approach, strong typesafety, nice DSL in TypeScript and a dedicated task queue worker that allows it to run solely on Supabase without any external tools. I'm super close to the alpha release, if you guys want more info, check…
Exactly what I was looking for without even knowing it :) EDIT: well I knew I need smt like this, but I thought I'd had to build a very rudimentary version myself. Thank you for saving me tons of time in my project
I'm very close to releasing an alpha, will post here when ready!
Earlier quoted context omitted.
Temporal’s great! That being said, there is something about being able to orchestrate LLMs and agents using what many already use to orchestrate their data workflows because there’s already proven out reliability, scalability, observability, etc. I’m sure there are boundary conditions for really advanced agentic workflows though…
Temporal is for a static graph with idempotent nodes. Powerful LLM workflows don’t fit this model.
Extremely bearish on existing tools solving agentic workflows well. If anyone, it will be temporal. Airflow and the like simply were not designed for high dynamic execution, and so have all sorts of annoyances that will make them lose.
Have you checked out DBOS Transact[0]? DBOS is designed for high dynamic execution, and doesn't have the overhead or complexity of Temporal [1]. Disclosure, I'm the CEO of DBOS. [0] https://github.com/dbos-inc/dbos-transact-py [1] https://www.dbos.dev/blog/durable-execution-coding-compariso...
If you want a product hint from me, I think that adding integrations natively into the platform that would allow vibe coders to build asynchronous agents easier would really boost revenue. Like email, text, etc.
Probably not your vision, just a suggestion
Earlier quoted context omitted.
Dated doesn’t mean bad (usually the opposite in my experience!) What issues do you have with Airflow?
Here's my problems with MWAA (amazon hosted airflow.) I have about 100 dags which maxes out the scheduler thread. Airflow parses all the files every minute so it's always parsing around 94% cpu. I could run a second scheduler thread if I coordinate with my SRE team and get the terraform deployed...it's really tedious. Related possibly, my dags get kill -9 for no apparent reason. The RAM usage is not that high, maybe…
Earlier quoted context omitted.
Here's my problems with MWAA (amazon hosted airflow.) I have about 100 dags which maxes out the scheduler thread. Airflow parses all the files every minute so it's always parsing around 94% cpu. I could run a second scheduler thread if I coordinate with my SRE team and get the terraform deployed...it's really tedious. Related possibly, my dags get kill -9 for no apparent reason. The RAM usage is not that high, maybe…
This feels like an MWAA issue but I understand how that often gets conflated with it being an Airflow issue.
I realize Amazon is taking an open source project and making a ton of money on it (the instance prices are ridiculous for what you get) and the incentives are misaligned for the Airflow team to help AWS make it better unless AWS paid them to help fix it.
It's crap all around, and Airflow gets a bad rap from AWS's terrible MWAA product based on it.
I'm sorry, I don't really know Airflow, but what's the point of `@task.agent`, as compared to plain old `return my_agent.run_sync(...)`? To me it feels like a more restrictive[1], and possibly less intuitive[2] API. [1]: Limited to what decorator arguments can do. I suspect it could become an issue with `@task.branch` if some post-processing would be needed to adjust for smaller models' finickinesses. [2]: As the fin…
It is _potentially_ more restrictive than writing pure Python functions, but the plus side is that we can interject certain Airflow-specific features into how the agent runs. And this isn't mean for someone who knows agents inside & out / wants the low-level customizability.
The best example of this today is log groups: Airflow lets you log things out as part of a "group" which has some UI abstractions to make it easier. This SDK takes the raw agent tool calls and turns them each into a log group, so you can see a) at a high level what the agent is doing, and b) drill down into a specific tool call to understand what's happening within the tool call.
To your point about the `@task.llm_branch`, the SDK & Pydantic AI (which the SDK uses under the hood) will re-prompt the LLM up to a certain number of attempts if it receives output that isn't the name of a downstream task. So there shouldn't be much finickiness.