Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

12-factor Agents: Patterns of reliable LLM applications

github.com

Post reply on HN
Print view View original

61–70 of 84 posts

Re: 12-factor Agents: Patterns of reliable LLM applications

#61

Post by dhorthy »

hhimanshu wrote: post #57
dhorthy wrote: post #21

Earlier quoted context omitted.

This is a great bit of feedback - what kinda of use cases do you think would make sense? Definitely wanna evolve this in the open with the community

May be if you pick a real-world agent workflow (toy from your production experience, trim it down), and showcase how all these factors will come along in a project. I am inspired by the simplicity of these 12 factors and definitely want to learn more with an example that embraces these factors.

I link in a few places to https://github.com/got-agents/agents where I have a few of these real agents

Re: 12-factor Agents: Patterns of reliable LLM applications

#62

Post by dhorthy »

mettamage wrote: post #60

I've noticed some of these factors myself as well. I'd love to build more AI applications like this. Currently I'm a data analyst and they don't fully appreciate that I can build stuff like this as it is not a technology oriented company. I'd love to work on stuff like this full-time. If anyone is interested in a chat, my email is on my profile (US/EU).

cool thing about open source is you can work on whatever you want, and it’s the best way to meet people who do similar work for their day job as well

Re: 12-factor Agents: Patterns of reliable LLM applications

#63

Post by dhorthy »

daxfohl wrote: post #22

Another one: plan for cost at scale. These things aren't cheap at scale, so whenever something might be handled by a deterministic component, try that first. Not only save on hallucinations and latency, but could make a huge difference in your bottom line.

Yeah definitely. I think the pattern I see people using most is “start with slow, expensive, but low dev effort, and then refine overtime as you fine speed/quality/cost bottlenecks worth investing in”

Re: 12-factor Agents: Patterns of reliable LLM applications

#64

Post by dhorthy »

wfn wrote: post #58

This could not have come at a better time for me, thank you! I've been tinkering with an idea for an audiovisual sandbox[1] (like vvvv[2] but much simpler of course, barebones). Idea is to have a way to insert LM (or some simple locally run neural net) "nodes" which are given specific tasks and whose output is expected to be very constrained. Hence your example: "question -> answer: float" Is very attractive here. Of…

Typed outputs from an LLM is a game changer!

Re: 12-factor Agents: Patterns of reliable LLM applications

#66

Post by dhorthy »

mgdev wrote: post #45

These are great. I had my own list of takeaways [0] after doing this for a couple years, though I wouldn't go so far as calling mine factors. Like you, biggest one I didn't include but would now is to own the lowest level planning loop. It's fine to have some dynamic planning, but you should own an OODA loop (observe, orient, decide, act) and have heuristics for determining if you're converging on a solution (e.g. sc…

this guide is great, i liked the "chat interfaces are dumb" take - totally agree. AI-based UIs have a very long way to go

Re: 12-factor Agents: Patterns of reliable LLM applications

#67

Post by dhorthy »

hhimanshu wrote: post #55

I am wondering how libraries like DSPY [0] fits in your factor-2 [1] As I was reading, I saw mention of BAML > (the above example uses BAML to generate the prompt ... Personally, in my experience hand-writing prompts for extracting structured information from unstructured data has never been easy. With DSPY, my experience has been quite good so far. As you have used raw prompt from BAML, what do you think of using th…

interesting - I think I have to side with the Boundary (YC W23) folks on this one - if you want bleeding edge performance, you need to be able to open the box and hack on the insides.

I don't agree fully with this article https://www.chrismdp.com/beyond-prompting/ but the comparison of punchards -> assembly -> c -> higher langs is quite useful here

I just don't know when we'll get the right abstraction - i don't think langchain or dspy are the "C programming language" of AI yet (they could get there!).

For now I'll stick to my "close to the metal" workbench where I can inspect tokens, reorder special tokens like system/user/JSON, and dynamically keep up with the idiosyncrasies of new models without being locked up waiting for library support.

Re: 12-factor Agents: Patterns of reliable LLM applications

#68

Post by chrismdp »

dhorthy wrote: post #67
hhimanshu wrote: post #55

I am wondering how libraries like DSPY [0] fits in your factor-2 [1] As I was reading, I saw mention of BAML > (the above example uses BAML to generate the prompt ... Personally, in my experience hand-writing prompts for extracting structured information from unstructured data has never been easy. With DSPY, my experience has been quite good so far. As you have used raw prompt from BAML, what do you think of using th…

interesting - I think I have to side with the Boundary (YC W23) folks on this one - if you want bleeding edge performance, you need to be able to open the box and hack on the insides. I don't agree fully with this article https://www.chrismdp.com/beyond-prompting/ but the comparison of punchards -> assembly -> c -> higher langs is quite useful here I just don't know when we'll get the right abstraction - i don't thin…

It's always true that you need to drop down a level of abstraction in order to extract the ultimate performance. (eg I wrote a decent-sized game + engine entirely in C about 10 years ago and played with SIMD vectors to optimise the render loop)

However, I think the vast majority of use cases will not require this level of control, and we will abandon prompts once the tools improve.

Langchain and DSPY are also not there for me either - I think the whole idea of prompting + evals needs a rethink.

(full disclaimer: I'm working on such a tool right now!)

Re: 12-factor Agents: Patterns of reliable LLM applications

#69

Post by [deleted] »

chrismdp wrote: post #68
dhorthy wrote: post #67

Earlier quoted context omitted.

interesting - I think I have to side with the Boundary (YC W23) folks on this one - if you want bleeding edge performance, you need to be able to open the box and hack on the insides. I don't agree fully with this article https://www.chrismdp.com/beyond-prompting/ but the comparison of punchards -> assembly -> c -> higher langs is quite useful here I just don't know when we'll get the right abstraction - i don't thin…

It's always true that you need to drop down a level of abstraction in order to extract the ultimate performance. (eg I wrote a decent-sized game + engine entirely in C about 10 years ago and played with SIMD vectors to optimise the render loop) However, I think the vast majority of use cases will not require this level of control, and we will abandon prompts once the tools improve. Langchain and DSPY are also not the…

[deleted]

Re: 12-factor Agents: Patterns of reliable LLM applications

#70

Post by dhorthy »

chrismdp wrote: post #68
dhorthy wrote: post #67

Earlier quoted context omitted.

interesting - I think I have to side with the Boundary (YC W23) folks on this one - if you want bleeding edge performance, you need to be able to open the box and hack on the insides. I don't agree fully with this article https://www.chrismdp.com/beyond-prompting/ but the comparison of punchards -> assembly -> c -> higher langs is quite useful here I just don't know when we'll get the right abstraction - i don't thin…

It's always true that you need to drop down a level of abstraction in order to extract the ultimate performance. (eg I wrote a decent-sized game + engine entirely in C about 10 years ago and played with SIMD vectors to optimise the render loop) However, I think the vast majority of use cases will not require this level of control, and we will abandon prompts once the tools improve. Langchain and DSPY are also not the…

i'd be interested to check it out

here's a take, I adapted this from someone on the notebookLM team on swyx's podcast

> the only way to build really impressive experiences in AI, is to find something right at the edge of the model's capability, and to get it right consistently.

So in order to build something very good / better than the rest, you will always benefit from being able to bring in every optimization you can.

Post reply on HN