Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building an internal agent: Code-driven vs. LLM-driven workflows

lethain.com

Post reply on HN
Print view View original

21–30 of 36 posts

Re: Building an internal agent: Code-driven vs. LLM-driven workflows

#21

Post by tacone »

jumploops wrote: post #18
galaxyLogic wrote: post #10

What I'm struggling with is, when you ask AI to do something, its answer is always undeterministically different, more or less. If I start out with a "spec" that tells AI what I want, it can create working software for me. Seems great. But let's say some weeks, or months or even years later I realize I need to change my spec a bit. I would like to give the new spec to the AI and have it produce an improved version of…

> why can't there be an LLM that would always give the exact same output for the exact same input LLMs are inherently deterministic, but LLM providers add randomness through “temperature” and random seeds. Without the random seed and variable randomness (temperature setting), LLMs will always produce the same output for the same input. Of course, the context you pass to the LLM also affects the determinism in a produ…

No, setting the temperature to zero is still going to yeld different results. One might think they add random seeds, but it makes no sense for temperature zero. One theory is that the distributed nature of their systems adds entropy and thus produces different results each time.

Random seeds might be a thing, but for what I see there's a lot demand for reproducibility and yet no certain way to achieve it.

Re: Building an internal agent: Code-driven vs. LLM-driven workflows

#22

Post by WhiteNoiz3 »

I'm struggling to understand why an LLM even needs to be involved in this at all. Can't you write a script that takes the last 10 slack messages and checks the github status for any URLs and adds an emoji? It could be a script or slack bot and it would work far more reliably and cost nothing in LLM calls. IMO it seems far more efficient to have an LLM write a repeatable workflow once than calling an LLM every time.

Re: Building an internal agent: Code-driven vs. LLM-driven workflows

#23

Post by fragmede »

galaxyLogic wrote: post #10

What I'm struggling with is, when you ask AI to do something, its answer is always undeterministically different, more or less. If I start out with a "spec" that tells AI what I want, it can create working software for me. Seems great. But let's say some weeks, or months or even years later I realize I need to change my spec a bit. I would like to give the new spec to the AI and have it produce an improved version of…

> What I'm struggling with is, when you ask AI to do something, its answer is always undeterministically different, more or less.

For some computer science definition of deterministic, sure, but who gives a shit about that? If I ask it build a login page, and it puts GitHub login first one day, and Google login first the next day, do I care? I'm not building login pages every other day. What point do you want to define as "sufficiently deterministic", for which use case?

"Summarize this essay into 3 sentences" for a human is going to vary from day to day, and yeah, it's weird for computers to no longer be 100% deterministic, but I didn't decide this future for us.

Re: Building an internal agent: Code-driven vs. LLM-driven workflows

#24

Post by shimman »

WhiteNoiz3 wrote: post #22

I'm struggling to understand why an LLM even needs to be involved in this at all. Can't you write a script that takes the last 10 slack messages and checks the github status for any URLs and adds an emoji? It could be a script or slack bot and it would work far more reliably and cost nothing in LLM calls. IMO it seems far more efficient to have an LLM write a repeatable workflow once than calling an LLM every time.

This reminds of when Adam Wathan admitted that LLMs really helped his workflow due to automating the process for turning SVG's into react components... something that can be handled with a single script rather than calling an LLM every time like you mentioned.

Sometimes people just don't know better.

Re: Building an internal agent: Code-driven vs. LLM-driven workflows

#25

Post by heliumtera »

WhiteNoiz3 wrote: post #22

I'm struggling to understand why an LLM even needs to be involved in this at all. Can't you write a script that takes the last 10 slack messages and checks the github status for any URLs and adds an emoji? It could be a script or slack bot and it would work far more reliably and cost nothing in LLM calls. IMO it seems far more efficient to have an LLM write a repeatable workflow once than calling an LLM every time.

Maybe the audience is not developers at all? Someone that does not know anything about computers and computation might not comprehend how easy or complex a given task is. For a whole class of people, checking a key in a json object might be as complex and difficult as creating a compiler. Some of those are in charge of evaluating progress and development of software. Here's the magic, by now everyone can understand that prompting and receiving an answer is easy.

Re: Building an internal agent: Code-driven vs. LLM-driven workflows

#26

Post by EMM_386 »

jumploops wrote: post #18
galaxyLogic wrote: post #10

What I'm struggling with is, when you ask AI to do something, its answer is always undeterministically different, more or less. If I start out with a "spec" that tells AI what I want, it can create working software for me. Seems great. But let's say some weeks, or months or even years later I realize I need to change my spec a bit. I would like to give the new spec to the AI and have it produce an improved version of…

> why can't there be an LLM that would always give the exact same output for the exact same input LLMs are inherently deterministic, but LLM providers add randomness through “temperature” and random seeds. Without the random seed and variable randomness (temperature setting), LLMs will always produce the same output for the same input. Of course, the context you pass to the LLM also affects the determinism in a produ…

> Without the random seed and variable randomness (temperature setting), LLMs will always produce the same output for the same input.

Except they won't.

Even at temperature 0, you will not always get the same output as the same input. And it's not because of random noise from inference providers.

There are papers that explore this subject because for some use-cases - this is extremely important. Everything from floating point precision, hardware timing differences, etc. make this difficult.

Re: Building an internal agent: Code-driven vs. LLM-driven workflows

#27

Post by westoncb »

shimman wrote: post #24
WhiteNoiz3 wrote: post #22

I'm struggling to understand why an LLM even needs to be involved in this at all. Can't you write a script that takes the last 10 slack messages and checks the github status for any URLs and adds an emoji? It could be a script or slack bot and it would work far more reliably and cost nothing in LLM calls. IMO it seems far more efficient to have an LLM write a repeatable workflow once than calling an LLM every time.

This reminds of when Adam Wathan admitted that LLMs really helped his workflow due to automating the process for turning SVG's into react components... something that can be handled with a single script rather than calling an LLM every time like you mentioned. Sometimes people just don't know better.

That depends on the content of the SVGs.. Of course you can write a script to do a very literally kind of conversion of regardless, but in practice a lot of interpretation would be required, and could be done by an LLM. Simple case is an SVG that's a static presentation of a button; the intended React component could handle hover and click states and change the cursor appropriately and set aria label etc. For anything but trivial cases a script isn't going to get you far.

Re: Building an internal agent: Code-driven vs. LLM-driven workflows

#29

Post by matusp »

tacone wrote: post #21
jumploops wrote: post #18

Earlier quoted context omitted.

> why can't there be an LLM that would always give the exact same output for the exact same input LLMs are inherently deterministic, but LLM providers add randomness through “temperature” and random seeds. Without the random seed and variable randomness (temperature setting), LLMs will always produce the same output for the same input. Of course, the context you pass to the LLM also affects the determinism in a produ…

No, setting the temperature to zero is still going to yeld different results. One might think they add random seeds, but it makes no sense for temperature zero. One theory is that the distributed nature of their systems adds entropy and thus produces different results each time. Random seeds might be a thing, but for what I see there's a lot demand for reproducibility and yet no certain way to achieve it.

It's not really a mystery why it happens. LLM APIs are non-deterministic from user's point of view because your request is going to get batched with other users' requests. The batch behavior is deterministic, but your batch is going to be different each time you send your request.

The size of the batch influences the order of atomic float operations. And because float operations are not associative, the results might be different.

Re: Building an internal agent: Code-driven vs. LLM-driven workflows

#30

Post by PacificSpecific »

shimman wrote: post #24
WhiteNoiz3 wrote: post #22

I'm struggling to understand why an LLM even needs to be involved in this at all. Can't you write a script that takes the last 10 slack messages and checks the github status for any URLs and adds an emoji? It could be a script or slack bot and it would work far more reliably and cost nothing in LLM calls. IMO it seems far more efficient to have an LLM write a repeatable workflow once than calling an LLM every time.

This reminds of when Adam Wathan admitted that LLMs really helped his workflow due to automating the process for turning SVG's into react components... something that can be handled with a single script rather than calling an LLM every time like you mentioned. Sometimes people just don't know better.

Reminds me of "XML to classes" and "JSON to classes"
Post reply on HN