Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Automating AI Away

replicated.live

Post reply on HN
Print view View original

31–40 of 69 posts

Re: Automating AI Away

#31

Post by verdverm »

inspectorSlap wrote: post #26
lubujackson wrote: post #2

Makes sense, I have had the biggest wins with AI by attacking nondeterminism whenever possible. BTW, you should probably fix the Beagle link on your homepage: https://replicated.live/beagle/

I find some of the most interesting, and catastrophic failures in my agent fine-tuning come from the clamping down of non-determinism. It is totally the correct approach, but must be handled delicately. The non-deterministic core remains, but now under bimodal pressure.

I think this is less about clamping down on non determinism and more remembering that a script is much more reliable than having the agent do some things. Think making a number of API requests to get info for context or running a sequence of testing steps to generate a report. Remove easy places where that non determinism rears its head and there is really no need. I talk about what I'm doing with PR review in a other comment, as an example.

In other words, are there places where a one liner for the agent would be more reliable than markdown instructions and crossing fingers?

I look at it this way... I wrote scripts over the years to make my life easier. Do the same for your agents and free their attention for the parts that matter.

Re: Automating AI Away

#32

Post by sethhochberg »

vinceguidry wrote: post #16

I'm seeing tons of blog posts which seemingly amount to having AI write code. It would have never occurred to me to repeatedly invoke an LLM to do what a simple script could, but I guess I shouldn't be too surprised. 20 line bash scripts replacing entire enterprise software stacks was a meme even in the 90s.

The concept of "tool building" is one of the areas my team has spent the most time coaching our less-technical employees on since widespread LLM rollout in our company.

Developers and developer-adjacent, technical people tend to think this way on their own... but every business has dark corners where repetitive, manual things still happen. We're leaning a lot on training and even org-wide LLM instructions to try and let the LLM (by its own assessment) be the vehicle use to codify a process and turn it into some good old-fashioned reviewable, deterministic automation.

Re: Automating AI Away

#33

Post by hadi121 »

derdi wrote: post #22
hadi121 wrote: post #14

Earlier quoted context omitted.

I agree with the humans aren't deterministic, but I feel like that wasn't the scope of the original commentator. Humans are not deterministic, yes. Neither are LLMs. Both should be phased out of processes that need to be deterministic. What do you think?

I don't think processes have to be deterministic. Results should be, in the following sense: Both humans and LLMs should write software that is well-written, well-tested, well-documented, and that meets the spec. But this still leaves a lot of room for creativity (or rolling dice).

Oh yeah totally agree

Re: Automating AI Away

#34

Post by FeteCommuniste »

derdi wrote: post #7
hadi121 wrote: post #5

Earlier quoted context omitted.

I love the way you put this. Are there any sites or forums or places where people discuss/hash this out? I've genuinely never considered it from this angle before.

Humans aren't deterministic. Determinism is a red herring. There are lots of other problems with agentic programming, but this is not at the top of the list.

> Humans aren't deterministic

Thus why we replaced computers (flesh and blood people writing out calculations) with computers (silicon-based number-crunching machines).

Re: Automating AI Away

#35

Post by datakan »

sdesol wrote: post #17
datakan wrote: post #11

Earlier quoted context omitted.

Thats the best description I have heard of the problem so far. I ran into this recently where I automated a ton of stuff and got essentially threatened by leadership for not using AI. My system produces the same output 100% of the time, is free, and scales plus is reliable. Doing what they wanted with an LLM was fragile, didn't always produce the same output and was subject to costs. I don't think they could wrap the…

> got essentially threatened by leadership for not using AI. This sounds made up or your workplace is rather odd to say the least. Maybe english isn't your first language and "threatened" is not the correct word?

You sound like someone thats never worked in a corporate environment. No, threatened is the correct word. I don't care if you like that or not.

Re: Automating AI Away

#36

Post by jt2190 »

stego-tech wrote: post #4

Basically what I’ve been saying since OldJob forced LLMs down our throats and pegging performance to usage metrics: why the fuck are we handing deterministic processes to probabilistic systems when it should be the other way around (using probabilistic systems to design deterministic ones)? LLMS should be abstracted out of a process as soon as practicable, replaced with deterministic processes or procedures. Otherwis…

> … deterministic processes…

Just to be clear, software development itself is not deterministic, though? The software developer pushes a given business process from less-deterministic toward more deterministic? When we say we’ve “abstracted LLMs out of a process” we’d also say that we’ve abstracted software developers out that process as well?

Re: Automating AI Away

#38

Post by sdesol »

datakan wrote: post #35
sdesol wrote: post #17

Earlier quoted context omitted.

> got essentially threatened by leadership for not using AI. This sounds made up or your workplace is rather odd to say the least. Maybe english isn't your first language and "threatened" is not the correct word?

You sound like someone thats never worked in a corporate environment. No, threatened is the correct word. I don't care if you like that or not.

It is not about me liking it or not. The word threaten comes with implications. This means they acted in hostile manner based on your words.

Re: Automating AI Away

#39

Post by contextfree »

A dumber but related habit I've gotten into is that if I want to use AI to do some sort of refactoring on a C# codebase, instead of asking it to edit the code directly I ask it to write a code transformation using the Roslyn compiler API, then run that on the code. The result is less likely to have subtle bugs if it appears to work and gets through a light code review on the transformation (i.e., attempts to cheat with weird special-casing are more likely to stand out amongst the Roslyn API code, and if there isn't such weird special-casing but the code is wrong, the result is more likely to be completely broken rather than subtly broken)

Re: Automating AI Away

#40

Post by Terr_ »

derdi wrote: post #7
hadi121 wrote: post #5

Earlier quoted context omitted.

I love the way you put this. Are there any sites or forums or places where people discuss/hash this out? I've genuinely never considered it from this angle before.

Humans aren't deterministic. Determinism is a red herring. There are lots of other problems with agentic programming, but this is not at the top of the list.

"Humans don't always sum two integers correctly. Getting the correct sum is a red herring! There are lots of other problems with my beehive-based calculator [0], but that is not at the top of the list..."

It doesn't matter what we are, what matters is what we want, and whether what we built actually works the way we want it to work.

[0] Discworld's Ponder Stibbons would be rolling in his, grave, or more likely his "Early Death package" pocket-dimension jar.

Post reply on HN