Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Automating AI Away

replicated.live

Post reply on HN
Print view View original

41–50 of 69 posts

Re: Automating AI Away

#41

Post by bob1029 »

I think semi-automation with contextual and domain-specific tooling is the key to the best quality outcomes.

For example, with browser automation, giving the LLM raw access to the literal DOM generally results in disaster for tasks that need to be stable across more than 5-10 interactions. The better approach is to write an intermediate layer that understands each view and can provide a list of tools that are precisely tailored for each case. E.g.:

  https://myapp/Login
  - 
  - Available Tools:
vs

  https://myapp/login
  - We detected that this is the application's login page. 
  - It has the following visible elements:
    + Username
    + Password
    + Login Button
  - Available Tools:
    + PerformLogin
    + Quit
The later case takes a lot more effort, but it also reduces a Turing complete problem space into a binary decision at this particular step.

Re: Automating AI Away

#42

Post by braebo »

bob1029 wrote: post #41

I think semi-automation with contextual and domain-specific tooling is the key to the best quality outcomes. For example, with browser automation, giving the LLM raw access to the literal DOM generally results in disaster for tasks that need to be stable across more than 5-10 interactions. The better approach is to write an intermediate layer that understands each view and can provide a list of tools that are precise…

Yea I’ve been warming up to the web mcp after doing more ai browser work and hitting so many pain points: https://developer.chrome.com/docs/ai/webmcp

Re: Automating AI Away

#43

Post by sensecall »

stego-tech wrote: post #4

Basically what I’ve been saying since OldJob forced LLMs down our throats and pegging performance to usage metrics: why the fuck are we handing deterministic processes to probabilistic systems when it should be the other way around (using probabilistic systems to design deterministic ones)? LLMS should be abstracted out of a process as soon as practicable, replaced with deterministic processes or procedures. Otherwis…

I think a big part is the misperception that it’s “easier” and less effort to run stuff through LLM than to design an effective deterministic process.

Would love to know how you’ve managed to counter this as the drive to throw everything at LLMs is driving me insane.

Re: Automating AI Away

#44

Post by oulipo2 »

auggierose wrote: post #28
Animats wrote: post #24

Earlier quoted context omitted.

For some problems, yes. Formal specification is particularly useful in two cases. 1) The problem is simple but an efficient implementation is hard or bug-prone. Examples are garbage collection, file systems, sorts, databases, and tree updating. 2) The inverse of the problem is simpler than the forward operation. Examples include matrix inversion and parsing.

I wouldn’t split it like that. Formal verification is useful in the case that the spec is simpler than the implementation. That’s it. Coming up with simple specs is not necessarily easy. You could say that is kind of what math is about. That’s how we actually make progress: find those cases where simple specs are possible and build upon them. That’s the kind of library made for eternity.

It could still be useful if the spec is roughly as hard as a simple implementation, in case you have automated methods to find more efficient implementations, guided by the constraints of the spec

Re: Automating AI Away

#45

Post by [deleted] »

Terr_ wrote: post #40
derdi wrote: post #7

Earlier quoted context omitted.

Humans aren't deterministic. Determinism is a red herring. There are lots of other problems with agentic programming, but this is not at the top of the list.

"Humans don't always sum two integers correctly. Getting the correct sum is a red herring! There are lots of other problems with my beehive-based calculator [0], but that is not at the top of the list..." It doesn't matter what we are, what matters is what we want, and whether what we built actually works the way we want it to work. [0] Discworld's Ponder Stibbons would be rolling in his, grave, or more likely his "E…

[deleted]

Re: Automating AI Away

#46

Post by derdi »

Terr_ wrote: post #40
derdi wrote: post #7

Earlier quoted context omitted.

Humans aren't deterministic. Determinism is a red herring. There are lots of other problems with agentic programming, but this is not at the top of the list.

"Humans don't always sum two integers correctly. Getting the correct sum is a red herring! There are lots of other problems with my beehive-based calculator [0], but that is not at the top of the list..." It doesn't matter what we are, what matters is what we want, and whether what we built actually works the way we want it to work. [0] Discworld's Ponder Stibbons would be rolling in his, grave, or more likely his "E…

See my reply to the sibling. Yes, it matters that the outcome is a working system! It doesn't matter whether the system was created by a human pressing keys on a keyboard.

Re: Automating AI Away

#47

Post by orbital-decay »

As it always is with these articles, that has nothing to do with non-determinism the author is talking about. Model's input is in natural language which isn't formally defined, unlike Ragel's input. This makes it open to interpretation by the model that isn't trained the same way as you, has very limited cognitive capabilities, and must generate something in very limited time by design, even if the result is incorrect. This also makes it not related to determinism in any way. You can make model outputs deterministic, but this won't solve your problem because it's not about determinism. Words have meaning.

Claude or any other model just translates your natural language instructions into formally defined tool calls. You cannot replace this layer with a formal tool like Ragel. You can write code for Ragel directly, in which case the responsibility for this is yours and not Claude's. (duh)

>What about Claude? Well, my instructions say in all caps: DO NOT PARSE ANYTHING MANUALLY, EVER. (...) It tries anyway

This needs a self-verification loop. It still won't guarantee that model's interpretation will match yours, but it will improve the accuracy. Almost every model will know that it went off the rails upon checking what it's trying to do. Harness has to provide the loopback for this, because the transformer architecture doesn't.

Re: Automating AI Away

#48

Post by twosdai »

contextfree wrote: post #39

A dumber but related habit I've gotten into is that if I want to use AI to do some sort of refactoring on a C# codebase, instead of asking it to edit the code directly I ask it to write a code transformation using the Roslyn compiler API, then run that on the code. The result is less likely to have subtle bugs if it appears to work and gets through a light code review on the transformation (i.e., attempts to cheat wi…

This sounds interesting, I am really naive. I don't code in C#, is there an analogy for other programming languages, like GO, or Python or Typescript?

Like are you prompting like:

--- I need code that does X,Y, and Z. Write it so that the Roslyn compiler on this machine can compile and the code passes the repo's styling and formatting requirements. ---

Or something else.

Re: Automating AI Away

#49

Post by bob1029 »

twosdai wrote: post #48
contextfree wrote: post #39

A dumber but related habit I've gotten into is that if I want to use AI to do some sort of refactoring on a C# codebase, instead of asking it to edit the code directly I ask it to write a code transformation using the Roslyn compiler API, then run that on the code. The result is less likely to have subtle bugs if it appears to work and gets through a light code review on the transformation (i.e., attempts to cheat wi…

This sounds interesting, I am really naive. I don't code in C#, is there an analogy for other programming languages, like GO, or Python or Typescript? Like are you prompting like: --- I need code that does X,Y, and Z. Write it so that the Roslyn compiler on this machine can compile and the code passes the repo's styling and formatting requirements. --- Or something else.

Here's the official docs if it helps

https://learn.microsoft.com/en-us/dotnet/csharp/roslyn-sdk/t...

Re: Automating AI Away

#50

Post by auggierose »

oulipo2 wrote: post #44
auggierose wrote: post #28

Earlier quoted context omitted.

I wouldn’t split it like that. Formal verification is useful in the case that the spec is simpler than the implementation. That’s it. Coming up with simple specs is not necessarily easy. You could say that is kind of what math is about. That’s how we actually make progress: find those cases where simple specs are possible and build upon them. That’s the kind of library made for eternity.

It could still be useful if the spec is roughly as hard as a simple implementation, in case you have automated methods to find more efficient implementations, guided by the constraints of the spec

Which is still a case of the spec being simpler than the implementation (you are after) ;-)

Very often, the spec is indeed just a very simple implementation. Often you can make the spec especially simple if there are no constraints on the resources it can use, at times even infinite ones.

Post reply on HN