Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling long-running autonomous coding

simonwillison.net

Post reply on HN
Print view View original

51–60 of 112 posts

Re: Scaling long-running autonomous coding

#51

Post by CuriouslyC »

heavyset_go wrote: post #20
ramraj07 wrote: post #16

Earlier quoted context omitted.

The fundamental idea that modern LLMs can only ever remix, even if its technically true (doubt), in my opinion only says to me that all knowledge is only ever a remix, perhaps even mathematically so. Anyone who still keeps implying these are statistical parrots or whatever is just going to regret these decisions in the future.

Yeah, Yann LeCun is just some luddite lol

You don't understand Yann's argument. It's similar to Richard Sutton's, in that these things aren't thinking, they're emulating thinking, and the weak implicit world models that get built in the weights are insufficient for true "AGI."

This is orthogonal to the issue of whether all ideas are essentially "remixes." For the record I agree that they are.

Re: Scaling long-running autonomous coding

#52

Post by encyclopedism »

gjadi wrote: post #25
NitpickLawyer wrote: post #24

Earlier quoted context omitted.

I don't think he's a luddite at all. He's brilliant in what he does, but he can also be wrong in his predictions (as are all humans from time to time). He did have 3 main predictions in ~23-24 that turned out to be wrong in hindsight. Debatable why they were wrong, but yeah. In a stage interview (a bit after the "sparks of agi in gpt4" paper came out) he made 3 statemets: a) llms can't do math. They can trick us with…

Have this shortcomings of llms been addressed by better models or by better integration with other tools? Like, are they better at coding because the models are truly better or because the agentic loops are better designed?

Fundamentally these shortcomings cannot be addressed.

They can and are improved (papered over) over time. For example by improving and tweaking the training data. Adding in new data sets is the usual fix. A prime example 'count the number of R's in Strawberry' caused quite a debacle at a time where LLM's were meant to be intelligent. Because they aren't they can trip up over simple problems like this. Continue to use an army of people to train them and these edge cases may become smaller over time. Fundamentally the LLM tech hasn't changed.

I am not saying that LLM's aren't amazing, they absolutely are. But WHAT they are is an understood thing so lets not confuse ourselves.

Re: Scaling long-running autonomous coding

#53

Post by NiloCK »

Chipshuffle wrote: post #37

The more I think about LLMs the stranger it feels trying to grasp what they are. To me, when I'm working with them, they don't feel intelligence but rather an attempt at mimicking it. You can never trust, that the AI actually did something smart or dump. The judge always has to be you. It's ability to pattern match it's way through a code base is impressive until it's not and you always have to pull it back to realit…

If you find yourself 50-first-dating your LLMs, it may be worth it to invest some energy into building up some better context indexing of both the codebase itself and of your roadmap.

Re: Scaling long-running autonomous coding

#54

Post by clbrmbr »

Gazoche wrote: post #39
Chipshuffle wrote: post #37

The more I think about LLMs the stranger it feels trying to grasp what they are. To me, when I'm working with them, they don't feel intelligence but rather an attempt at mimicking it. You can never trust, that the AI actually did something smart or dump. The judge always has to be you. It's ability to pattern match it's way through a code base is impressive until it's not and you always have to pull it back to realit…

> they don't feel intelligence but rather an attempt at mimicking it Because that's exactly what they are. An LLM is just a big optimization function with the objective "return the most probabilistically plausible sequence of words in a given context". There is no higher thinking. They were literally built as a mimicry of intelligence.

Life is more fun as a scruffie.

[0] http://www.catb.org/~esr/jargon/html/N/neats-vs--scruffies.h...

Re: Scaling long-running autonomous coding

#55

Post by myrmidon »

encyclopedism wrote: post #49
Gazoche wrote: post #39

Earlier quoted context omitted.

> they don't feel intelligence but rather an attempt at mimicking it Because that's exactly what they are. An LLM is just a big optimization function with the objective "return the most probabilistically plausible sequence of words in a given context". There is no higher thinking. They were literally built as a mimicry of intelligence.

I don't understand why this point is NOT getting across to so many on HN. LLM's do not think, understand, reason, reflect, comprehend and they never shall. I have commented elsewhere but this bears repeating If you had enough paper and ink and the patience to go through it, you could take all the training data and manually step through and train the same model. Then once you have trained the model you could use even…

> If you had enough paper and ink and the patience to go through it, you could take all the training data and manually step through and train the same model.

But you could make the exact same argument for a human mind? (could just simulate all those neural interactions with pen and paper)

The only way to get out of it is to basically admit magic (or some other metaphysical construct with a different name).

Re: Scaling long-running autonomous coding

#56

Post by Zababa »

encyclopedism wrote: post #49
Gazoche wrote: post #39

Earlier quoted context omitted.

> they don't feel intelligence but rather an attempt at mimicking it Because that's exactly what they are. An LLM is just a big optimization function with the objective "return the most probabilistically plausible sequence of words in a given context". There is no higher thinking. They were literally built as a mimicry of intelligence.

I don't understand why this point is NOT getting across to so many on HN. LLM's do not think, understand, reason, reflect, comprehend and they never shall. I have commented elsewhere but this bears repeating If you had enough paper and ink and the patience to go through it, you could take all the training data and manually step through and train the same model. Then once you have trained the model you could use even…

Can you give examples of how that "LLM's do not think, understand, reason, reflect, comprehend and they never shall" or that "completely mechanical process" helps you understand better when LLM works and when they don't?

Many people are throwing around that they don't "think", that they aren't "conscious", that they don't "reason", but I don't see those people sharing interesting heuristics to use LLMs well. The "they don't reason" people tend to, in my opinion/experience, underestimate them by a lot, often claiming that they will never be able to do .

To be fair, the "they reason/are conscious" people tend to, in my opinion/experience, overestimate how much a LLM being able to "act" a certain way in a certain situation says about the LLM/LLMs as a whole ("act" is not a perfect word here, another way of looking at it is that they visit only the coast of a country and conclude that the whole country must be sailors and have a sailing culture).

Re: Scaling long-running autonomous coding

#57

Post by cess11 »

Chipshuffle wrote: post #37

The more I think about LLMs the stranger it feels trying to grasp what they are. To me, when I'm working with them, they don't feel intelligence but rather an attempt at mimicking it. You can never trust, that the AI actually did something smart or dump. The judge always has to be you. It's ability to pattern match it's way through a code base is impressive until it's not and you always have to pull it back to realit…

It's a compressed database with diffuse indices. It's using probability matching rather than pattern matching. Write operations are called 'training' and 'fine-tuning'.

Re: Scaling long-running autonomous coding

#58

Post by encyclopedism »

myrmidon wrote: post #55
encyclopedism wrote: post #49

Earlier quoted context omitted.

I don't understand why this point is NOT getting across to so many on HN. LLM's do not think, understand, reason, reflect, comprehend and they never shall. I have commented elsewhere but this bears repeating If you had enough paper and ink and the patience to go through it, you could take all the training data and manually step through and train the same model. Then once you have trained the model you could use even…

> If you had enough paper and ink and the patience to go through it, you could take all the training data and manually step through and train the same model. But you could make the exact same argument for a human mind? (could just simulate all those neural interactions with pen and paper) The only way to get out of it is to basically admit magic (or some other metaphysical construct with a different name).

> But you could make the exact same argument for a human mind?

It would be an argument and you are free to make it. What the human mind is, is an open scientific and philosophical problem many are working on.

The point is that LLM's are NOT the same because we DO know that LLM's are. Please see the myriad of tutorials 'write an LLM from scratch'

Re: Scaling long-running autonomous coding

#59

Post by encyclopedism »

Zababa wrote: post #56
encyclopedism wrote: post #49

Earlier quoted context omitted.

I don't understand why this point is NOT getting across to so many on HN. LLM's do not think, understand, reason, reflect, comprehend and they never shall. I have commented elsewhere but this bears repeating If you had enough paper and ink and the patience to go through it, you could take all the training data and manually step through and train the same model. Then once you have trained the model you could use even…

Can you give examples of how that "LLM's do not think, understand, reason, reflect, comprehend and they never shall" or that "completely mechanical process" helps you understand better when LLM works and when they don't? Many people are throwing around that they don't "think", that they aren't "conscious", that they don't "reason", but I don't see those people sharing interesting heuristics to use LLMs well. The "the…

We know what an LLM is in fact you can build one from scratch if you like. e.g. https://www.manning.com/books/build-a-large-language-model-f...

It's an algorithm and a completely mechanical process which you can quite literally copy time and time again. Unless of course you think 'physical' computers have magical powers that a pen and paper Turing machine doesn't?

> Many people are throwing around that they don't "think", that they aren't "conscious", that they don't "reason", but I don't see those people sharing interesting heuristics to use LLMs well.

My digital thermometer doesn't think. Imbibing LLM's with thought will start leading to some absurd conclusions.

A cursory read of basic philosophy would help elucidate why casually saying LLM's think, reason etc is not good enough.

What is thinking? What is intelligence? What is consciousness? These questions are difficult to answer. There is NO clear definition. Some things are so hard to define (and people have tried for centuries) e.g. what is consciousness? That they are a problem set within themselves please see Hard problem of consciousness.

https://en.wikipedia.org/wiki/Hard_problem_of_consciousness

Re: Scaling long-running autonomous coding

#60

Post by cess11 »

myrmidon wrote: post #55
encyclopedism wrote: post #49

Earlier quoted context omitted.

I don't understand why this point is NOT getting across to so many on HN. LLM's do not think, understand, reason, reflect, comprehend and they never shall. I have commented elsewhere but this bears repeating If you had enough paper and ink and the patience to go through it, you could take all the training data and manually step through and train the same model. Then once you have trained the model you could use even…

> If you had enough paper and ink and the patience to go through it, you could take all the training data and manually step through and train the same model. But you could make the exact same argument for a human mind? (could just simulate all those neural interactions with pen and paper) The only way to get out of it is to basically admit magic (or some other metaphysical construct with a different name).

I'm not so sure "a human mind" is the kind of newtonian clockwork thingiemabob you "could just simulate" within the same degree of complexity as the thing you're simulating, at least not without some sacrifices.
Post reply on HN