Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Coding agents think ahead of time

arxiv.org

Post reply on HN
Print view View original

41–50 of 82 posts

Re: Coding agents think ahead of time

#41

Post by chrisjj »

astro1234 wrote: post #15
chrisjj wrote: post #5

> A coding agent solving a software-engineering task spends dozens of steps reasoning No. That's simple PR hype. Parrotry is not reasoning.

Why not? I think there’s fairly strong evidence that there is something that convincingly looks like reasoning. I think anthropic has done some nice circuit tracing and mechanistic interpretability work on this for instance.

> there is something that convincingly looks like reasoning

Convincing ... people who believe chatbots are intelligent? Well sure.

Re: Coding agents think ahead of time

#42

Post by pixl97 »

jdw64 wrote: post #18

In other words, since the next semantic prediction for forecasting the future is built on the training dataset, it's hard for anything truly new to emerge. Then how do humans create something 'creative'—something that didn't exist before? I think it might be because the process of simplifying the complex system of nature differs between individuals. The data being learned now is all labeled by humans and simplified t…

We need to ask deeper questions on what human creativity actually is.

Why didn't humans 10,000 years ago make a car or a spaceship? We had the same minds back then from what we can tell biologically. Why in the 1500's did the precursors of these ideas start to come about? Data is needed, along with some method of analogy. Quite often when big breakthroughs happen there has been a massive amount of information gathered over the years. This is why said breakthroughs are not generally random. They are by people with the time, wealth, and information ability to put the pieces together.

>The data being learned now is all labeled by humans and simplified through human cognition.

Eh, that was a 'few years ago' thinking at this point. AI learning is working with a large amount of self gathered/generated training data now. At the same time almost everything you gather information wise is based on the interpretation of the society you live in. Reality tunnel is the term for this. Entire societies, millions of people, can be blind to something you see as obvious. Humans are not standalone machines, throw us in the woods as babies and you don't get a person that sees the world differently, you get a feral child that may never be capable of higher learning.

In this sense AI may be hobbled for some time. There are very few large models and they have a lot of the same biases, it's like a world that only 10 people live in for AI. Maybe over time training models will get far cheaper and then we'll be able to explore the frontier of having models 'do crazy shit for the fun of it' kind of like humans do quite often.

Re: Coding agents think ahead of time

#43

Post by Planktonne »

astro1234 wrote: post #15
chrisjj wrote: post #5

> A coding agent solving a software-engineering task spends dozens of steps reasoning No. That's simple PR hype. Parrotry is not reasoning.

Why not? I think there’s fairly strong evidence that there is something that convincingly looks like reasoning. I think anthropic has done some nice circuit tracing and mechanistic interpretability work on this for instance.

> I think there’s fairly strong evidence that there is something that convincingly looks like reasoning.

No one is disputing that, but there is an enormous difference between evidence of 'something that looks like a thing' and evidence of the thing itself.

Re: Coding agents think ahead of time

#44

Post by astro1234 »

Planktonne wrote: post #43
astro1234 wrote: post #15

Earlier quoted context omitted.

Why not? I think there’s fairly strong evidence that there is something that convincingly looks like reasoning. I think anthropic has done some nice circuit tracing and mechanistic interpretability work on this for instance.

> I think there’s fairly strong evidence that there is something that convincingly looks like reasoning. No one is disputing that, but there is an enormous difference between evidence of 'something that looks like a thing' and evidence of the thing itself.

I tend to agree but to adjudicate that someone has to define what they consider reasoning to be.

Re: Coding agents think ahead of time

#45

Post by chrisjj »

coldtea wrote: post #39
chrisjj wrote: post #5

> A coding agent solving a software-engineering task spends dozens of steps reasoning No. That's simple PR hype. Parrotry is not reasoning.

You keep repeating this throughout the thread, one could even say just like a certain class of exotic birds :) Why couldn't parroting, after a certain level and complexity of the parroting infrastructure, be reasoning? What a priori restriction forbids it from being such? Flipping a NAND is not calculation either. Billions of them? Things change.

> Flipping a NAND is not calculation either.

But it is.

Re: Coding agents think ahead of time

#46

Post by astro1234 »

chrisjj wrote: post #41
astro1234 wrote: post #15

Earlier quoted context omitted.

Why not? I think there’s fairly strong evidence that there is something that convincingly looks like reasoning. I think anthropic has done some nice circuit tracing and mechanistic interpretability work on this for instance.

> there is something that convincingly looks like reasoning Convincing ... people who believe chatbots are intelligent? Well sure.

I don’t mean to say you are wrong, I just mean to ask what criteria you would consider necessary to satisfy in order for you to consider a system to be reasoning.

Re: Coding agents think ahead of time

#47

Post by sigbottle »

energy123 wrote: post #2

Confirmatory of Sutskever's view that predicting the next token forces a deep understanding. To effectively predict the next token it needs a good idea of what comes after the next token.

If you want to take it that far, we've had results like this since the 60s (solomonoff induction). But of course if you state it like that, your (rightful) objection is that it's pretty vacuous (if I had the computational omniscience to just brute force all possible turing machines, whatever that even means, then sure, any 'f' gets subsumed into this paradigm).

A lot of philosophers, mathematicians, scientists, etc. effectively say, "Yeah, everything's just f(inputs of world) -> outputs!" That 'f' is doing a lot of heavy lifting. Which is kind of the point of mechanistic interperability - to make sure we're not jumping ahead of ourselves, and to make sure we're careful when we claim what "deep" and "structure" means, when it pertains to that 'f'.

Re: Coding agents think ahead of time

#48

Post by HarHarVeryFunny »

chrisjj wrote: post #5

> A coding agent solving a software-engineering task spends dozens of steps reasoning No. That's simple PR hype. Parrotry is not reasoning.

Copying someone else's reasoning process (as best you understand it) is still a limited form of reasoning, even if it might be considered as cargo-cult reasoning (copying without understanding).

Re: Coding agents think ahead of time

#49

Post by rlt »

chrisjj wrote: post #6
energy123 wrote: post #2

Confirmatory of Sutskever's view that predicting the next token forces a deep understanding. To effectively predict the next token it needs a good idea of what comes after the next token.

> To effectively predict the next token it needs a good idea of what comes after the next token. And that's all it needs. Not reasoning.

Isn't "reasoning" in LLMs just training it to have an internal monologue to think through problems like a human would? i.e. extra tokens.

Re: Coding agents think ahead of time

#50

Post by Planktonne »

astro1234 wrote: post #44
Planktonne wrote: post #43

Earlier quoted context omitted.

> I think there’s fairly strong evidence that there is something that convincingly looks like reasoning. No one is disputing that, but there is an enormous difference between evidence of 'something that looks like a thing' and evidence of the thing itself.

I tend to agree but to adjudicate that someone has to define what they consider reasoning to be.

Crucially it's the responsibility of those who are making the claim here to bring evidence and definitions. The other approach, very popular on HN, has led to endless carping by people making outlandish claims without evidence and then demanding that every term possible be defined before they can be dismissed out of hand.

The claim that any two things are equivalent just because they look similar is a strong one, and the onus is on the person advancing it to bring evidence, not to insist that they are considered correct until it is disproved. That's how it works with the flat earth and religion and everything else; we shouldn't accept an unsupported conclusion just because some people really, really like AI and don't want to have to justify their claims.

The traditional response to such claims is ridicule [1]; we know trivially from all sorts of examples that presentation is not identity, and so 'but it looks similar' is not a convincing stance.

1 https://sites.psu.edu/sierraastle/2019/10/21/behold-a-man/

Post reply on HN