Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Large Language Models Are Neurosymbolic Reasoners

arxiv.org

Post reply on HN
Print view View original

141–150 of 172 posts

Re: Large Language Models Are Neurosymbolic Reasoners

#141

Post by bubblyworld »

kaesve wrote: post #96
bubblyworld wrote: post #6

The authors get LLMs to perform pretty well in a variety of IF-style text based games. Which is pretty cool, these kinds of games are played and read in natural language, which makes them pretty hard to write AIs for normally. Something I'd love to see one day is modern AI applied to other kinds of text based games like nethack. Last I checked nobody had managed to solve the problem of nethack AI without using hard c…

There's https://arxiv.org/abs/2310.00166 , which uses an LLM for intrinsic rewards for a RL agent. They use it on nethack. It was discussed on the TalkRL podcast: https://www.talkrl.com/episodes/pierluca-doro-and-martin-kli...

Ooh, really cool! Thanks for the links.

Re: Large Language Models Are Neurosymbolic Reasoners

#142

Post by bubblyworld »

kybernetikos wrote: post #85
bubblyworld wrote: post #6

The authors get LLMs to perform pretty well in a variety of IF-style text based games. Which is pretty cool, these kinds of games are played and read in natural language, which makes them pretty hard to write AIs for normally. Something I'd love to see one day is modern AI applied to other kinds of text based games like nethack. Last I checked nobody had managed to solve the problem of nethack AI without using hard c…

One of the things that llms are mainly not great at is understanding spatial relationship characters to each other. Get chat gpt to convert between different forms of chess notation and you'll start seeing errors very quickly. Ascii art is generally fairly bad unless it's regurgitation. It's not too surprising, but it will make games like nethack harder.

Yeah, I've struggled to get GPT to do ASCII art. I wonder if it's a tokenization thing? If we were to group characters in ASCII art in twos or threes we'd have a hard time making it too!

Re: Large Language Models Are Neurosymbolic Reasoners

#143

Post by calf »

mjburgess wrote: post #136
calf wrote: post #131

Earlier quoted context omitted.

So I'm trying to understand your argument here, but why isn't "Reasoning is a cognitive process" circular logic? AlphaGo wasn't reasoning, how? I think reasonably AlphaGo has modeled a world, and it is by design subject to the bounded rationality of a game-theoretic optimization problem. So two of your criteria are satisfied. So - I'm just reading your definition here - AlphaGo wasn't reasoning because it is not a co…

> Reasoning is a cognitive process You could have a non-cognitive view of reasoning, or an embodied one. NNs are a-cognitive systems, they do not engage in reasoning of any form. Reasoning concerns inference across truth-apt propositions (eg., A->B, A thef. B). NNs have no propositions, nor are any parts truth-apt, true or false. NNs are statistical systems which select answers by weights found from optimisation. No…

To focus on one issue, the neural machine that is chosen by optimization is one that "best" fits the photos of the sky. But those multiple optima do not preclude a neural machine whose parameter values are computationally equivalent to, say, a 3D representation of the sky projected onto a 2D perspective -- a kind of partial world theory or world model, that was picked randomly out of many optima. First, it's not impossible, just highly difficult to find at present technology. Second, the papers describing emergent structures or emergent information inside of actually-existing neural nets point to an empirical possibility that these machines are more than their statistical parts. Both these reasons incline me to stay on the fence on whether neural nets are purely stochastic parrots.

Re: Large Language Models Are Neurosymbolic Reasoners

#144

Post by still_grokking »

neurallambda wrote: post #128
still_grokking wrote: post #77

Earlier quoted context omitted.

May I ask why there are NNs in your project at all? Just to heat up the planet and make Nvidia share holders even more happy? :-) I mean, what I've seen in the Readme makes sense. But doing basic computer stuff with NNs just makes the resource usage go brr by astronomical factors, for imho no reason, while making the results brittle, random, and often just completely made up. Also: Do you know about the (already 40 y…

"AI heats the planet"... really? You mean marginally? I'll assume you're asking in good faith. Using NNs allows this project to stand on the shoulders of giants: philosophically, mathematically, programmatically, but also I expect this to plug in to OSS LLMs, and leverage their knowledge, similarly to how a human child learns in a Pavlovian/intuitive response, and only later starts to learn to reason. Wrt inefficienc…

> Wrt inefficiency, training will be inefficient

That's the "heating the planet part" I was referring to. :-)

> but the programs can be extracted to CPU instructions / CUDA kernels during inference

This just makes my original question more pressing: What are the NNs good for if the result will be normal computer programs? (Just created with astronomical overhead!)

> Cyc looks cool, but from my cursory glance, is it capable of learning, or is its knowledge graph largely hand coded?

The whole point is that it can infer new knowledge from known facts through a logical reasoning process.

This inference process was run since 40 years. The result is the most comprehensible "world knowledge" archive ever created. Of course this wouldn't be possible to create "by hand". And in contrast to NN hallucinations there is real logical reasoning behind, and everything is explainable.

I still don't get how some "dreamed up" programs from your project are supposed to work. Formal reasoning and NNs don't go well with each other. (One could even say they're opposites). Imho it's "real reasoning" OR "dreamed up stuff". How "dreamed up stuff" could improve "real reasoning"? Especially as the "dreamed up stuff" won't be included in the end results anyway, where only the formal things remain. To what effect are the NNs included in your project? (I mean besides the effect that the HW and energy demands will go through the roof, ending up billion times higher than just doing some lambda calculus directly…)

And yes, these are genuine questions. I just don't get it. It looks for me like "let's do things maximally inefficiently, but at least we can put a 'works with AI' rubber stamp on it"; which is maybe good to collect VC money, but else?

What do I overlook here?

Re: Large Language Models Are Neurosymbolic Reasoners

#145

Post by still_grokking »

Legend2440 wrote: post #82
still_grokking wrote: post #60

Earlier quoted context omitted.

Every computable function can be represented by a (possibly infinite)¹ lookup table. Computer programs can only compute computable functions. Therefore any computer program is (in theory) equivalent to a table lookup . ¹ For finite inputs, the lookup table can be finite, and for infinite inputs, the lookup table can be infinite but still countable, as the set of computable functions is countable.

This table is not computable. If you had this table, you could solve the halting problem by simply looking up whether the program produced an output.

You've just restated the halting problem.

Nobody claimed that there is an algorithm to translate arbitrary programs into an equivalent lookup table. (Because that's the exact same proposition as stating that there is a program that can compute whether an arbitrary program halts when executed).

The point is: Any specific program can be translated into a lookup table. Computer programs and lookup tables are equivalent!

You claimed that computer programs are somehow "more powerful" than lookup tables. That's just plain wrong. They're exactly equivalent in "power".

Re: Large Language Models Are Neurosymbolic Reasoners

#146

Post by still_grokking »

bubblyworld wrote: post #89
still_grokking wrote: post #81

Earlier quoted context omitted.

I think my wording was bad. There are two kinds of "randomness" here at play and I didn't differentiate properly. For me the game mechanics as such are "random". Because you can't discover them by just playing (imho). At the same time the game is ruled by a dice. (So even the best players will fail almost 50% of the time which is almost as random as tossing a coin, and strictly not skill based). > Given that AIs can…

Fair enough - the mechanics are certainly random in the sense that they involve dice rolls. > So even the best players will fail almost 50% of the time which is almost as random as tossing a coin, and strictly not skill based Even moderately experienced players will fail close to 100% of the time. So getting to an almost 50% success rate, to my mind, shows a great deal of skill! The difference between this and a coin…

> the mechanics are certainly random in the sense that they involve dice rolls.

Sorry, but that's still not what I've meant.

Dice rolls are randomness in the usual meaning of this word. But "random" can also mean "arbitrary" and/or "illogical" things. Imho Nethack mechanics are also random in this sense. They don't make sense at all… :-)

> Even moderately experienced players will fail close to 100% of the time. So getting to an almost 50% success rate, to my mind, shows a great deal of skill!

OK, you have definitely a point here.

Still not my cup of tea, such games. (And this is strictly personal, and unrelated to the rest of the discussion). I just don't like games where it's very likely that I will loose despite "doing everything right". I have no problem with games punishing merciless even small mistakes. That's OK. But having an outcome that depends mostly on the whim of the RNG is just nothing I enjoy. When I "do everything right" I like to get rewarded appropriately for it. (Of course some level of randomness is still OK. But if the RNG kills you most of the time no mater what you do this is just too frustrating for me. So I'm clearly not the target audience for Nethack… :-D).

> I don't think this is true any more for modern AIs such as AlphaGo and its predecessors, which learn distributions of possible outcomes rather than deterministic predictions.

Sure. But how do you learn from a distribution where no matter what you do you will fail in, say, 99,9% of the cases?

> IIRC the latest versions can even self-play games like Poker to a superhuman level.

Do you have some links regarding this? I thought Poker is still one of the games where AIs don't play better than humans. OK, maybe it depends on the Poker variant. There are simpler and more difficult ones.

> If there is any systematic deviation from pure randomness, enough data will reveal it!

I would agree in general.

But now we're back to the initial question: Is there enough systematic deviation from pure randomness in Nethack? Given that even people who know all the mechanics, and know some good end-to-end strategies will fail in most cases (actually, like you said, in almost all cases). And given that it's (imho) impossible to come up with this knowledge about mechanics and strategy just by playing this game. I have my doubts.

Re: Large Language Models Are Neurosymbolic Reasoners

#147

Post by still_grokking »

klyrs wrote: post #125
still_grokking wrote: post #54

Earlier quoted context omitted.

Is it actually possible to beat Nethack without reading up some "spoilers" upfront? I've never heard of anybody who managed to do that. Even when you read up all kinds of info about the game before you attempt a run it's extremely hard to reach higher levels, yet beat the game. (I myself never reached any later levels despite I know some tricks by now. Tricks impossible to infer from just playing the game; you need t…

> Is it actually possible to beat Nethack without reading up some "spoilers" upfront? If people are playing nethack without reading the source, they're needlessly hobbling themselves.

That's my point. Imho it's impossible to even come close to a realistic chance to beat this game without knowing more or less every detail of its internal mechanics. And you can't infer this knowledge just from playing. You need to look it up from some out-of-band source.

So only letting an AI play this game will not reveal anything about the "reasoning" capabilities of said AI.

Letting the AI read the source (or some other "spoilers") and than look at how it performs playing would be maybe a usable test. But that's still a big "Maybe" imho as Nethack is just "too random" I think.

Re: Large Language Models Are Neurosymbolic Reasoners

#148

Post by mjburgess »

calf wrote: post #140
bubblyworld wrote: post #119

Earlier quoted context omitted.

Wild, well, I'm a lowly math PhD so that's where my interests lie =) I'm _not_ suggesting we replace R with Q. I'm suggesting that you bake in the desired accuracy of your computational approximation as an input . This is how Turing evades self-referential problems in his conception of computational reals, and also perhaps how you evade your criticisms with CM requiring infinite precision. Similarly - I think it's re…

I've read this thread exchange with interest, but what about the results that quantum computers are simulatable by classical computers? See David Deutsch 1985. This would reduce the issue of infinite Hilbert spaces to simulation using quantum computers, and in turn, Deutsch's result which says classical Turing machines can actually simulate quantum computers.

You can always make local arguments that, say, some g can be substituted with some c.

The issue is broader than that. It concerns the premises of vast areas of physics -- you have to show they are more likely false than true.

This isnt an argument saying no c can be found for any given g, it's saying, "g-c gaps have empirical consequences we havent observed" and if we did, physics would be foundationally wrong

Re: Large Language Models Are Neurosymbolic Reasoners

#149

Post by mjburgess »

FrustratedMonky wrote: post #139
mjburgess wrote: post #136

Earlier quoted context omitted.

> Reasoning is a cognitive process You could have a non-cognitive view of reasoning, or an embodied one. NNs are a-cognitive systems, they do not engage in reasoning of any form. Reasoning concerns inference across truth-apt propositions (eg., A->B, A thef. B). NNs have no propositions, nor are any parts truth-apt, true or false. NNs are statistical systems which select answers by weights found from optimisation. No…

This is kind of ignoring other NN's. You're very focused on LLM's as the example. AlphaGo learned by playing itself. And is able to anticipate multiple moves ahead. Then, that same 'engine', was able to be applied to Chess, and learned how to beat a master from scratch, by playing itself, in just a few hours. There was no lookups, or zip'ing of aggregated data. A lot of what you are postulating as cognition, humans d…

Yes, it is impossible for anyone to figure out gravity by induction. That's the problem with AI.

The way we build explanations is largely by reasoning-by-analogy. We build physical models with our hands, to resolve ambiguities in our environment, ever more complexly -- and then, eventually, land upon the right analogy that then falls away.

Prior to gravity we had crystal spheres -- reasoning by ananlogy with such things.

Since machines arent in the world, as in my robot example, they can never build explantory conceptualisations of it.

Chess et al. are not worlds in any relevant sense. No one cares, or doubts, that a system with a fully mathematically specific "world" can be "learnt" by a computer.

The full specification of a "world" in mathematical terms doesnt require intelligence. At that point you can use the dumb strategies of alphago.

Intelligence is what you do when you don't know what you're doing. The "World Model"s we're interested in are those that arent already specified to the machine.

All these formal games are just outcomes spaces where every event is known a priori.

As I said, the AI people arent even operating in the category of intelligence. It just the profound lack of background knowledge of any field outside pytorch.lol() that their meglomania plausible

Go look at every paper in AI or ML that purports to build a model of anything: can you find a single one where the outcome space cannot be fully specified either formally (as with chess, etc.) or empirically (as with data samples)?

This has nothing to do with intelligence. We do not either start from the answers, or samples of the answers. We have no answers.

The resolution to this problem requires having a body: you have to move in order to think, in direct causal contact with the world beyond the capacities of clay, to think about it.

Re: Large Language Models Are Neurosymbolic Reasoners

#150

Post by eru »

still_grokking wrote: post #81
bubblyworld wrote: post #75

Earlier quoted context omitted.

Relax, friend, I understand your question =) I'm just pointing out that it's not as random as you think. Mechanics in NetHack are quite predictable for the most part, but they are difficult to _discover_ without dying. Given that AIs can play hundreds of thousands of runs in the time it would take me to play one, I'm a bit more optimistic that they could learn the mechanics eventually. I think the fundamental problem…

I think my wording was bad. There are two kinds of "randomness" here at play and I didn't differentiate properly. For me the game mechanics as such are "random". Because you can't discover them by just playing (imho). At the same time the game is ruled by a dice. (So even the best players will fail almost 50% of the time which is almost as random as tossing a coin, and strictly not skill based). > Given that AIs can…

> An "exploration-based reward function" wouldn't be enough. Because this would assume that exploration has (more or less) deterministic outcomes. But given the dice in Nethack it actually does not! You can do "everything right" and still die in almost 50% of the cases. How to infer any meaningful "world model" from such events? Imho you can't.

You can actually do that very well. When they train LLM on text, the same prefix doesn't always lead to the same next token. And they handle that just fine.

Btw, Nethack isn't actually random: the dice use a pretty broken PRNG, and 'luck manipulation' is a thing. A computer might not actually care about the difference between spoiler-y tactics that are legible to humans, and PRNG manipulation.

(The state of the PRNG is a relatively small number of bits. Various actions can advance the PRNG, without causing any other change in the world. So you can basically make sure that you are always maximally lucky, if you can somehow recover the hidden PRNG state from the output of the program, and then model it.)

Post reply on HN