Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Emergent Symbolic Structure of Artificial Neural Networks

arxiv.org

Post reply on HN
Print view View original

101–110 of 117 posts

Re: The Emergent Symbolic Structure of Artificial Neural Networks

#101

Post by antonvs »

jephs wrote: post #71
suddenlybananas wrote: post #34

Earlier quoted context omitted.

These aren't really strawmen, they're more or less than mainstream opinion in the cognitive sciences from the 80s to maybe 2015-2020 or so.

Paul Smolensky is a cognitive science titan from that era. He worked with Hinton, Rumelhart, and McClelland on parallel distributed processing, and literally wrote the book on tensor product representations in cognition, with Geraldine Legendre: https://mitpress.mit.edu/9780262516198/the-harmonic-mind-vol... He's the axis of this particular group of researchers, being the most senior at the place where they all met,…

If it was supposed to be sarcasm, that does help.

Re: The Emergent Symbolic Structure of Artificial Neural Networks

#102

Post by riceflippa »

drdeca wrote: post #98
riceflippa wrote: post #66

Earlier quoted context omitted.

it is not. in this case these researchers have lost their way, with how symbolics entered the conversation. to remind, it is via wanting to prove that ai vs traditional program is understanding deeper. well guess what, ai is not understanding your input better, its your mind playing tricks with language. For analogy, digital world is not real in a physical sense. if output is food, and input is ingredients, then symb…

> while ai is micro level spice & heat that doesn't especially conscious to the ingredients, just that chemistry appears magical. Huh? Do you mean “isn’t especially conscious of the ingredients”? Though, that interpretation seems confusing, because it seems to be meant to be contrasting with “macro slicing dicing stacking them” which doesn’t sound more attentive to what the ingredients are than “micro level spice and…

yes, you are right there is no contrast, because both ai(vector nn) and traditional symbolic programs are not aware of inputs/understanding deeper/"excel in domain" outside our interpretation of it.

In the way digital is the symbolic representation of physical, ai programs may have internal symbolic representation. This paper shows that nn vectors can be closely approx with symbolic structures.

Re: The Emergent Symbolic Structure of Artificial Neural Networks

#103

Post by sublinear »

khalic wrote: post #48
sublinear wrote: post #26

Earlier quoted context omitted.

Doesn't that say more about the massive crumb tray nobody ever bothered to empty at the bottom of mathematics? I'm sure someone will point out something like the 4-color theorem as a counterargument. Where is that kind of theorem proving in this generation of AI? We seem to have hit a dead end rather quickly.

Lol, are you saying the Erdos problems are a "crumb" that nobody bothered to empty? Are you doing a comedy routine?

Explain to the rest of us how far along those problems were before AI.

Re: The Emergent Symbolic Structure of Artificial Neural Networks

#104

Post by sublinear »

dboreham wrote: post #51
sublinear wrote: post #17

Earlier quoted context omitted.

Too shallow of a dismissal, and you don't determine what everyone else takes seriously. It's been several years now of LLMs only appeasing those with low expectations and inexperience. Unless the only goal was generating boilerplate or really sloppy proofs of concept, LLMs are a waste time for everyone else. This argument is so over already. We're all just hoping for a soft landing when the hangover really kicks in.

I have 40+ years experience and extremely high standards. What you say is entirely wrong.

Why is it always the ones with the highest karma giving the most hollow defense? How's retirement treating you?

Re: The Emergent Symbolic Structure of Artificial Neural Networks

#105

Post by drdeca »

taneq wrote: post #54
philipswood wrote: post #47

Earlier quoted context omitted.

I think it would be fair to say that at least a few people can do a small amount of 4d visualisation. Agreed, I don't think anyone has a good grasp on, say, 64K D, but we can do a bit better than just 3.

I’ve heard people claiming that they can but I’ve never heard compelling evidence that they’re directly visualising 4D objects rather than 3D projections of them, or some symbolic representation (arrays of numbers etc.)

All I’ve been able to do is mentally tag parts of a shape as being at different “depths” (possibly with two parts at the same 3D location but tagged as having different “depths”). Which, yeah, I agree isn’t really visualizing a 4D shape. I think someone better at it than me might be able to use this to leverage one’s visual spatial intuition in a way that is a bit closer to what 4D visualization might be like if it were possible.

But I don’t think it is possible for us. At least, for strict senses of visualization. I suspect that the architecture of our brains doesn’t support it. This is rather speculative because I don’t know basically any neuroscience, but I wonder if the whole “the brain is largely a very wrinkled surface (though with several layers to this surface), and the retinas are also surfaces” may have something to do with this. A retina in a eye-4-ball would have a 3D boundary (I.e. hypersurface), and I imagine that the amount of information that it would take in if it had an at all similar angular resolution, would be too great for our visual cortex to be able to represent all of it.

Of course, our vision only really has the high resolution it seems to everywhere in the narrow region we’re directly looking at, but still.

Re: The Emergent Symbolic Structure of Artificial Neural Networks

#106

Post by xg15 »

chrisjj wrote: post #70
squidbeak wrote: post #68

Reading stuff like this (as a layman), diminishing these things as 'Next token predictors' seems absurdly reductive. At some point we'll need to concede that 'selection' is a better term for this than prediction.

> diminishing these things as 'Next token predictors' seems absurdly reductive. This shows a deep misunderstanding of the paper's claims, which in no way challenge the established view that these bots are next-token predictors. Regardless, if all you want is a next-token selector , save your money and roll a die.

> which in no way challenge the established view that these bots are next-token predictors.

I mean, of course they are, that's literally what the inference loop does. You can look at the source of your favorite model runner and you'll see exactly that.

What I find misleading about this term is that it focuses attention on the "next token" part and glosses over the "prediction" part as some sort of unspecified "statistical algorithm" - even though this is where most of the work happens and where the interesting questions are.

Re: The Emergent Symbolic Structure of Artificial Neural Networks

#107

Post by khalic »

sublinear wrote: post #103
khalic wrote: post #48

Earlier quoted context omitted.

Lol, are you saying the Erdos problems are a "crumb" that nobody bothered to empty? Are you doing a comedy routine?

Explain to the rest of us how far along those problems were before AI.

Are they crumbs? Why aren’t you answering?

Re: The Emergent Symbolic Structure of Artificial Neural Networks

#108

Post by andytratt »

subsistence234 wrote: post #41
andytratt wrote: post #19

this is an obvious result. for example, this guy has been writing on substack about this for at least a year or two (with code snippets) explaining the phenomenon of grokking and the ghostbasin.com concept - https://richardaragon.substack.com/ their algorithm is even named "DISCOVER" so they set out to discover the connective tissue of why the universe has invariants like math, and lo it was discovered. i guess good…

Post the actual articles that you have in mind. What I've seen is vague slop. A good example is https://richardaragon.substack.com/p/a-universal-prime-funct... describing a supposed "universal prime function" which is simply a finite approximation using a sum of 50 sines (each applied to a linear term plus a sine-log offset). The 53 parameters are fitted to the first 10^3 or so prime numbers. This is followed by the…

thanks for engaging but tbh just post your own articles. i actually talked to Richard after reading quite in depth and reproducing his code results myself, and do think he's brilliant. my point is he gets the punch line of these math academia papers, much faster.

that's all. easy to call things "vague slop", hard to be right first. Jensen's def'n of intelligence "can you see around corners and predict the future?" kinda all that matters at this point.

just an fyi. congrats on your nitpicking of what math counts as math lol

here's my own paper: https://computerfuture.substack.com/p/demoting-laplaces-demo... (pdf - https://drive.google.com/file/d/1s4VER7SNjUFuCBodX_0myXRpNCW... ) and here's my MIT degree on the blockchain somewhere: https://trattner.github.io/img/18c-diploma.png

feedback welcome from nits... you made zero comments on ghostbasin.com which makes some extraordinary claims too.

Re: The Emergent Symbolic Structure of Artificial Neural Networks

#109

Post by andytratt »

andytratt wrote: post #108
subsistence234 wrote: post #41

Earlier quoted context omitted.

Post the actual articles that you have in mind. What I've seen is vague slop. A good example is https://richardaragon.substack.com/p/a-universal-prime-funct... describing a supposed "universal prime function" which is simply a finite approximation using a sum of 50 sines (each applied to a linear term plus a sine-log offset). The 53 parameters are fitted to the first 10^3 or so prime numbers. This is followed by the…

thanks for engaging but tbh just post your own articles. i actually talked to Richard after reading quite in depth and reproducing his code results myself, and do think he's brilliant. my point is he gets the punch line of these math academia papers, much faster. that's all. easy to call things "vague slop", hard to be right first. Jensen's def'n of intelligence "can you see around corners and predict the future?" ki…

lol you also have 107 karma on hackernews while i have 200+ so how's that for math? lol

Re: The Emergent Symbolic Structure of Artificial Neural Networks

#110

Post by andytratt »

andytratt wrote: post #31
antonvs wrote: post #28

Earlier quoted context omitted.

I hate that whole intro - the first four sentences - so much. It’s nothing but unsupported assumptions. Basically, a strawman that they can do battle with in the paper. Not an auspicious start.

got to top 2 HN tho lol

-1 karma jesus guess i should never make meta commentary lol
Post reply on HN