Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Emergent Symbolic Structure of Artificial Neural Networks

arxiv.org

Post reply on HN
Print view View original

21–30 of 114 posts

Re: The Emergent Symbolic Structure of Artificial Neural Networks

#21

Post by gfody »

hliyan wrote: post #16
profsummergig wrote: post #8

The human mind cannot comprehend the capacity of massively multidimensional space. Just going from 2D to 3D creates massive new positional potential (e.g. surface of the earth, vs. the atmosphere above earth...). Now imagine 1,000 dimensions.

This is nonsense. The human mind cannot visualize more than 3 dimensions. It can perfectly comprehend any number of dimensions as long as they are represented in a vector space. In fact, that's what linear algebra does.

I think the human mind can visualize anything, eg here are some visual descriptions of higher dimensional objects: https://gfody.substack.com/p/mini-hyper-golf

Re: The Emergent Symbolic Structure of Artificial Neural Networks

#22

Post by andytratt »

colordrops wrote: post #15

"Vectors seem inadequate for capturing the structure of language, logic, and other cognitive domains, yet neural networks achieve impressive performance in these areas". Missing the forest for the trees? Aren't neural networks modeled after biological systems? Our brains are obviously able to contain symbolic structure despite not having a "symbol processing unit".

yep

Re: The Emergent Symbolic Structure of Artificial Neural Networks

#23

Post by andytratt »

4b11b4 wrote: post #6

Sounds reasonable... That the model is sometimes learning a lossy vector representation of something symbolic in nature... Sure, a NN can approximate a function? They say this holds in... Some examples they found? I don't enough about this area

yep

Re: The Emergent Symbolic Structure of Artificial Neural Networks

#24

Post by andytratt »

monster_truck wrote: post #4
0xdeadbeefbabe wrote: post #3

It's like Neo says "You get used to it, though. Your brain does the translating. I don't even see the code." He was referring to something like a K, Q, V vector at the time I believe.

Cypher says that, and he's clearly referring to a blonde, a brunette, and a redhead.

lol

Re: The Emergent Symbolic Structure of Artificial Neural Networks

#25

Post by andytratt »

jkingsman wrote: post #2

The math and core experimentation here is beyond my abilities, but what I think I understand is that there are possible deeper patterns of representation that exist in LLMs that are distillations of core conceptual relations in grammar that we can get our heads around in a mathematical sense rather than apparent layer-smeared noise that somehow, un-interpretably (in a meaningful sense), resolve to correct grammar/inf…

distillation is now illegal tho

Re: The Emergent Symbolic Structure of Artificial Neural Networks

#26

Post by sublinear »

pineaux wrote: post #18
sublinear wrote: post #17

Earlier quoted context omitted.

Too shallow of a dismissal, and you don't determine what everyone else takes seriously. It's been several years now of LLMs only appeasing those with low expectations and inexperience. Unless the only goal was generating boilerplate or really sloppy proofs of concept, LLMs are a waste time for everyone else. This argument is so over already. We're all just hoping for a soft landing when the hangover really kicks in.

What about the mathematical advancements?

Doesn't that say more about the massive crumb tray nobody ever bothered to empty at the bottom of mathematics?

I'm sure someone will point out something like the 4-color theorem as a counterargument. Where is that kind of theorem proving in this generation of AI? We seem to have hit a dead end rather quickly.

Re: The Emergent Symbolic Structure of Artificial Neural Networks

#27

Post by NitpickLawyer »

hliyan wrote: post #16
profsummergig wrote: post #8

The human mind cannot comprehend the capacity of massively multidimensional space. Just going from 2D to 3D creates massive new positional potential (e.g. surface of the earth, vs. the atmosphere above earth...). Now imagine 1,000 dimensions.

This is nonsense. The human mind cannot visualize more than 3 dimensions. It can perfectly comprehend any number of dimensions as long as they are represented in a vector space. In fact, that's what linear algebra does.

I think you accidentally a word, there. GP is talking about comprehending the capacity of massively multi-dimensional space.

Re: The Emergent Symbolic Structure of Artificial Neural Networks

#28

Post by antonvs »

colordrops wrote: post #15

"Vectors seem inadequate for capturing the structure of language, logic, and other cognitive domains, yet neural networks achieve impressive performance in these areas". Missing the forest for the trees? Aren't neural networks modeled after biological systems? Our brains are obviously able to contain symbolic structure despite not having a "symbol processing unit".

I hate that whole intro - the first four sentences - so much. It’s nothing but unsupported assumptions. Basically, a strawman that they can do battle with in the paper. Not an auspicious start.

Re: The Emergent Symbolic Structure of Artificial Neural Networks

#29

Post by larodi »

profsummergig wrote: post #8

The human mind cannot comprehend the capacity of massively multidimensional space. Just going from 2D to 3D creates massive new positional potential (e.g. surface of the earth, vs. the atmosphere above earth...). Now imagine 1,000 dimensions.

Imagine a box of balls. They have size, weight, colour, density… etc. These properties, each a measure, are dimensions and they are orthogonal to each other. Taken together are multi-dimensional.

Re: The Emergent Symbolic Structure of Artificial Neural Networks

#30

Post by qsera »

larodi wrote: post #29
profsummergig wrote: post #8

The human mind cannot comprehend the capacity of massively multidimensional space. Just going from 2D to 3D creates massive new positional potential (e.g. surface of the earth, vs. the atmosphere above earth...). Now imagine 1,000 dimensions.

Imagine a box of balls. They have size, weight, colour, density… etc. These properties, each a measure, are dimensions and they are orthogonal to each other. Taken together are multi-dimensional.

Now take a set of words. They have "sizeness", "weightness", "colorness" and "densityness"...and "pythonness" and "haskellness" and even "adjectiveness" and "verbness" and so on and so forth...You can readily see that this can encode arbitrary patters (like language grammer or program syntax)

Training identifies these dimensions in the training data and links it with each word/token. Then given a stream of such tokens, each with its own set of dimensions (which can be huge), and LLM predicts the dimensions that the next token is most likely to have...

Post reply on HN