Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What's Going on in Machine Learning? Some Minimal Models

writings.stephenwolfram.com

Post reply on HN
Print view View original

31–40 of 73 posts

Re: What's Going on in Machine Learning? Some Minimal Models

#31

Post by wredue »

dbrueck wrote: post #19

I believe that this is one of the key takeaways for reasoning about LLMs and other seemingly-magical recent developments in AI: "tasks—like writing essays—that we humans could do, but we didn’t think computers could do, are actually in some sense computationally easier than we thought." It hurts one's pride to realize that the specialized thing they do isn't quite as special as was previously thought.

Computers still aren’t writing essays. They are stringing words together using copied data.

If they were writing essays, I would suggest that it wouldn’t be so ridiculously easy to pick out the obviously AI articles everywhere.

Re: What's Going on in Machine Learning? Some Minimal Models

#32

Post by e12e »

SkyBelow wrote: post #25
nyrikki wrote: post #22

Earlier quoted context omitted.

Nit, it predicts that it is the token '4'. Token frequency in pre-training corpus and the way tokenization is implemented impacts arithmetic proficiency for LLMs. OpenAI calls this out in the GPT4 technical report.

You can see this by giving it broken code and seeing what it can predict. I gave copilot a number of implementations of factorial with the input of 5. When it recognized the correct implementations, it was able to combine the ideas of "factorial", "5", and "correct implementation" to output 120. But when I gave it buggy implementations, it could recognize they were wrong, but the concepts of "factorial", "5", and "in…

This makes very little sense (as a contrast to chatgpt predicted that the likely continuation of factorial and 5 is 120).

Perhaps if you are able to share the chat session it's possible to see if you likely confused the issue with various factorial implementations - or got chatgpt to run your code with 5 as input?

I mean the code is redundant:

https://chatgpt.com/share/be249097-5067-4e3d-93c7-3eebedb510...

Re: What's Going on in Machine Learning? Some Minimal Models

#33

Post by aeonik »

This article does a good job laying the foundation of why I think homiconic languages are so important, and doing AI in languages that aren't, are doomed to stagnation in the long term.

The acrobatics that Wolfram can do with the code and his analysis is awesome, and doing the same without the homoiconicity and metaprogramming makes my poor brain shudder.

Do note, Wolfram Language is homoiconic, and I think I remember reading that it supports Fexprs. It has some really neat properties, and it's a real shame that it's not Open Source and more widely used.

Re: What's Going on in Machine Learning? Some Minimal Models

#34

Post by sdenton4 »

wrsh07 wrote: post #4

Because of the computational simplicity, I think there's a possibility that we will discover very cheap machine learning techniques that are discrete like this. I think this is novel (I've seen BNN https://arxiv.org/pdf/1601.06071 This actually makes things continuous for training, but if inference is sufficiently fast and you have an effective mechanism for permutation, training could be faster using that) I am curi…

So, the thing is that linear algebra operations are very cheap already... you just need a lot of them. Any other 'cheap' method is going to have a similar problem: if the unit is small and not terribly expressive, you need a whole lot of them. But it will be compounded by the fact that we don't have decades of investment in making these new atomic operations as fast and cheap as possible.

A good take-away from the Wolfram writeup is that you can do machine learning on any pile of atoms you've got lying around, so you might as well do it on whatever you've got the best tooling for - right now this is silicon doing fixed-point linear algebra operations, by a long shot.

Re: What's Going on in Machine Learning? Some Minimal Models

#35

Post by jderick »

phyalow wrote: post #16
vessenes wrote: post #13

Classic Wolfram — brilliant, reimplements / comes at a current topic using only cellular automata, and draws some fairly deep philosophical conclusions that are pretty intriguing. The part I find most interesting is his proposal that neural networks largely work by “hitching a ride” on fundamental computational complexity, in practice sort of searching around the space of functions representable by an architecture fo…

Agree. (D)NNs have a powerful but somewhat loose inductive bias. They're great at capturing surface-level complexity but often miss the deeper compositional structure. This looseness, in my opinion, stems from a combination of factors: architectures that are not optimally designed for the specific task at hand, limitations in computational resources that prevent us from exploring more complex and expressive models, a…

What is GPR?

Re: What's Going on in Machine Learning? Some Minimal Models

#36

Post by wrsh07 »

sdenton4 wrote: post #34
wrsh07 wrote: post #4

Because of the computational simplicity, I think there's a possibility that we will discover very cheap machine learning techniques that are discrete like this. I think this is novel (I've seen BNN https://arxiv.org/pdf/1601.06071 This actually makes things continuous for training, but if inference is sufficiently fast and you have an effective mechanism for permutation, training could be faster using that) I am curi…

So, the thing is that linear algebra operations are very cheap already... you just need a lot of them. Any other 'cheap' method is going to have a similar problem: if the unit is small and not terribly expressive, you need a whole lot of them. But it will be compounded by the fact that we don't have decades of investment in making these new atomic operations as fast and cheap as possible. A good take-away from the Wo…

Sort of, yes. But if the existing thing were "the cheapest", quantization wouldn't exist.

It depends on what your constraint is! So if you're memory constrained (or don't have a GPU), a bunch of 1 bit atoms with operations that are very fast on CPU might be better

I haven't thought very deeply about whether it's provably faster to do gradient descent on 32 bits vs 8, but it probably always is. What's the next step to speed up training?

Re: What's Going on in Machine Learning? Some Minimal Models

#37

Post by wrsh07 »

sdenton4 wrote: post #34
wrsh07 wrote: post #4

Because of the computational simplicity, I think there's a possibility that we will discover very cheap machine learning techniques that are discrete like this. I think this is novel (I've seen BNN https://arxiv.org/pdf/1601.06071 This actually makes things continuous for training, but if inference is sufficiently fast and you have an effective mechanism for permutation, training could be faster using that) I am curi…

So, the thing is that linear algebra operations are very cheap already... you just need a lot of them. Any other 'cheap' method is going to have a similar problem: if the unit is small and not terribly expressive, you need a whole lot of them. But it will be compounded by the fact that we don't have decades of investment in making these new atomic operations as fast and cheap as possible. A good take-away from the Wo…

But to your point - that is how I feel about graph nns vs transformers or the fully connected set (GPUs are so good at transformers and fully connected nns, even if there is a structure that makes sense we don't have the hardware to have it make sense.... Unless grok makes it cheap??)

Re: What's Going on in Machine Learning? Some Minimal Models

#38

Post by captainclam »

vessenes wrote: post #13

Classic Wolfram — brilliant, reimplements / comes at a current topic using only cellular automata, and draws some fairly deep philosophical conclusions that are pretty intriguing. The part I find most interesting is his proposal that neural networks largely work by “hitching a ride” on fundamental computational complexity, in practice sort of searching around the space of functions representable by an architecture fo…

"Classic Wolfram — brilliant, reimplements / comes at a current topic using only cellular automata, and draws some fairly deep philosophical conclusions that are pretty intriguing."

Wolfram has a hammer and sees everything as a nail. But its a really interesting hammer.

Re: What's Going on in Machine Learning? Some Minimal Models

#39

Post by dboreham »

wredue wrote: post #31
dbrueck wrote: post #19

I believe that this is one of the key takeaways for reasoning about LLMs and other seemingly-magical recent developments in AI: "tasks—like writing essays—that we humans could do, but we didn’t think computers could do, are actually in some sense computationally easier than we thought." It hurts one's pride to realize that the specialized thing they do isn't quite as special as was previously thought.

Computers still aren’t writing essays. They are stringing words together using copied data. If they were writing essays, I would suggest that it wouldn’t be so ridiculously easy to pick out the obviously AI articles everywhere.

> They are stringing words together using copied data.

Which is what we will eventually realize is what humans are doing too.

Re: What's Going on in Machine Learning? Some Minimal Models

#40

Post by dboreham »

sdenton4 wrote: post #34
wrsh07 wrote: post #4

Because of the computational simplicity, I think there's a possibility that we will discover very cheap machine learning techniques that are discrete like this. I think this is novel (I've seen BNN https://arxiv.org/pdf/1601.06071 This actually makes things continuous for training, but if inference is sufficiently fast and you have an effective mechanism for permutation, training could be faster using that) I am curi…

So, the thing is that linear algebra operations are very cheap already... you just need a lot of them. Any other 'cheap' method is going to have a similar problem: if the unit is small and not terribly expressive, you need a whole lot of them. But it will be compounded by the fact that we don't have decades of investment in making these new atomic operations as fast and cheap as possible. A good take-away from the Wo…

My take is that the neural network is a bit of a red herring -- people poked around in brains to see what was going on and noticed a network structure with many apparently simple computing nodes. So they tried making similar structures in software and quickly discovered they could do some interesting things. But it may turn out that the neural network was just nature's best implementation for "field programmable matrix manipulation". You can implement the functionality in other ways, not resembling neural networks.
Post reply on HN