What's Going on in Machine Learning? Some Minimal Models
writings.stephenwolfram.com
What's Going on in Machine Learning? Some Minimal Models
1–10 of 73 posts
Re: What's Going on in Machine Learning? Some Minimal Models
#2There should be a Godwin’s Law for Stephen Wolfram. Wolfram’s Law: as the length of what he’s saying increases, the probability it will be about cellular automata approaches 1.
That being said, I’m enjoying this. I often experiment with neural networks in a similar fashion and like to see people’s work like this.
Re: What's Going on in Machine Learning? Some Minimal Models
#3There should be a Godwin’s Law for Stephen Wolfram. Wolfram’s Law: as the length of what he’s saying increases, the probability it will be about cellular automata approaches 1. That being said, I’m enjoying this. I often experiment with neural networks in a similar fashion and like to see people’s work like this.
...and the probability that he names something after himself approaches 1/e.
Re: What's Going on in Machine Learning? Some Minimal Models
#4Because of the computational simplicity, I think there's a possibility that we will discover very cheap machine learning techniques that are discrete like this.
I think this is novel (I've seen BNN https://arxiv.org/pdf/1601.06071 This actually makes things continuous for training, but if inference is sufficiently fast and you have an effective mechanism for permutation, training could be faster using that)
I am curious what other folks (especially researchers) think. The takes on Wolfram are not always uniformly positive but this is interesting (I think!)
Re: What's Going on in Machine Learning? Some Minimal Models
#5>Instead what seems to be happening is that machine learning is in a sense just “hitching a ride” on the general richness of the computational universe. It’s not “specifically building up behavior one needs”; rather what it’s doing is to harness behavior that’s “already out there” in the computational universe.
Is this similar to the lottery ticket hypothesis?
Also the visualizations are beautiful and a nice way to demonstrate the "universal approximation theorem"
Re: What's Going on in Machine Learning? Some Minimal Models
#6There should be a Godwin’s Law for Stephen Wolfram. Wolfram’s Law: as the length of what he’s saying increases, the probability it will be about cellular automata approaches 1. That being said, I’m enjoying this. I often experiment with neural networks in a similar fashion and like to see people’s work like this.
There's a Wolfram Derangement Syndrome to go along with it, too: https://news.ycombinator.com/item?id=38975876
Re: What's Going on in Machine Learning? Some Minimal Models
#7Wow- Wolfram "invented" cellular automata, neural nets, symbolic algebra, physics and so much more.
Re: What's Going on in Machine Learning? Some Minimal Models
#8Is a TL;DR available or at least some of the ideas covered? Because after 3 paragraphs it seems the good old "it is actually something resembling a cellular automata" post by Wolfram.
Re: What's Going on in Machine Learning? Some Minimal Models
#9Tsetlin machines have been around for some time:
https://en.wikipedia.org/wiki/Tsetlin_machine
They are discrete, individually interpretable, and can be configured into complicated architectures.
Re: What's Going on in Machine Learning? Some Minimal Models
#10> But now we get to use a key feature of infinitesimal changes: that they can always be thought of as just “adding linearly” (essentially because ε2 can always be ignored to ε). Or, in other words, we can summarize any infinitesimal change just by giving its “direction” in weight space
> a standard result from calculus gives us a vastly more efficient procedure that in effect “maximally reuses” parts of the computation that have already been done.
This partially explains why gradient descent becomes mainstream.