better remove all the Disney-based Alice in Wonderland character intellectual property from the book.
[deleted]
Alice's adventures in a differentiable wonderland
91–100 of 103 posts
Re: Alice's adventures in a differentiable wonderland
#92Earlier quoted context omitted.
Alice in Wonderland (the book) is in the public domain. The old Disney movie is still in copyright, and the cover image does look very much like it's from the movie, but that character design is from John Tenniel's illustrations which are also in the public domain.
The character design is. The image, however, is clearly Disney flavoured if not traced directly. His version, for example, does not have the distinctive bow. The art style is also completely different.
True - it would be a good idea to use a Tenniel piece instead.
Edit to add: I was mostly trying to push back on the implication that Disney owns Alice in Wonderland (and Peter Pan, Winnie the Pooh, etc). Now I re-read the original comment, they did specify "Disney-based", so maybe I'm over-reacting!
Re: Alice's adventures in a differentiable wonderland
#93Earlier quoted context omitted.
idk, just using attention and massive web crawls gets you pretty far. a lot of the rest is more product-style decisions about what personality you want your LM to take. I fundamentally don't think this technology is that complex.
No? In his recent tutorial, Karpathy showed just how much complexity there is in the tokenizer. This technology has been years in the making with many small advances pushing the performance ever so slightly. There’s been theoretical and engineering advances that contributed to where we are today. And we need many more to get the technology to an actually usable level instead of the current word spaghetti that we get.…
Are there micro-optimizations that eke out small advancements? Yes, absolutely - the modern tokenizer is a good example of that.
Is the core of the technology that complex? No. You could get very far with a naive tokenizer that just tokenized by words and replaced unknown words with . This is extremely simple to implement and I've trained transformers like this. It (of course) makes a perplexity difference but the core of the technology is not changed and is quite simple. Most of the complexity is in the hardware, not the software innovations.
> And we need many more to get the technology to an actually usable level instead of the current word spaghetti that we get.
I think the current technology is useable.
> you shouldn’t just choose the attention hammer and hammer away
It's a good first choice of hammer, tbph.
Re: Alice's adventures in a differentiable wonderland
#94> Stripped of anything else, neural networks are compositions of differentiable primitives I’m a sucker for statements like this. It almost feels philosophical, and makes the whole subject so much more comprehensible in only a single sentence. I think François Chollet says something similar in his book on deep learning: one shouldn’t fall into the trap of anthropomorphising and mysticising models based on the ‘neural…
I feel like this statement is both obvious after spending a few minutes working with neural networks and completely useless in helping you build better neural networks. It's kind of like saying, "Stripped of anything else, works of literature are compositions of words"
Let me try to one-up that: "Stripped off anything else, works of literature are compositions of the uppercase letter I"
Re: Alice's adventures in a differentiable wonderland
#95Earlier quoted context omitted.
> the biological version of a neuron (i.e. a neuron) is much more complicated than the neural network version This is a difference of degree not of kind, because neural networks are Turning complete. Whatever additional complexity the neuron has can itself be modelled as a neural network. Edit: meaning, that if the greater complexity of a biological neuron is relevant to its information processing component, then tha…
> This is a difference of degree not of kind Nope. Neurons in our brain operate fundamentally differently. They work by transient spikes and information is carried not by the intensity of the spike voltage, but by the frequency of spiking. This is a fundamentally different phenomenon than ANNs where the output (voltage) is a squash transformed aggregated input values (voltages).
>> Neurons in our brain operate fundamentally differently. They work by transient spikes and information is carried not by the intensity of the spike voltage, but by the frequency of spiking.
I thought they worked like accumulators where the spike "energy" accumulates until the output "fires". If that's the case then the artificial NNs are still an approximation of that process. I agree that this is a significant difference, but the mathematical version is still a rough approximation inspired by the biological one.
Re: Alice's adventures in a differentiable wonderland
#96Earlier quoted context omitted.
you have an interesting point of view, and some of the things you have said are correct, but if you try to use gradient descent on a function from, say, ℤ → ℝ, you are going to be a very sad xanda. i would indeed describe such a function as being discontinuous not just at π but everywhere, at least with the usual definition of continuity (though there is a sense in which such a function could be, for example, scott-c…
Note that any function ℤ → ℝ is continuous on its domain but nowhere differentiable. A Scott-continuous function ℤ → ℝ must be monontonous. So not every such function is Scott-continuous.
aha, thanks!
Re: Alice's adventures in a differentiable wonderland
#97Earlier quoted context omitted.
Analog circuits have the same computational power. Piecewise linear functions have the same computational universality.
Except we didn't know any of that, nor did know how to construct physical analogs in order to achieve universal computation. At best, we had limited task-specific computation, like clocks and planetary motion.
We knew about universal function approximators like. Polynomials and trig functions since the 1700s. Turing and godel were around 1910 and 1920. The cybernetics movement is big in the 30s and 40s. Perceptrons 50s and 60s
Re: Alice's adventures in a differentiable wonderland
#98Earlier quoted context omitted.
Except we didn't know any of that, nor did know how to construct physical analogs in order to achieve universal computation. At best, we had limited task-specific computation, like clocks and planetary motion.
We knew about universal function approximators like. Polynomials and trig functions since the 1700s. Turing and godel were around 1910 and 1920. The cybernetics movement is big in the 30s and 40s. Perceptrons 50s and 60s
Taylor expansions for all functions do not exist. Furthermore, our characterization of infinity was still poor, so we didn't even have a solid notion of what it would mean for a formalism to be able to compute all computable functions. The notion of a universal computer arguably didn't exist until Babbage.
I stand by my position that having a mathematical proof of computational universality is a significant difference that separates today from all prior eras that sought to understand the brain through contemporaneous technology.
Re: Alice's adventures in a differentiable wonderland
#99Earlier quoted context omitted.
We knew about universal function approximators like. Polynomials and trig functions since the 1700s. Turing and godel were around 1910 and 1920. The cybernetics movement is big in the 30s and 40s. Perceptrons 50s and 60s
Taylor expansions for all functions do not exist. Furthermore, our characterization of infinity was still poor, so we didn't even have a solid notion of what it would mean for a formalism to be able to compute all computable functions. The notion of a universal computer arguably didn't exist until Babbage. I stand by my position that having a mathematical proof of computational universality is a significant differenc…
> Taylor expansions
That’s not what I’m talking about. This is a basic analysis topic:
https://en.m.wikipedia.org/wiki/Stone%E2%80%93Weierstrass_th...
At least mid 1800s for a proof. 1700s also explored Fourier series.
> stand by my position
And you’re still ignoring the cybernetics, and perceptrons movement I keep referring to which was more than 100 years ago, and informed by Turing.
Re: Alice's adventures in a differentiable wonderland
#100Earlier quoted context omitted.
Taylor expansions for all functions do not exist. Furthermore, our characterization of infinity was still poor, so we didn't even have a solid notion of what it would mean for a formalism to be able to compute all computable functions. The notion of a universal computer arguably didn't exist until Babbage. I stand by my position that having a mathematical proof of computational universality is a significant differenc…
> Taylor expansions That’s not what I’m talking about. This is a basic analysis topic: https://en.m.wikipedia.org/wiki/Stone%E2%80%93Weierstrass_th... At least mid 1800s for a proof. 1700s also explored Fourier series. > stand by my position And you’re still ignoring the cybernetics, and perceptrons movement I keep referring to which was more than 100 years ago, and informed by Turing.
> That’s not what I’m talking about. This is a basic analysis topic:
It's the same basic flaw: requiring continuous functions. Not all functions are continuous, therefore this is not sufficient.
> And you’re still ignoring the cybernetics, and perceptrons movement I keep referring to which was more than 100 years ago, and informed by Turing.
What about them? As long as they're universal, they can all simulate brains. Anything after Church and Turing is just window dressing. Notice how none of these new ideas claimed to change what could in principle be computed, only how much easier or more natural this paradigm might be for simulating or creating brains.