EURISKO Lives

And yes, I'm serious. Can you please be less confrontational?

I think we have a fundamental disconnect somewhere, so let's try to diagnose it. Where do you start to disagree in the following series of claims:

1. People can have kinematic skills, like throwing and catching balls, without having math or physics skills, like solving kinematic equations.

2. In order to have kinematic skills, something in your brain must be doing something that can be equated by some mapping to solving kinematic equations, because the actions that your muscles perform when performing kinematic skills are the solutions to kinematic equations, so your brain must be producing those (things that map to) solutions somehow.

3. As far as we can tell, brains don't operate symbolically at the neurobiological level. Individual neurons operate according to laws having to do with electrical impulses, synapse firings, neurotransmitters, etc. none of which have anything to do with kinematics.

4. People with kinematic skills generally have only limited insight into how they do what they do when they apply those skills. Being able to catch a ball doesn't by itself give you enough insight to be able to describe to someone how to build a machine that would catch a ball. But someone with math and physics and engineering skills but no kinematic skills (your streotypical geek) could plausibly build a machine that could catch a ball much better than they themselves could. But the workings of a machine built using knowledge of math would almost certainly operate in a very different manner than the brain of a human with kinematic skills.

I think I'll stop there and ask if there is anything you disagree with so far.

The tricky part here is that "efficiency" is not a single dimension! Transformers are much more "efficient" in one sense, in that they appear to be able to absorb much more data before they saturate; they're in general less computationally efficient in that you can't exploit symmetries as hard, for example, at implementation time.

Let's talk about that in terms of a concrete example: the big inductive bias of CNNs for vision problems is that CNNs essentially presuppose that the model should be translation-invariant. This works great — speeds up training and makes it more stable – until it doesn't and that inductive bias starts limiting your performance, which is in the large-data limit.

Fully-connected NNs are more general than transformers, but they have _so many_ degrees of freedom that the numerical optimization problem is impractical. If someone figures out how to stabilize that training and make these implementable on current or future hardware, you're absolutely right that you'll see people use them. I don't think transformers are magic; you're entirely correct in saying that they're the current knee on the implementability/trainability curve, and that can easily shift given different unit economics.

I think one of the fundamental disconnects here is that people who come at AI from the perspective of logic down think of things very differently to people like me who come at it from thermodynamics _up_.

Modern machine learning is just "applications of maximum entropy", and to someone with a thermodynamics background, that's intuitively obvious (not necessarily correct! just obvious) –in a meaningful sense the _universe_ is a process of gradient descent, so "of course" the answer for some local domain models is maximum-entropy too. In that world view, the higher-order structure is _entirely emergent_. I'm, by training, a crystallographer, so the idea that you can get highly regular structure emerging from merciless application of a single principle is just baked into my worldview very deeply.

Someone who comes at things from the perspective of mathematical logic is going to find that worldview very weird, I suspect.

To clarify what I mean on this specific bit: the SOTA results in 2D and 3D vision, audio, translation, NLP, etc are all transformers. Past results do not necessarily predict future performance, and it would be absurd to claim that an immutable state of affairs, but it's certainly interesting that all of the domain-specific architectures have been flattened in a very short period of time.