They mention symmetries and invariances and what not but I wonder if it would be better to clearly emphasize that, in some problems, when you remove certain kinds of symmetries you are are combinatorially worse off. And this is ridiculously bad in some settings. Learning symmetries automatically used to be something I would see people working on but haven't kept up lately.
The Bitter Lesson Is Misunderstood
251–259 of 259 posts
Re: The Bitter Lesson Is Misunderstood
#252Earlier quoted context omitted.
> Nature didn't first make the human body and then go "that's perfect for filling the dishwasher, now to make it talk amongst itself" and then evolve intelligence. It all evolved at the same time, in conjunction. Nature didn't make decisions about anything. But it also absolutely didn't "all evolved at the same time, in conjunction" (if by that you mean all features, regarding body and intelligence, at the same rate)…
> The substrate is. Doesn't mean the nature of abstract thinking is the same as the nature of the body, in the same way the software as algorithm is not the same as hardware, even if it can only run on hardware. In this you are positing the existance of a _soul_ that exists separately from the body, and is portable amongst bodies. Analogues to how an algorithm (disembodied software) exists outside of the hardware and…
I think whether it's "portable amongst bodies" is orthogonal. A specific consciousness of person X can very well only exist within the specific body of person X, and my argument still remains the same (not saying it's right, just that it's not premised on the constraint that there's a soul and it's independent/portable being true).
The argument is that whether consciousness is independent of a specific body or not, it's still of a different nature.
The consciousness part uses the body (e.g. nerve system, neurons etc), but it's nature is the informational exchange and it's essense is not in the construction of the body as a physical machine (though that's its base), but in the stored "weights" encoding memories and world-knowledge.
Same how with a CPU a specific program it runs is not defined by the CPU but the memory contents (data and variables and logic code). It might as well run in an abstract CPU, or one made of water tubes or billiard balls.
Of course in our case, the consciousness runs on a body - and only a specific body - and can't exist without one (same way a program can't exist as a running program without a CPU). But it doesn't mean its of the same nature as the body - just that the body is its substrate.
Re: The Bitter Lesson Is Misunderstood
#253Earlier quoted context omitted.
I think it's Benedict Evans who frequently posts about 'blue collar' AI work not looking like humanoid robots but instead Amazon fulfillment centers keeping track of millions of individual items or tomato picking robots with MV cameras only keeping the ripe ones as it picks at absurd rates. There are endless corners of the physical world right now where it's not worth automating a task if you need to assign an engine…
Or a generalized model to develop them in a virtual sandbox before deploying them physically, which I think is more likely.
I think the bottleneck for this is still the cost of the physical hw of the robot, and its maintenance.
You need a fairly robust one that needs little maintenance, with a multitude of good sensors and precise actuators to be even remotely useful for sufficiently wide range of tasks (so that you have economy of scales). None of that comes cheap.
Re: The Bitter Lesson Is Misunderstood
#254I am not an expert in AI by any means but I think I know enough about it to comment on one thing: there was an interesting paper not too long ago that showed if you train a randomly-initialized model from scratch on questions, like a bank of physics questions & answers, models will end up with much higher quality if you teach it the simple physics questions first, and then move up to more complex physics questions. T…
Curriculum learning: https://en.wikipedia.org/wiki/Curriculum_learning
Re: The Bitter Lesson Is Misunderstood
#255Earlier quoted context omitted.
> We manage to dominate the world mainly by using brute force to simplify our environment and then maintaining and building systems on top of that simplified environment. If we didn't have the proper tools to selectively ablate our environment's complexity… This is very interesting and I feel there is a lot to unpack here. Could you elaborate on this theory with a few more paragraphs (or books / blogs that elucidate…
> Why does our intelligence only operate on simplified forms? Part of the issue with discussing this is that our understanding of complexity is subjective and adapted to our own capabilities. But the gist of it is that the difficulty of modelling and predicting the behavior of a system scales very sharply with its complexity. At the end of the scale, chaotic systems are basically unintelligible. Since modelling is th…
I think a lot of this is true, but not as critical as is being interpreted.
Compare the economics of purely cognitive AI to in-world robotics AI.
Pure cognitive: Massive scale systems for fast, frictionless and incredibly efficient cognitive system deployment and distribution of benefits are solved. On tap even. Cloud computing and the Internet.
What is the amortized cost per task? Almost nothing.
In-world: The cost of extracting raw resources, parts chain, material process chain, manufacturing, distributing, maintaining, etc.
Then what is the amortized cost per task, for one robot?
Several orders of magnitude more expensive, per task! There is no comparison.
Doing that profitably isn’t going to be the norm for many years.
At what price does a kitchen robot make sense? Not at $1,000,000. “Only $100,000?” “Only $25,000? “Only $10k”? Lower than that?
Compared to a Claude plan? That many people still turn down just to use free tier?
Long before general house helper robots makes any economic sense, we will have had walking talking, socializing, profitable-to-build sex robots at higher price points for price insensitive owners.
There are people who will pay high prices for that, when costs come down.
That will be the canary for general robotic servants or helpers.
The cost isn’t intelligence. There isn’t a particular challenge with in-world information processing and control. It’s the cost of the physical thing that processing happens in.
This is a purely economic problem. Not an AI problem at all.
Re: The Bitter Lesson Is Misunderstood
#256Perhaps the author should have acknowledged the catalyst for this blog post? https://x.com/andrewgwils/status/1953814226188841417
Re: The Bitter Lesson Is Misunderstood
#257In any field where there is a creative element, progress comes in fits and starts that are difficult to predict in advance. No one can accurately predict when we'll get the cure for cancer, for example, in spite of people working on it. But that isn't how investors operate. They want to know what they will get in exchange for giving a company a billion dollars. If you're running an AI business, you need to set expect…
Why do you assume investors don’t know about this? They know some investments follow the power law - very few of them work out but they bring most value. The very existence of openAI and Anthropic are proof of it happening. Imagine you were an investor and you know what you know now (creativity can’t be predicted). How would you then invest in companies? Your answer might converge on existing VC strategies.
I don't assume that at all. Investors absolutely know, but investment is predicated on returns. You can't do that if you can't give a timeline for when value will be generated unless your investment is so small it's practically a donation. Obviously you can invest in moonshots, but you don't want to bet your whole portfolio. Why do you think OpenAI had the governing structure it did before it made its breakthroughs but suddenly both them and Anthropic can do insane raises?
Re: The Bitter Lesson Is Misunderstood
#258Earlier quoted context omitted.
> There is n separating embodiment and intelligence. And yet whetever IQ you have, it can't make you just play the violin without actually having embodied practice first.
If you have sufficient motor control and dexterity, the amount of required practice should be approximately zero. Just calculate the required finger position and bow orientation, pressure, and velocity for optimal production of the desired sound and do that. That is not how humans perform physical tasks though.
> That is not how humans perform physical tasks though
is it not though? wouldn't it just be that our processing center isn't located completely in the skull as we typically think, but is extended to our spinal cord and nervous system? Something is being processed, you're just not conscious of the entire process. This is especially clear to me as a musician: as you're learning to play, you have to be absolutely aware of all of those processes until you can finally just let go and play!