Human perception is essentially 2D+depth. Shouldn't we be feeding the transformers 2D data? Like a convolution front-end? Instead of tokenizing images, shouldn't we be rendering texts?
Spatial intelligence is AI’s next frontier
121–130 of 136 posts
Re: Spatial intelligence is AI’s next frontier
#122Earlier quoted context omitted.
There’s no functional similarity in the slightest. Notice you can’t cite examples.
Hard metrics: LLMs perform NLP, NLU and CSR tasks at humanlike levels. Research findings: LLMs have and use world models. They use some type of abstract thinking - with internal representation that often correspond to human abstract concepts. Which adds up to a capability profile that's amusingly humanlike. Humans, however, don't like that. They really don't. AI effect is too strong, and it demands that humans must b…
These are not examples, they’re narrative (false) equivalencies.
Brains don’t perform Natural language CSR etc, those are cultural extensions separate from mental states etc. there are no functional equivalencies here.
There are many many empirical disputes for function eg
Aru et al “The feasibility of artificial consciousness through the lens of neuroscience” December 2023
Re: Spatial intelligence is AI’s next frontier
#123Earlier quoted context omitted.
> I used to make the comment, pre-LLM, that we needed to get to mouse/squirrel level intelligence rather than trying to get to human level abstract AI. But we got abstract AI first. That surprised me. "AI" is not based on physical real world data and models like our brain. Instead, we chose to analyze human formal (written) communication. ("formal": actual face to face communication has tons of dimensions adding to t…
You seem to be suggesting that current frontier models are only trained on text and not "sensor data". Multi-modal models are trained on the entire internet + vast amounts of synthetic data. Images and videos are key inputs. Camera sensors are capable of capturing much more "sensor data" than the human eye. Neural networks are the worst way to model intelligence, except all other models. You may find this talk enligh…
> You seem to be suggesting
As soon as you start a response like that you should just stop. After all, this is written communication, and what I wrote is plain to see right there.
When you need to start a response that way you should become self-aware that you are not responding to what the person you respond to wrote, but to your own ideas.
There is no need to "interpret" what other people wrote.
Relevant: https://i.imgur.com/Izrqp7d.jpeg
Re: Spatial intelligence is AI’s next frontier
#124Earlier quoted context omitted.
> From reading that, I'm not quite sure if they have anything figured out. I actually agree, but her notes are mostly fluff with no real info in there and I do wonder if they have anything figured out besides "collect spatial data" like imagenet. Right. I was thinking about this back in the 1990s. That resulted in a years-long detour through collision detection, physically based animation, solving stiff systems of no…
>There's some progress in video generation which takes a short clip and extrapolates what happens next. That's a promising line of development. The key to "common sense" is being able to predict what happens next well enough to avoid big mistakes in the short term, a few seconds. How's that coming along? And what's the internal world model, assuming we even know? https://www.youtube.com/watch?v=udPY5rQVoW0 This has b…
That GTA demo isn't about control. The user, not the net, is driving.
That's more like the demos where someone trains on a scene and the neural net can make plausible extensions to the scene as you move the viewpoint. It's more spatial imagination, like the tool in Photoshop that fills in plausible but imaginary backgrounds.
It does handle collisions with the edge of the road. Collisions with other cars don't really work; they mostly disappear. One car splits in half in confusion. The spatial part is making progress, but the temporal part, not so much.
Re: Spatial intelligence is AI’s next frontier
#125I'm very loosely tracking the state of the art of LLM spatial reasoning in this blog post - https://arcturus-labs.com/blog/2025/03/31/visual-reasoning-i...
Hint ... we've got a long way to go.
Re: Spatial intelligence is AI’s next frontier
#126I think a lot of people are really bad at evaluating world models. Feifei is right here that they are multimodal but really they must codify a physics. I don't mean "physics" but "a physics". I also think it's naïve to think this can be done through data alone. I mean just ask a physicist...[0]. But why people are really bad at evaluating them is because the details dominate. What matters here is consistency. We need…
The YouTube video tells a fascinating story. Who would be our Fermi today who can tell the truth and save five years of work, billions of dollars and careers of Ph.D. students? We wouldn’t expect LLM to review a paper and tell us the truth like Fermi did. That is super-intelligence. Thanks for sharing.
My problem with the current environment is that we are too rushed. I think in today's culture no one would have told Dyson to not continue. Or just not care. You got to publish, or perish.
Hard things take time and deep thought. But in our age we seem to not want to think deep. The environment discourages it because it takes longer. It's incredibly difficult to have both speed and quality. They are always in contention.
Mind you, there are a number of Nobel laureates who have claimed they wouldn't have succeeded in today's environment because of this[0]. I'm confident we're so concerned with finding the best that we hinder our ability to.
[0] https://www.sciencealert.com/peter-higgs-says-he-wouldn-t-ha...
Re: Spatial intelligence is AI’s next frontier
#127Earlier quoted context omitted.
The YouTube video tells a fascinating story. Who would be our Fermi today who can tell the truth and save five years of work, billions of dollars and careers of Ph.D. students? We wouldn’t expect LLM to review a paper and tell us the truth like Fermi did. That is super-intelligence. Thanks for sharing.
My problem with the current environment is that we are too rushed. I think in today's culture no one would have told Dyson to not continue. Or just not care. You got to publish, or perish. Hard things take time and deep thought. But in our age we seem to not want to think deep. The environment discourages it because it takes longer. It's incredibly difficult to have both speed and quality. They are always in contenti…
Thanks for the link to Higgs story.
I was trained as a Physicist but went to software engineering. These days I would describe my job as a digital plumber. There are two kinds of holes we are dealing with: rabbit holes and potholes. We tend to get into the rabbit holes because of the love of tools and fall into potholes because of not paying attention.
It turned out the industry might do the same. But the cost would be billions of dollars and decades of work of young talented people.
Re: Spatial intelligence is AI’s next frontier
#128Earlier quoted context omitted.
My problem with the current environment is that we are too rushed. I think in today's culture no one would have told Dyson to not continue. Or just not care. You got to publish, or perish. Hard things take time and deep thought. But in our age we seem to not want to think deep. The environment discourages it because it takes longer. It's incredibly difficult to have both speed and quality. They are always in contenti…
Thanks for the link to Higgs story. I was trained as a Physicist but went to software engineering. These days I would describe my job as a digital plumber. There are two kinds of holes we are dealing with: rabbit holes and potholes. We tend to get into the rabbit holes because of the love of tools and fall into potholes because of not paying attention. It turned out the industry might do the same. But the cost would…
I was originally a physicist too and then went into ML. I absolutely love research, yet this is my biggest gripe with the community. Everything is so product focused that I don't think we're even capable of making good products. Those small problems compound and add up to big problems.
The thing I don't understand is that we're really good at recognizing that big problems can be broken down into small ones and that's how we get them solved. So why is it so difficult to understand that small problems compound to create big ones? It's just going in the other direction, where you didn't explicitly do the breakdown. We're just too quick to dismiss rather than asking "does this lead to a big problem?" It is no wonder so much software seems so broken these days.
Given your comment I get the sentiment that you're also frustrated by the environment. I would also add that every pothole has a rabbit hole underneath. We can go around patching everything but we should be asking what causes the potholes to be created in the first place. If we can prevent them from happening to begin with then that's a far better solution than any amount of patching. I'm absolutely certain it will also be far cheaper. Though I'm also certain you'll be able to patch quite a few potholes before you're able to formulate the new formula of asphalt and start pouring it down. Though nothing prevents you from doing both....
Re: Spatial intelligence is AI’s next frontier
#129Earlier quoted context omitted.
Thanks for the link to Higgs story. I was trained as a Physicist but went to software engineering. These days I would describe my job as a digital plumber. There are two kinds of holes we are dealing with: rabbit holes and potholes. We tend to get into the rabbit holes because of the love of tools and fall into potholes because of not paying attention. It turned out the industry might do the same. But the cost would…
I was originally a physicist too and then went into ML. I absolutely love research, yet this is my biggest gripe with the community. Everything is so product focused that I don't think we're even capable of making good products. Those small problems compound and add up to big problems. The thing I don't understand is that we're really good at recognizing that big problems can be broken down into small ones and that's…
I like your metaphor and will look into asphalt, from physics to chemistry…
This also reminds me of another famous metaphor: “Attention Is All You Need”. The question is: are we paying attention to the wrong things as a society? Could it be that we are pouring all the resources into to getting the perfect solution to the wrong problem?
Attention is the most valuable resource we have, as individuals and society.
Re: Spatial intelligence is AI’s next frontier
#130This is pie in the sky swing for the hills nonsense that lumps everything into space. First world models are oxymoronic. Ecological principles, processes and material outcomes eschew models, they’re unnecessary because they’re provided for in the exchange between DNA, biology, bauplan, brain and environment as fast acting optic flow. Any animal relying on anything as preposterous as a world model is dead, splat, drowned etc. world models and survival are incompatible.
The notion it is reducible to spatial ‘intelligence’ is just another case of flavor of the month in AIs contraction and desperate search for success and funding.
That a world model as inconsistent and inaccurate as storytelling is given here as a premise to hoist cash for flavor of the month spatial details how desperate the search has become in Frontierland.