mr. yann-le-cunn's jepa paper is quite instructive.
Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]
11–20 of 163 posts
Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]
#12I tried LLM's for geolocation recently and it is both amazing how good they are at recognizing patterns and how terrible they are with recognizing and utilizing basic spatial relationships.
I would like to read a complete example if you want to share (I am not disputing yourbpoint, I'd just to understand better because this is not my field so I cannot immediately map your comment to my own experience)
Happy to share an complete example privately, contact data is in my profile.
Will add condensed version here in half an hour.
Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]
#13I appreciate the video and generally agree with Fei-Fei but I think it almost understates how different the problem of reasoning about the physical world actually is. Most dynamics of the physical world are sparse, non-linear systems at every level of resolution. Most ways of constructing accurate models mathematically don’t actually work. LLMs, for better or worse, are pretty classic (in an algorithmic information t…
"Most ways of constructing accurate models mathematically don’t actually work"
> This is true for almost anything at the limit, we are already able to model spatiotemporal dynamics to some useful degree (see: progress in VLAs, video diffusion, 4D Gaussians)
"We’ve known for well over a decade that you cannot cram real-world spatial dynamics into those models. It is a clear impedance mismatch" > What's the source that this is a physically impossible problem? Not sure what you mean by impedance mismatch but do you mean that it is unsolvable even with better techniques?
Your whole third paragraph could have been said about LLMs and isn't specific enough, so we'll skip that.
I don't really understand the other 2 paragraphs, what's this "dark state-of-the-art computer science" you speak of and what is this "area of research became highly developed over two decades but is not in the public literature" how is "the computer science required is so different from classical AI research"?
Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]
#14It's hard to describe, but it's felt like LLMs have completely sucked the entire energy out of computer vision. Like... I know CVPR still happens and there's great research that comes out of it, but almost every single job posting in ML is about LLMs to do this and that to the detriment of computer vision.
yeah, see my other comment.
To me its totally obvious that we will have a plethora of very valuable startups who use RL techniques to solve realworld problems in practical areas of engineering .. and I just get blank stares when I talk about this :]
Ive stopped saying AI when I mean ML or RL .. because people equate LLMs with AI.
We need better ML / RL algos for CV tasks :
- detecting lines from pixels
- detecting geometry in pointclouds
- constructing 3D from stereo images, photogrammetry, 360 panoramas
Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]
#15I appreciate the video and generally agree with Fei-Fei but I think it almost understates how different the problem of reasoning about the physical world actually is. Most dynamics of the physical world are sparse, non-linear systems at every level of resolution. Most ways of constructing accurate models mathematically don’t actually work. LLMs, for better or worse, are pretty classic (in an algorithmic information t…
I agree that the problem is hard. However, biological brain is able to handle it quite "easily" ( is not really easy - bilions of iterations were needed ). The current brains are solving this 3D physical world _only_ via perception.
So this is place were we must look. It starts with the sensing and the integration of that sensing. I am working at this problem since more than 10 years and I came to some results. I am not a real scientist but a true engineer and I am looking from that perspective quite intesely: The question that one must ask is: how do you define the outside physical world from the perspective of a biological sensing "device" ? what exactly are we "seeing" or "hearing"? So yes, working on that brought it further in defining the physical world.
Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]
#16makes sense - humans have evolved a lot of wetware dedicated to 3D processing from stereo 2D. I've made some progress on a PoC in 3D reconstruction - detecting planes, edges, pipes from pointclouds from lidar scans, eg : https://youtu.be/-o58qe8egS4 .. and am bootstrapping with in-house gigs as I build out the product. Essentially it breaks down to a ton of matmulls, and I use a lot of tricks from pre-LLM ML .. this…
[dead]
Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]
#17I appreciate the video and generally agree with Fei-Fei but I think it almost understates how different the problem of reasoning about the physical world actually is. Most dynamics of the physical world are sparse, non-linear systems at every level of resolution. Most ways of constructing accurate models mathematically don’t actually work. LLMs, for better or worse, are pretty classic (in an algorithmic information t…
All (ALL!!) AI/optimization problems boil down to energy minimization or dually entropy maximization.
Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]
#18makes sense - humans have evolved a lot of wetware dedicated to 3D processing from stereo 2D. I've made some progress on a PoC in 3D reconstruction - detecting planes, edges, pipes from pointclouds from lidar scans, eg : https://youtu.be/-o58qe8egS4 .. and am bootstrapping with in-house gigs as I build out the product. Essentially it breaks down to a ton of matmulls, and I use a lot of tricks from pre-LLM ML .. this…
ps. its handy to compare the relative data sizes of [ models of ] the same scene : typically for something like a house, the data will be ballpark :
- 15GB of pointcloud data ( 100Mn xyzRGB points from a lidar laser scanner )
- 3 GB of 360 panorama photos
- 50MB obj 3D textured model
- 2MB CAD model
Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]
#19It's hard to describe, but it's felt like LLMs have completely sucked the entire energy out of computer vision. Like... I know CVPR still happens and there's great research that comes out of it, but almost every single job posting in ML is about LLMs to do this and that to the detriment of computer vision.
I feel like 3D reconstruction/bundle adjustment is one of those things where LLMs and new AI stuff haven't managed to get a significant foothold. Recently VGGT won best paper which is good for them, but for the most part, stuff like NERF and Gaussian Splatting still rely on good old COLMAP for bundle adjustment using SIFT features.
Also, LLMs really suck at some basic tasks like counting the sides of a polygon.
Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]
#20I've always wondered how spatial reasoning appears to be operating quite differently from other cognitive abilities, with significant individual variations. Some people effortlessly parallel park while others struggle with these tasks despite excelling at other forms of pattern recognition. What was particularly intriguing for me is that some people with aphantasia have no difficulty with spatial reasoning tasks, so spatial reasoning may be distinct from reasoning based on internal visualization.