Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]

youtube.com

Post reply on HN
Print view View original

81–90 of 163 posts

Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]

#81

Post by otodus »

coldtea wrote: post #37
lsy wrote: post #22

Earlier quoted context omitted.

To make this more concrete: ImageNet enabled computer "vision" by providing images + labels, enabling the computer to take an image and spit out a label. LLM training sets enable text completion by providing text + completions, enabling the computer to take a piece of text and spit out its completion. Learning how the physical world works (not just kind of works a la videogames, actually works ) is not only about a j…

> there is really only one usable dataset: the world itself, which cannot be compacted or fed into a computer at high speed. Why wouldn't it be? If the world is ingressed via video sensors and lidar sensor, what's the hangup in recording such input and then replaying it faster?

How would you handle olfactory and proprioceptive data?

Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]

#82

Post by tmilard »

ccozan wrote: post #15
jandrewrogers wrote: post #8

I appreciate the video and generally agree with Fei-Fei but I think it almost understates how different the problem of reasoning about the physical world actually is. Most dynamics of the physical world are sparse, non-linear systems at every level of resolution. Most ways of constructing accurate models mathematically don’t actually work. LLMs, for better or worse, are pretty classic (in an algorithmic information t…

I agree that the problem is hard. However, biological brain is able to handle it quite "easily" ( is not really easy - bilions of iterations were needed ). The current brains are solving this 3D physical world _only_ via perception. So this is place were we must look. It starts with the sensing and the integration of that sensing. I am working at this problem since more than 10 years and I came to some results. I am…

I do agree with you. We have an natural eye (what you call a 'biological brain') automat that inconsciouly 'feels' the structure of a geometric of the places we enter to.

Once this layer of "natural eye automat" is programmed behind a camera, it will spit out this crude geometry : the Spacial-data-bulk (SDB). This SDB is small data.

From now on, our programs will only do reason, not on data froms camera(s) but only on this small SBD.

This is how I see it.

Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]

#83

Post by tmilard »

tmilard wrote: post #82
ccozan wrote: post #15

Earlier quoted context omitted.

I agree that the problem is hard. However, biological brain is able to handle it quite "easily" ( is not really easy - bilions of iterations were needed ). The current brains are solving this 3D physical world _only_ via perception. So this is place were we must look. It starts with the sensing and the integration of that sensing. I am working at this problem since more than 10 years and I came to some results. I am…

I do agree with you. We have an natural eye (what you call a 'biological brain') automat that inconsciouly 'feels' the structure of a geometric of the places we enter to. Once this layer of "natural eye automat" is programmed behind a camera, it will spit out this crude geometry : the Spacial-data-bulk (SDB). This SDB is small data. From now on, our programs will only do reason, not on data froms camera(s) but only o…

==> And now the LLMs, to feel Spacial knowledge, will have a very reduce dataset. This will make spacial data reasoning very less intencive than we can't imagine.

Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]

#84

Post by tmilard »

jgord wrote: post #14
skwb wrote: post #3

It's hard to describe, but it's felt like LLMs have completely sucked the entire energy out of computer vision. Like... I know CVPR still happens and there's great research that comes out of it, but almost every single job posting in ML is about LLMs to do this and that to the detriment of computer vision.

yeah, see my other comment. To me its totally obvious that we will have a plethora of very valuable startups who use RL techniques to solve realworld problems in practical areas of engineering .. and I just get blank stares when I talk about this :] Ive stopped saying AI when I mean ML or RL .. because people equate LLMs with AI. We need better ML / RL algos for CV tasks : - detecting lines from pixels - detecting ge…

You said : "- detecting lines from pixels - detecting geometry in pointclouds - constructing 3D from stereo images, photogrammetry, 360 panoramas"

  ==> For me it is more something like :
   Source = crude video-or-photo pixels  (to) ===> Find simple many rectangle-surface  that are glued together one another.
This is, for me, how you really go easily to detecting rather complexes geometry of any room.

Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]

#85

Post by cma »

jampekka wrote: post #60
cma wrote: post #34

Earlier quoted context omitted.

Horses can be blindfolded at birth and when removed do basic navigation with no time for any training. Other non-visually precocious animals like cats, if they miss a critical development period without getting natural vision data, will never develop a functioning visual system. Baby chicks can do bipedal balance pretty much as soon as they dry off. Wood ducks can visually imprint very soon after hatching and drying…

It's not clear whether humans have natural reactions to snakes. https://link.springer.com/article/10.11133/j.tpr.2013.63.4.0...

If not, another visual system one in newborns is preference for faces with open eyes and direct gaze:

https://pubmed.ncbi.nlm.nih.gov/17030037/

Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]

#86

Post by devenson »

m-s-y wrote: post #71
loa_in_ wrote: post #62

Earlier quoted context omitted.

But the playpen will contain objects that are inherently breakable. You cannot rough handle the glass vessel and have it too.

The world Is breakable. Any model based on it will need to know this anyway. Am I missing your argument?

Can't reset state after breakage.

Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]

#87

Post by smath »

skwb wrote: post #3

It's hard to describe, but it's felt like LLMs have completely sucked the entire energy out of computer vision. Like... I know CVPR still happens and there's great research that comes out of it, but almost every single job posting in ML is about LLMs to do this and that to the detriment of computer vision.

agreed about sucking the air out by LLM. The positive side is that its a good time to innovate in other areas while a chunk of ppl are absorbed in LLMs. A proven improvement in any other non LLM space will attract investment.

Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]

#88

Post by AStrangeMorrow »

owenpalmer wrote: post #78

While spatial intelligence is certainly a major limitation of current AI systems, I have been able to get LLMs to do quite impressive things. Here's an on the fly video I made (no retakes) of Claude generating a Godot scene file. https://youtu.be/2gARJpDG7Jo?si=W4rlISO-J4EPJYyG

Yeah funnily I did a project where we had an LLM based interface to in house 3D parametric modeling system and it did fairly well.

However I’ve been trying to use LLM, both as orchestrators and in other cases to write code for 2D optimization problems with many spatial relationships and it has done terribly.

I have talking it can generate 1000s of lines over many rounds of prompting/iteration that solve maybe 30% of the problem (and the 30% very easy cases) while completely messing up the rest. When doing that code myself, in less than 1000 lines, the “30% part” was maybe 3% of the total code. Even when basically providing pseudo code to solve specific part of the problem chances are these LLM solutions would also have many blind spots or issues.

The thing is, that is a 2D problem for which there basically no ressources about online, and all the slightly similar problems all have careful handcrafted specialized solutions. So I think it has no good frame of reference how to solve the problem

Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]

#89

Post by rtaylorgarlock »

Workaccount2 wrote: post #66
moktonar wrote: post #58

Intelligence is not only embodied (it needs a body), it is also embedded in the environment (it needs the environment). If you want an intelligence in your computer, you need an environment in your computer first, as the substrate from which the intelligence will evolve. The more accurate the environment the better the intelligence that will be obtained. The universe is able to create intelligence and we are proof. T…

The silicon exists in the same environment that our brains do.

The drivers of its failure to adequately assimilate to said environment being?

Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]

#90

Post by AStrangeMorrow »

weinzierl wrote: post #4

I tried LLM's for geolocation recently and it is both amazing how good they are at recognizing patterns and how terrible they are with recognizing and utilizing basic spatial relationships.

Yeah even LLM generated code for a 2D optimization problems with many spatial relationships has being absolutely terrible, while I had great success in other domains.
Post reply on HN