Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]

youtube.com

Post reply on HN
Print view View original

91–100 of 163 posts

Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]

#91

Post by rybosome »

HappMacDonald wrote: post #55
psb217 wrote: post #41

Earlier quoted context omitted.

I think there's an implicit assumption here that interaction with the world is critical for effective learning. In that case, you're bottlenecked by the speed of the world... when learning with a single agent. One neat thing about artificial computational agents, in contrast to natural biological agents, is that they can share the same brain and share lived experience, so the "speed of reality" bottleneck is much les…

Yeah I'm envisioning putting a thousand simplistic robotic "infants" into a vast "playpen" to gather sensor data about their environment, for some (probably smaller) number of deep learning models to ingest the input and guess at output strategies (move this servo, rotate this camshaft this far in that direction, etc) and make predictions about resulting changes to input. In principle a thousand different deep learni…

A very interesting idea. I am curious about this sharing and blending of the various nets; I wonder if something as naive as averaging the weights (assuming the neural nets all have the same dimensions) would actually accomplish that?

Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]

#92

Post by queuebert »

jandrewrogers wrote: post #8

I appreciate the video and generally agree with Fei-Fei but I think it almost understates how different the problem of reasoning about the physical world actually is. Most dynamics of the physical world are sparse, non-linear systems at every level of resolution. Most ways of constructing accurate models mathematically don’t actually work. LLMs, for better or worse, are pretty classic (in an algorithmic information t…

I think you are using "sparse" and "non-linear" as scare terms. Sparse is a good thing, as it reduces degrees of freedom, and non-linear does not mean unsolvable.

Also "impedance mismatch" doesn't mean no go, but rather less efficient.

Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]

#93

Post by CSMastermind »

skwb wrote: post #3

It's hard to describe, but it's felt like LLMs have completely sucked the entire energy out of computer vision. Like... I know CVPR still happens and there's great research that comes out of it, but almost every single job posting in ML is about LLMs to do this and that to the detriment of computer vision.

Everyone is trying to jam transformers into CV workflows at the moment. Possibly productively.

Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]

#94

Post by CamperBob2 »

jandrewrogers wrote: post #8

I appreciate the video and generally agree with Fei-Fei but I think it almost understates how different the problem of reasoning about the physical world actually is. Most dynamics of the physical world are sparse, non-linear systems at every level of resolution. Most ways of constructing accurate models mathematically don’t actually work. LLMs, for better or worse, are pretty classic (in an algorithmic information t…

We’ve known for well over a decade that you cannot cram real-world spatial dynamics into those models. It is a clear impedance mismatch.

Then again, not much that we "knew" a decade ago is still relevant today. Of course transformer networks have proven capable of representing spatial intelligence. How could they work with 2D images, if not?

Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]

#95

Post by pixl97 »

smath wrote: post #87
skwb wrote: post #3

It's hard to describe, but it's felt like LLMs have completely sucked the entire energy out of computer vision. Like... I know CVPR still happens and there's great research that comes out of it, but almost every single job posting in ML is about LLMs to do this and that to the detriment of computer vision.

agreed about sucking the air out by LLM. The positive side is that its a good time to innovate in other areas while a chunk of ppl are absorbed in LLMs. A proven improvement in any other non LLM space will attract investment.

Isn't this what Nvidia is already doing with a lot of their sim software?

Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]

#96

Post by pixl97 »

moktonar wrote: post #58

Intelligence is not only embodied (it needs a body), it is also embedded in the environment (it needs the environment). If you want an intelligence in your computer, you need an environment in your computer first, as the substrate from which the intelligence will evolve. The more accurate the environment the better the intelligence that will be obtained. The universe is able to create intelligence and we are proof. T…

>Intelligence is not only embodied (it needs a body), it is also embedded in the environment (it needs the environment).

Of course don't make the mistake that we need anything like a human body, or any singular object containing 'intelligence'. That's simply the way nature had to do it to connect a sensor platform to a brain. AI seems much more like it will be a hive mind and distributed system of data collection.

Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]

#98

Post by whiplash451 »

skwb wrote: post #3

It's hard to describe, but it's felt like LLMs have completely sucked the entire energy out of computer vision. Like... I know CVPR still happens and there's great research that comes out of it, but almost every single job posting in ML is about LLMs to do this and that to the detriment of computer vision.

It felt the same back in 2012-2015 when deep learning was flooding over computer vision. Yet 10 years later there is a net benefit for computer vision: a lot of tasks are now solved much better/more efficiently with deep learning including those that seemed "unfit" to deep learning like tracking.

I'm hopeful that VLMs will "fan out" into a lot of positive outcomes for computer vision.

Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]

#99

Post by jandrewrogers »

teemur wrote: post #42
jandrewrogers wrote: post #8

I appreciate the video and generally agree with Fei-Fei but I think it almost understates how different the problem of reasoning about the physical world actually is. Most dynamics of the physical world are sparse, non-linear systems at every level of resolution. Most ways of constructing accurate models mathematically don’t actually work. LLMs, for better or worse, are pretty classic (in an algorithmic information t…

> We know that universal solutions can’t exist and that all practical solutions require exotic high-dimensionality computational constructs that human brains will struggle to reason about. This has been the status quo since the 1980s. This particular set of problems is hard for a reason. This made me a bit curious. Would you have any pointers to books/articles/search terms if one wanted to have a bit deeper look on t…

I'm not aware of any convenient literature but it is relatively obvious once someone explains it to you (as it was explained to me).

At its root it is a cutting problem, like graph cutting but much more general because it includes things like non-trivial geometric types and relationships. Solving the cutting problem is necessary to efficiently shard/parallelize operations over the data models.

For classic scalar data models, representations that preserve the relationships have the same dimensionality as the underlying data model. A set of points in 2-dimensions can always be represented in 2-dimensions such that they satisfy the cutting problem (e.g. a quadtree-like representation).

For non-scalar types like rectangles, operations like equality and intersection are distinct and there are an unbounded number of relationships that must be preserved that touch on concepts like size and aspect ratio to satisfy cutting requirements. The only way to expose these additional relationships to cutting algorithms is to encode and embed these other relationships in a (much) higher dimensionality space and then cut that space instead.

The mathematically general case isn't computable but real-world data models don't need it to be. Several decades ago it was determined that if you constrain the properties of the data model tightly enough then it should be possible to systematically construct a finite high-dimensionality embedding for that data model such that it satisfies the cutting problem.

Unfortunately, the "should be possible" understates the difficulty. There is no computer science literature for how one might go about constructing these cuttable embeddings, not even for a narrow subset of practical cases. The activity is also primarily one of designing data structures and algorithms that can represent complex relationships among objects with shape and size in dimensions much greater than three, which is cognitively difficult. Many smart people have tried and failed over the years. It has a lot of subtlety and you need practical implementations to have good properties as software.

About 20 years ago, long before "big data", the iPhone, or any current software fashion, this and several related problems were the subject of an ambitious government research program. It was technically successful, demonstrably. That program was killed in the early 2010s for unrelated reasons and much of that research was semi-lost. It was so far ahead of its time that few people saw the utility of it. There are still people around that were either directly involved or learned the computer science second-hand from someone that was but there aren't that many left.

Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]

#100

Post by hackyhacky »

psb217 wrote: post #41
coldtea wrote: post #37

Earlier quoted context omitted.

> there is really only one usable dataset: the world itself, which cannot be compacted or fed into a computer at high speed. Why wouldn't it be? If the world is ingressed via video sensors and lidar sensor, what's the hangup in recording such input and then replaying it faster?

I think there's an implicit assumption here that interaction with the world is critical for effective learning. In that case, you're bottlenecked by the speed of the world... when learning with a single agent. One neat thing about artificial computational agents, in contrast to natural biological agents, is that they can share the same brain and share lived experience, so the "speed of reality" bottleneck is much les…

> In that case, you're bottlenecked by the speed of the world

Why not have the AI train on a simulation of the real world? We can build those pretty easily using traditional software and run them at any speed we want.

Post reply on HN