Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]

youtube.com

Post reply on HN
Print view View original

111–120 of 163 posts

Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]

#111

Post by ryeguy_24 »

jandrewrogers wrote: post #8

I appreciate the video and generally agree with Fei-Fei but I think it almost understates how different the problem of reasoning about the physical world actually is. Most dynamics of the physical world are sparse, non-linear systems at every level of resolution. Most ways of constructing accurate models mathematically don’t actually work. LLMs, for better or worse, are pretty classic (in an algorithmic information t…

Agree. Also, with respect to training, what is the goal that we are maximizing? LLMs are easy, predicting the next word and we have lots of training data. But what are we training for in real world? Modeling the next spatial photograph to predict things that will happen next? It’s not intuitive to me what that objective function would be in spatial intelligence.

Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]

#112

Post by calf »

dopadelic wrote: post #107
jandrewrogers wrote: post #8

I appreciate the video and generally agree with Fei-Fei but I think it almost understates how different the problem of reasoning about the physical world actually is. Most dynamics of the physical world are sparse, non-linear systems at every level of resolution. Most ways of constructing accurate models mathematically don’t actually work. LLMs, for better or worse, are pretty classic (in an algorithmic information t…

You're pointing out a real class of hard problems — modeling sparse, nonlinear, spatiotemporal systems — but there’s a fundamental mischaracterization in lumping all transformer-based models under “LLMs” and using that to dismiss the possibility of spatial reasoning. Yes, classic LLMs (like GPT) operate as sequence predictors with no inductive bias for space, causality, or continuity. They're optimized for language f…

How do we prove a trained LLM has no inductive bias for space, causality, etc.? We can't assume this is true by construction, can we?

Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]

#113

Post by alganet »

How can I be sure that spatial intelligence AIs will not be just intricate sensoring that ultimately fails to demonstrate actual intelligence?

> "trilobite"

The trilobite ancestor had a nervous system before it had an eye. It was able to make decisions and interact with the environment before the ability to see or speak a language.

It feels to me like this basic step is still missing. We haven't even crossed the first AI frontier yet.

Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]

#114

Post by calf »

jandrewrogers wrote: post #99
teemur wrote: post #42

Earlier quoted context omitted.

> We know that universal solutions can’t exist and that all practical solutions require exotic high-dimensionality computational constructs that human brains will struggle to reason about. This has been the status quo since the 1980s. This particular set of problems is hard for a reason. This made me a bit curious. Would you have any pointers to books/articles/search terms if one wanted to have a bit deeper look on t…

I'm not aware of any convenient literature but it is relatively obvious once someone explains it to you (as it was explained to me). At its root it is a cutting problem, like graph cutting but much more general because it includes things like non-trivial geometric types and relationships. Solving the cutting problem is necessary to efficiently shard/parallelize operations over the data models. For classic scalar data…

But then that sounds more like that person explained it wrong. They didn't explain why it is necessary to reduce to GRAPHCUT, it seems to me to beg the question. We should not assume this is true based on some vague anthropomorphic appeal to spatial locality, surely?

Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]

#115

Post by Barrin92 »

skwb wrote: post #3

It's hard to describe, but it's felt like LLMs have completely sucked the entire energy out of computer vision. Like... I know CVPR still happens and there's great research that comes out of it, but almost every single job posting in ML is about LLMs to do this and that to the detriment of computer vision.

>but almost every single job posting in ML is about LLMs

not in the defense sector, or aviation, or UAVS, automotive, etc. Any proper real-time vision task where you have to computationally interact with visual data is unsuited for LLMs.

Nobody controls a drone, missile or vehicle by taking a screenshot and sending it to ChatGPT and has it do math while it's on flight, anything that requires as the title of the thread says, spatial intelligence is unsuited for a language model

Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]

#116

Post by dopadelic »

calf wrote: post #112
dopadelic wrote: post #107

Earlier quoted context omitted.

You're pointing out a real class of hard problems — modeling sparse, nonlinear, spatiotemporal systems — but there’s a fundamental mischaracterization in lumping all transformer-based models under “LLMs” and using that to dismiss the possibility of spatial reasoning. Yes, classic LLMs (like GPT) operate as sequence predictors with no inductive bias for space, causality, or continuity. They're optimized for language f…

How do we prove a trained LLM has no inductive bias for space, causality, etc.? We can't assume this is true by construction, can we?

Why would we need to prove such a thing? Human vision has strong inductive biases, which is why you can perceive objects in abstract patterns. This is why you can lay down at a park and see a duck in a cloud. It's also why we can create abstracted representations of things with graphics. Having inductive biases makes it more relatable to the way we work.

And again, you're using the term LLMs again when vision based transformers in multimodal models aren't simply LLMs.

Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]

#117

Post by curiouscavalier »

ryeguy_24 wrote: post #111
jandrewrogers wrote: post #8

I appreciate the video and generally agree with Fei-Fei but I think it almost understates how different the problem of reasoning about the physical world actually is. Most dynamics of the physical world are sparse, non-linear systems at every level of resolution. Most ways of constructing accurate models mathematically don’t actually work. LLMs, for better or worse, are pretty classic (in an algorithmic information t…

Agree. Also, with respect to training, what is the goal that we are maximizing? LLMs are easy, predicting the next word and we have lots of training data. But what are we training for in real world? Modeling the next spatial photograph to predict things that will happen next? It’s not intuitive to me what that objective function would be in spatial intelligence.

Or that there is a sufficiently generalizable objective function for all “spatial intelligence.”

Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]

#118

Post by sota_pop »

czbond wrote: post #79

Thanks for joining the obvious Fei-Fei about 5 years late. Spatial web standards approved by IEEE that have been in the works for years. https://spatialwebfoundation.org/

Wow, I watched a presentation of this idea (2) years ago. It reads like classic flavor-of-the-week jargon-soup engineered to be catnip for unsuspecting VC; ie combining buzzwords HTTP, IoT, AI, blockchain, and the notion of “digital twin” from the AEC industry. Given by a guy who seemed extremely excited (and heavily energized - possibly chemically). The presenter tried to describe how this differs from HTTP. I’m highly confident no one in the room was able to make anything of it.

Before any questions could be asked, the presenter said “OK, I need to run to give this presentation at the World Economic Forum in Davos now.”, and quite literally ran out of the room.

Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]

#119

Post by epr »

jandrewrogers wrote: post #8

I appreciate the video and generally agree with Fei-Fei but I think it almost understates how different the problem of reasoning about the physical world actually is. Most dynamics of the physical world are sparse, non-linear systems at every level of resolution. Most ways of constructing accurate models mathematically don’t actually work. LLMs, for better or worse, are pretty classic (in an algorithmic information t…

Human beings get by quite well with extremely oversimplified (low resolution) abstractions. There is no need whatsoever for something even approaching universal or perfect. Humans aren't thinking about fundamental particles or solving differential equations in their head when they're driving a car or playing sports.

Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]

#120

Post by sega_sai »

lsy wrote: post #22
jandrewrogers wrote: post #8

I appreciate the video and generally agree with Fei-Fei but I think it almost understates how different the problem of reasoning about the physical world actually is. Most dynamics of the physical world are sparse, non-linear systems at every level of resolution. Most ways of constructing accurate models mathematically don’t actually work. LLMs, for better or worse, are pretty classic (in an algorithmic information t…

To make this more concrete: ImageNet enabled computer "vision" by providing images + labels, enabling the computer to take an image and spit out a label. LLM training sets enable text completion by providing text + completions, enabling the computer to take a piece of text and spit out its completion. Learning how the physical world works (not just kind of works a la videogames, actually works ) is not only about a j…

I feel that if words/phrases/whole texts can be embedded well in high dimensional spaces as points, the same must apply to the 3d world. I'm sure there will be embeddings of it (i.e. mapping the 3-d scene into a high-D vector) and then we'll be work with those embeddings as LLMs work with text (disclaimer: I am not an expert in the field ).
Post reply on HN