Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]

youtube.com

Post reply on HN
Print view View original

141–150 of 163 posts

Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]

#141

Post by awakeasleep »

m-s-y wrote: post #64
lsy wrote: post #22

Earlier quoted context omitted.

To make this more concrete: ImageNet enabled computer "vision" by providing images + labels, enabling the computer to take an image and spit out a label. LLM training sets enable text completion by providing text + completions, enabling the computer to take a piece of text and spit out its completion. Learning how the physical world works (not just kind of works a la videogames, actually works ) is not only about a j…

> the world itself, which cannot be compacted or fed into a computer at high speed. …yet. 15 years ago LLMs as they are today seemed like science fiction too.

Yes! It only requires a few fundamental breakthroughs in areas that seem constrained by physical reality.

Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]

#142

Post by jandrewrogers »

jedharris wrote: post #133
jandrewrogers wrote: post #99

Earlier quoted context omitted.

I'm not aware of any convenient literature but it is relatively obvious once someone explains it to you (as it was explained to me). At its root it is a cutting problem, like graph cutting but much more general because it includes things like non-trivial geometric types and relationships. Solving the cutting problem is necessary to efficiently shard/parallelize operations over the data models. For classic scalar data…

some pointers to the research program please?

It was a national security program with no public face. I was recruited into it because I solved a fundamental computer science problem they were deeply interested in. I did not get my extensive supercomputing experience in academia. It was a great experience if you just wanted to do hardcore computer science research, which at the time I did.

There are several VCs with knowledge of the program. It is obscure but has cred with people that know about it. I’ve raised millions of dollars off the back of my involvement.

A lot of really cool computer science research has happened inside the government. I think it is a bit less these days but people still underestimate it.

Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]

#143

Post by jandrewrogers »

tehjoker wrote: post #128
jandrewrogers wrote: post #99

Earlier quoted context omitted.

I'm not aware of any convenient literature but it is relatively obvious once someone explains it to you (as it was explained to me). At its root it is a cutting problem, like graph cutting but much more general because it includes things like non-trivial geometric types and relationships. Solving the cutting problem is necessary to efficiently shard/parallelize operations over the data models. For classic scalar data…

Did that research program have a public code name?

Not that I know of. If I drop the program director’s name, people that know, know. That is all the handshake you usually need.

Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]

#144

Post by jandrewrogers »

calf wrote: post #47
machinelearning wrote: post #13

Earlier quoted context omitted.

"Most ways of constructing accurate models mathematically don’t actually work" > This is true for almost anything at the limit, we are already able to model spatiotemporal dynamics to some useful degree (see: progress in VLAs, video diffusion, 4D Gaussians) "We’ve known for well over a decade that you cannot cram real-world spatial dynamics into those models. It is a clear impedance mismatch" > What's the source that…

Above commenter also asserts "highly developed research but no public literature" shrug ...

It was a national security program that plenty of people are familiar with and has been used across several countries. None of those programs publish.

As much as the literature doesn’t exist, the tech has been used in production for over a decade. That’s just my word of course but a lot of people know. :shrug:

Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]

#145

Post by sho »

tehjoker wrote: post #128
jandrewrogers wrote: post #99

Earlier quoted context omitted.

I'm not aware of any convenient literature but it is relatively obvious once someone explains it to you (as it was explained to me). At its root it is a cutting problem, like graph cutting but much more general because it includes things like non-trivial geometric types and relationships. Solving the cutting problem is necessary to efficiently shard/parallelize operations over the data models. For classic scalar data…

Did that research program have a public code name?

Sounds like Genoa/Topsail

Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]

#146

Post by kadushka »

ryeguy_24 wrote: post #111
jandrewrogers wrote: post #8

I appreciate the video and generally agree with Fei-Fei but I think it almost understates how different the problem of reasoning about the physical world actually is. Most dynamics of the physical world are sparse, non-linear systems at every level of resolution. Most ways of constructing accurate models mathematically don’t actually work. LLMs, for better or worse, are pretty classic (in an algorithmic information t…

Agree. Also, with respect to training, what is the goal that we are maximizing? LLMs are easy, predicting the next word and we have lots of training data. But what are we training for in real world? Modeling the next spatial photograph to predict things that will happen next? It’s not intuitive to me what that objective function would be in spatial intelligence.

Why wouldn’t predicting the next frame in a video stream be as effective as predicting the next word?

Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]

#147

Post by sabman »

fnands wrote: post #53
sabman wrote: post #50

Earlier quoted context omitted.

Thanks! This live demo uses metadata and stats only. Right now we are testing ViTs and Foundation Models as well. But quality of results from EO FMs haven't been worth the inference cost so far. Early days though. Also starting to fine tune models for specific downstream tasks ourselves.

Cool, makes sense. Yeah, have you considered maybe looking into just running it on embeddings [1], instead of the imagery itself? Would save on most of the inference cost, at the cost of flexibility (i.e. you are locked into whatever embeddings have been created). [1] https://developers.google.com/earth-engine/datasets/catalog/...

ah yes we have been testing other embedding models but not google's. I'll try this too. Its interesting most of them are doing land cover classes which is kinda solved already. We are also testing mixing agenic workflows with smaller directed prompts for users to provide the classes. Incidentally we are Berlin based. We should grab a coffee :)

Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]

#148

Post by leftcenterright »

She says "there is no language in nature" which does not seem accurate. Even though she might mean something else or a particular form of language but even then, bees and birds still use sound and something similar to language.

Is it just me?

for e.g. the form of communication used by bees is very well known now, it involves not just spatial movements but also "buzzing" which is totally similar tot he sounds we make, they just lack vocal cords.

https://www.noemamag.com/how-to-speak-honeybee/

Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]

#149

Post by MITSardine »

jgord wrote: post #139
MITSardine wrote: post #56

Earlier quoted context omitted.

Have you tried "traditional" approaches like a Delaunay triangulation on the point cloud, and how does your method compare to that? Or did you encounter difficulties with that? Regarding what you say of planes and compression, you can look into metric-based surface remeshing. Essentially, you estimate surface curvature (second derivatives) and use that to distort length computations, remeshing your surface to length…

short answer yes .. I tried a _lot_ of approaches, many worked partially. I think I linked to a YT video screencast showing edges of planes that my algo had detected in a sample pointcloud ? Efficient re-meshings are important, and its worth improving on the current algorithms to get crisper breaklines etc, but you really want to go a step further and do what humans do manually now when they make a CAD model from a p…

I'm not sure I'm sold on the necessity to detect lines and planes specifically. My issue is, suppose you could do that perfectly, then what of the rest of the geometry? If you're aiming for a CAD model in the end (BREP), you'll want to fit the whole thing, not only the planes and lines. And it seems to me an approach specialized for lines and planes is helpless at fitting general surfaces and curves. In my mind, a general approach that incidentally also finds straight lines and planes would be better (necessary).

Note if you can fit a BREP, it's fairly trivial to find whether a curve is close enough to a straight line that you can just stipulate it's a straight line (same for a plane).

Have you looked into NURBS fitting through point clouds? I understand those can be noisy and over sampled. A colleague got away with sorting point clouds by a Hilbert curve (or other space filling curve) and then keeping 1/N points (just by index), a simple but elegant way to remove N-1 every N points while keeping the general distribution mostly intact (you could also use an octree). Though I recall in some cases the distribution of points was not uniformly too dense, but e.g. dense along scanning lines and sparse between those lines.

Once it's tractable to triangulate the point cloud, you have two important pieces of information at your disposal: local connectivity (prior to that, it'd have been nlog(n) at best to find nearby points) and a notion of topology after some basic processing (e.g. detecting ridges to make out surface patches). With the former, you could do things like smooth the surface to do away with noisiness (say your points are randomly a small distance away from the plane, but any shape really) for better NURBS fitting, estimating normals, etc. and with the latter you could split the domain into faces and curves for your BREP.

At the very least, you'd get much cleaner data to feed an ML algo than basic point clouds. I just find it strange to tackle the raw data head on when there's so many methods for dealing with geometry already, at the very least to clean things up and make some sense of the data.

Have you looked at existing products that do point cloud -> CAD? What are they lacking in?

Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]

#150

Post by smohare »

m-s-y wrote: post #64
lsy wrote: post #22

Earlier quoted context omitted.

To make this more concrete: ImageNet enabled computer "vision" by providing images + labels, enabling the computer to take an image and spit out a label. LLM training sets enable text completion by providing text + completions, enabling the computer to take a piece of text and spit out its completion. Learning how the physical world works (not just kind of works a la videogames, actually works ) is not only about a j…

> the world itself, which cannot be compacted or fed into a computer at high speed. …yet. 15 years ago LLMs as they are today seemed like science fiction too.

[dead]
Post reply on HN