Isn't it what Karpathy has been advocating for since the early days of Tesla Vision?
Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]
61–70 of 163 posts
Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]
#62Earlier quoted context omitted.
I think there's an implicit assumption here that interaction with the world is critical for effective learning. In that case, you're bottlenecked by the speed of the world... when learning with a single agent. One neat thing about artificial computational agents, in contrast to natural biological agents, is that they can share the same brain and share lived experience, so the "speed of reality" bottleneck is much les…
Yeah I'm envisioning putting a thousand simplistic robotic "infants" into a vast "playpen" to gather sensor data about their environment, for some (probably smaller) number of deep learning models to ingest the input and guess at output strategies (move this servo, rotate this camshaft this far in that direction, etc) and make predictions about resulting changes to input. In principle a thousand different deep learni…
But the playpen will contain objects that are inherently breakable. You cannot rough handle the glass vessel and have it too.
Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]
#63I appreciate the video and generally agree with Fei-Fei but I think it almost understates how different the problem of reasoning about the physical world actually is. Most dynamics of the physical world are sparse, non-linear systems at every level of resolution. Most ways of constructing accurate models mathematically don’t actually work. LLMs, for better or worse, are pretty classic (in an algorithmic information t…
> how do you represent arbitrary spatial relationships on computers in a general and scalable way?
Isn't this essentially what the convolutional layers do in LeNet?
Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]
#64I appreciate the video and generally agree with Fei-Fei but I think it almost understates how different the problem of reasoning about the physical world actually is. Most dynamics of the physical world are sparse, non-linear systems at every level of resolution. Most ways of constructing accurate models mathematically don’t actually work. LLMs, for better or worse, are pretty classic (in an algorithmic information t…
To make this more concrete: ImageNet enabled computer "vision" by providing images + labels, enabling the computer to take an image and spit out a label. LLM training sets enable text completion by providing text + completions, enabling the computer to take a piece of text and spit out its completion. Learning how the physical world works (not just kind of works a la videogames, actually works ) is not only about a j…
> the world itself, which cannot be compacted or fed into a computer at high speed.
…yet.
15 years ago LLMs as they are today seemed like science fiction too.
Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]
#65I appreciate the video and generally agree with Fei-Fei but I think it almost understates how different the problem of reasoning about the physical world actually is. Most dynamics of the physical world are sparse, non-linear systems at every level of resolution. Most ways of constructing accurate models mathematically don’t actually work. LLMs, for better or worse, are pretty classic (in an algorithmic information t…
[dead]
Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]
#66Intelligence is not only embodied (it needs a body), it is also embedded in the environment (it needs the environment). If you want an intelligence in your computer, you need an environment in your computer first, as the substrate from which the intelligence will evolve. The more accurate the environment the better the intelligence that will be obtained. The universe is able to create intelligence and we are proof. T…
The silicon exists in the same environment that our brains do.
Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]
#67An immaterial side note: funny how obsessed she seems to be with her age. She said once that people in the audience could be half or even third of her age. Given that she's 49, is it really typical that 16-year olds attend these fireside YC chats?
> An immaterial side note: funny how obsessed she seems to be with her age.
Given her intellectual stature, Professor Li likely was one of the strongest minds in any room she found herself in and, for the first half of her life, also one of the youngest voices.
Now that she’s entering mid-life, she’s still one of the most powerful minds, but no longer one of the youngest.
It’s something middle-aged thinkers can’t help but notice.
For the rest of us, we can only be grateful to share space and time with such gifted thinkers.
Coincidentally, today is Professor Li’s birthday! [0] I hope I will be around to see many more 3rds of July.
[0] Maybe her coming birthday was on her mind, hence the frequency of her remarks about her relative age.
Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]
#68I appreciate the video and generally agree with Fei-Fei but I think it almost understates how different the problem of reasoning about the physical world actually is. Most dynamics of the physical world are sparse, non-linear systems at every level of resolution. Most ways of constructing accurate models mathematically don’t actually work. LLMs, for better or worse, are pretty classic (in an algorithmic information t…
To make this more concrete: ImageNet enabled computer "vision" by providing images + labels, enabling the computer to take an image and spit out a label. LLM training sets enable text completion by providing text + completions, enabling the computer to take a piece of text and spit out its completion. Learning how the physical world works (not just kind of works a la videogames, actually works ) is not only about a j…
Well, you can use one or two cameras and a lidar, and use that to generate data to train a depth-map.
Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]
#69The next frontier is eliminating hallucinations?
Once that happens it’s all over.
Re: Fei-Fei Li: Spatial intelligence is the next frontier in AI [video]
#70Earlier quoted context omitted.
Happy to share an complete example privately, contact data is in my profile. Will add condensed version here in half an hour.
Condensing it for HN was harder than I thought because most of it makes only sense when you also see the images, so here is more like a summary of parts of the dialogue. Prompted by this comment https://news.ycombinator.com/item?id=44366753 I tried to geolocate the camera. I uploaded a screenshot from https://walzr.com/weather-watching to ChatGPT and it said a lot of things but concluded with “New York City street co…
I'd take text-to-image capabilities with a grain of salt, because they are dramatically lower than their text to text abilities. I don't know the exact mechanics with current multimodal models, but it is pretty clear that there is a disconnect between what the text modal wants, and what the text model outputs. It's almost feels like asking someone with a blindfold to draw a cat, you kinda get a mess.
If you ask chatgpt to describe a new image based off an input image, it will do dramatically better. But ask it to use it's image generation tooling and the "awareness" judged by the image output falls off a cliff.
Another example is infographics or flow charts. The models can easily output that information and put it in a nicely formatted text grid for you. But ask them to put it in a visual image, and it's just a mess. I don't think it's the models, I think it's the text-image translation layer.