281–290 of 1001 posts
Earlier quoted context omitted.
AGI is the Sisyphean task of our age. We’ll push this boulder up the mountain because we have to, even if it kills us.
Do we know LLMs are the path to AGI? If they're not, we'll just end up with some neat but eye wateringly expensive LLMs.
Earlier quoted context omitted.
Technically…? Does anyone here believe that the EU and Europe is the same thing? Would you find it weird if someone said that a Norwegian company was in Europe?
Many people certainly seem to! And it annoys me. I wasn’t talking about the EU, though. I was just commenting on the fact that in the UK, ‘Europe’ generally means ‘continental Europe’. > Would you find it weird if someone said that a Norwegian company was in Europe? I’d find it weird if a European did. But from Americans it’s to be expected.
One fundamental challenge to me is that if each training run because more and more expensive, the time it takes it to learn what works/doesn't work widens. Half a billion dollars for training a model is already nuts, but if it takes 100 iterations to perfect it, you've cumulatively spent 50 billion dollars... Smaller models may actually be where rapid innovation continues simply because of tighter feedback loops. O3…
But if the scaling law holds true, more dollars should at some point translate into AGI, which is priceless. We haven't reached the limits yet of that hypothesis.
This also isn't true. It'll clearly have a price to run. Even if it's very intelligent, if the price to run it is too high it'll just be a 24/7 intelligent person that few can afford to talk to. No?
Earlier quoted context omitted.
How many important problems are there where a 3 month head start on the data side is enough to win permanently and retain your advantage in the long run? I'm struggling to think of a scenario where "I have AGI in January and everyone else has it in April" is life-changing. It's a win, for sure, and it's an advantage, but success in business requires sustainable growth and manageable costs. If (random example) the bar…
Well that's the thought exercise. Is there something you can do with almost unlimited "brains" of roughly human capability but much faster, within a few days / weeks / months. Lets say you can instantiate 1 million agents, for 3 months, and each of them is roughly 100x faster than a human, that means you have the equivalent of 100 million human-brain-hours to dump into whatever you want, as long as your plans don't r…
What does it take to instantiate 1 million agents? Who has that kind of money and hardware? Would they still have it if they burn everything in the tank to be first?
Earlier quoted context omitted.
I think the wildest thing is actually Meta’s latest paper where they show a method for LLMs reasoning not in English, but in latent space https://arxiv.org/pdf/2412.06769 I’ve done research myself adjacent to this (mapping parts of a latent space onto a manifold), but this is a bit eerie, even to me.
Is it "eerie"? LeCun has been talking about it for some time, and may also be OpenAI's rumored q-star, mentioned shortly after Noam Brown (diplomacybot) joining OpenAI. You can't hill climb tokens, but you can climb manifolds.
I am hopeful that progress in mechanistic interpretability will serve as a healthy counterbalance to this approach when it comes to explainability.. though I kinda worry that at a certain point it may be that something resembling a scaling law puts an upper bound on even that.
Earlier quoted context omitted.
"There is no evidence that LLMs are the roadmap to AGI." - There's plenty of evidence. What do you think the last few years have been all about? Hell, GPT-4 would already have qualified as AGI about a decade ago.
Have you ever heard of a local maxima? You don't get an attack helicopter by breeding stronger and stronger falcons.
The default assumption should be that this is a local maximum, with evidence required to demonstrate that it's not. But the hype artists want us all to take the inevitability of LLMs for granted—"See the slope? Slopes lead up! All we have to do is climb the slope and we'll get to the moon! If you can't see that you're obviously stupid or have your head in the sand!"
Earlier quoted context omitted.
Have we really hit the wall? Do they use GPS based data? Feels like there’s data all around us. Sure they’ve hit the wall with obvious conversations and blog articles that humans produced, but data is a by product of our environment. Surely there’s more. Tons more.
We also could just measure the background noise of the universe and produce unlimited data. But just like GPS data it isn't suited for LLMs given that you know it has no relevance what so ever to language.
GPS data as it relates to location names, people, cultures, path finding.
Earlier quoted context omitted.
I think it would be many decades before I'd trust a robot like that around small children or pets. Robots with that kind of movement capability, as well as the ability it pick up and move things around, will be heavy enough that a small mistake could easily kill a small child or pet.
That's a solved problem for small devices. And we effectively have "robots" like that all over the place. Sliding doors in shops/trains/elevators have been around for ages and they include sensors for resistance. Unless there's 1. extreme cost cutting, or 2. bug in the hardware, devices like that wouldn't kill children these days.
Some of these are pretty crazy too.
Here's a video from 14 years ago where a table saw stops fast enough that it didn't scratch a hotdog: https://www.youtube.com/watch?v=fq3o0VGUh50
So even if this hypothetical robot had saws for hands it could be mostly safe (in theory).
Earlier quoted context omitted.
> What do you think the last few years have been all about? Next token language-based predictors with no more intelligence than brute force GIGO which parrot existing human intelligence captured as text/audio and fed in the form of input data. 4o agrees: "What you are describing is a language model or next-token predictor that operates solely as a computational system without inherent intelligence or understanding. T…
Everything you said is parroting data you’ve trained on, two thirds of it is actual copy paste
"Just like" an LLM, yeah sure...
Like how the brain was "just like" a hydraulic system (early industrial era), like a clockwork with gears and differentiation (mechanical engineering), "just like" an electric circuit (Edison's time), "just like" a computer CPU (21st century), and so on...
You're just assuming what you should prove