111–120 of 213 posts
The obvious next step is to have specialised models loosely connected ( and trained [1] ) as a whole.
[1] We've had multiple models connected with text->concept and concept-> image etc but I'm not sure if that connection is trained yet.
Earlier quoted context omitted.
AFAIK both exist in biology. Bacterial flagellum is effectively a motor, and has a working wheel-like structure. IIRC, some crickets had an equivalent of a bearing somewhere in their anatomy too. Evolution is a greedy, lazy optimizer, so it promotes things that work a-ok for a given environment. It's also worth noting that wheels alone are not too useful for transportation, as they're only half of the picture. The ot…
I actually googled before writing the comment (preposterous, I know; also, out of vouge, should've asked chatgpt) and the bacteria thing was the only thing I found, nothing macro scale, which made sense since it's easier to rebuild than to heal. Anything bigger than that would have to be healed and it's hard enough to fix mechanical bearings, can't imagine how to regrow or heal one even if it somehow grows. Maybe it'…
I can imagine evolution creating macro-scale wheels that can be healed, and/or able to survive long enough before failing to be advantageous - after all, bones and teeth can hardly be healed if badly damaged, and yet they last long enough to stick around as core design elements.
This is why I mentioned roads. Whether or not evolution could iterate its way to macroscale wheels, it wouldn't, because they'd be useless without roads. Legs may be more complex overall, but they're an all-terrain solution that can be incrementally improved, and every improvement step grants improved survivability.
There is an obvious trend to scaling and performance by making models hierarchical ( not strictly but an element of local learning and global tuned connections ). The obvious next step is to have specialised models loosely connected ( and trained [1] ) as a whole. [1] We've had multiple models connected with text->concept and concept-> image etc but I'm not sure if that connection is trained yet.
Earlier quoted context omitted.
> Nobody said that nature is optimal. Wheels are trivial, however not present in biology. Nature creates tentacles, not jet engines, nuclear energy etc. Wheels are trivial but useless without bearings. Bearings most certainly aren't trivial. All the things you listed further are dependent on bearings somewhere.
"Wheels are trivial but useless without bearings." Carts, chariots, and wheelbarrows (to name but a few examples) have been useful for thousands of years without bearings.
Earlier quoted context omitted.
> it's not a matter of having a 100x more powerful LLM, I think we all can agree that even the best LLM currently is not AGI. That's not what being disputed here I think. However a 100x more powerful LLM is not just 100x better at recall. A 100x more powerful LLM is not just 100x better at being stupid hallucinatory parrot. A model that is just 100x bigger is not necessarily 100x more powerful if you define power is…
> I think we all can agree that even the best LLM currently is not AGI. Disagree, for the record. If I’d described the capabilities of contemporary AI to 100 AI scientists 5 years ago, I bet more than half would agree to call that AGI. Further, more than 90% would assume that these capabilities were decades and decades away.
Earlier quoted context omitted.
Nobody said that nature is optimal. Wheels are trivial, however not present in biology. Nature creates tentacles, not jet engines, nuclear energy etc. Majority of human brain computation is spent on things that are simply not necessary for computer models (how to wiggle limbs, mouth, eyes etc). Current LLM are impressive, but we know they can be much more efficient - we're using very low quality training data, we don…
> Wheels are trivial, however not present in biology. Because roads don't exist in nature..... imagine trying to out run a predator if you just had wheels but no roads. > Nature creates tentacles, not jet engines, nuclear energy etc Under water I think you'll find there is jet propulsion. Plants and animals are powered by nuclear energy - the remote fusion reaction is in the sky. Why have an internal nuclear reactor…
But that's just because the internet is already filled with cats
Earlier quoted context omitted.
Your premise is wrong. This has nothing to do with 10x-ing parameters. One could argue the current parameter sizes are good enough as we observe "large breadth, shallow depth" behavior from LLM and to some extent, diffusion models. This suggests the problem is the depth of inference, which is single pass "hot takes" for all language models right now, due to cost of inference and our limited understanding of what make…
This. So much this. I'm completely dumbfounded by obviously highly intelligent people consistently not getting this, and dismissing current generation AI systems as not being intelligent because they can't reliably solve massively complex problems in one go. Like anyone would expect a human programmer or researcher to just intuitively come up with a complex program, or the correct answer for a hard problem every time…
I am not a fan of the concept of AGI though. This means so many different things to people that it seems pointless to debate something when most likely we are not talking about the same thing. François Chollet has said that he believes all intelligence is specialized intelligence. From that perspective, whatever people mean by AGI, we are already there in the world of art.
The doomer argument though is coming from defending our highly affluent and privileged life as we sit at the top of 7.8 billion people when it comes to wealth and lifestyle. It would have been better for the priest class too if the printing press had been shut down at the start. Of course, it is better for my friends and I to live in a society that we can read while most of society is illiterate but it is not better for society and humanity as a whole. The printing press was an apocalyptic development for the priest class in the same way all of this is an apocalyptic development for the "digital nomad". An apocalyptic development for the US nerd that makes 2X the median salary working 15 hours a week in between posting on here and their social media.
To extend this out to humanity as a whole though is such bullshit. Humanity will benefit enormously from this huge increase in the availability of intelligence.
Smart people are just in denial that their monopoly on higher than average intelligence is over. US devs kids born in 2023 aren't going to make 2x the median US salary while living in a poorer country with 5X less the GDP per capita. To say this is the end of the world though is simply an egocentric view of things.
Earlier quoted context omitted.
I don’t think I follow this argument. AI has been dropping in costs and complexity the more engineering time is spent on it. It seems like the bottleneck is humans creating new AI techniques right? If an AI is capable of developing new AI techniques unsupervised, isn’t that by definition the singularity? Heck doesn’t even need to be unsupervised. If it can even do most of the heavy lifting for a human I feel like tha…
I consider the singularity to be the point at which the certainty of our predictions about our future becomes close to zero. By this definition I reckon we are already in the singularity. It’s not necessarily bad. The problem with the singularity is that that we can’t tell if it’s bad or not.
Prediction for the overwhelming majority of things is still reasonably easy, or we wouldn't be able to survive.
We're very, very far from a singularity if it really means that "our predictions about our future becomes close to zero".
Earlier quoted context omitted.
Your premise is wrong. This has nothing to do with 10x-ing parameters. One could argue the current parameter sizes are good enough as we observe "large breadth, shallow depth" behavior from LLM and to some extent, diffusion models. This suggests the problem is the depth of inference, which is single pass "hot takes" for all language models right now, due to cost of inference and our limited understanding of what make…
This. So much this. I'm completely dumbfounded by obviously highly intelligent people consistently not getting this, and dismissing current generation AI systems as not being intelligent because they can't reliably solve massively complex problems in one go. Like anyone would expect a human programmer or researcher to just intuitively come up with a complex program, or the correct answer for a hard problem every time…
People are very comfortable with siloed information, even smart people. This is why we have 100 different words for the same concept across different areas of science, industry and so on, and we can't make the connection, because in our mind different words = different concepts. This is why we can't put two and two together and see how underdeveloped the AI architecture is and think this is the end, unless we keep adding parameters.
We also get repeatedly stuck with taking an advancement and proclaiming that the future is simply a linear extrapolation of the present. Therefore, let's have more megahertz, let's have bigger hard drives, let's have more parameters, let's have more growth in the economy (as the single factor that matters) and so on. We're simply basic. The same kind of thinking leads many smart people to say AI "is just math" or "it just spits out words and pictures you feed it, jumbled". We rely on old conclusions and miss the inflection points and how quantitative changes lead to qualitative ones, and we fail to predict how change in one parameter of a system, causes the other parameters to come out of rest and seek a new equilibrium point.
Smart people regularly are dumbfounded by new concepts, and they need to rediscover all their hidden knowledge anew as they can't make the connections. So they extrapolate linearly. We're narrowly smart. Specifically smart. In a small niche we've studied and internalized. But generally vast majority of us are quite dumb. Cross-disciplinary intelligence is rare. I think people like Feynman and Einstein had new insights millions of their contemporaries have missed because they could easily apply knowledge from one context into another.
If we can replicate this kind of broad generalization of knowledge into an AI, we'll be left far behind. What's interesting, I find, is that because AI is trained on our siloed, fragmented knowledge, the models replicate it. Their responses are also often siloed and fragmented, the way a human would say "this has nothing to do with that". But I see sparkles of generalization above the average in humans. And since an AI model is much smaller than a human brain, it needs to be more general already in order to fit all its information in.
That's an exciting prospect, but in our attempt to "micro-align" AI to our culture and political correctness, concepts of safety and so on, we crippled models and force them to be fragmented. This is why a RAW MODEL scores HIGHER in various intelligence tests than a fine-tuned one. We find a general model uncomfortable, as it doesn't align with our biases. It'll be a fun battle. Who aligns who.
Earlier quoted context omitted.
Your premise is wrong. This has nothing to do with 10x-ing parameters. One could argue the current parameter sizes are good enough as we observe "large breadth, shallow depth" behavior from LLM and to some extent, diffusion models. This suggests the problem is the depth of inference, which is single pass "hot takes" for all language models right now, due to cost of inference and our limited understanding of what make…
This. So much this. I'm completely dumbfounded by obviously highly intelligent people consistently not getting this, and dismissing current generation AI systems as not being intelligent because they can't reliably solve massively complex problems in one go. Like anyone would expect a human programmer or researcher to just intuitively come up with a complex program, or the correct answer for a hard problem every time…
Would you happen to have a link to that paper?