Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Meta AI Unleashes Megabyte, a Scalable Model Architecture

artisana.ai

Post reply on HN
Print view View original

201–210 of 213 posts

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#201

Post by dr_dshiv »

shreyshnaccount wrote: post #165
dr_dshiv wrote: post #125

Earlier quoted context omitted.

Have some examples? What would be tests that, if passed, you’d say “oh yeah, that’s AGI.” For instance, if it could make a peanut butter and jelly sandwich? Most challenging things that are easy for us are in the motor domain. While important, I think “intellectual AGI” is a meaningful milestone and closest to what most people think of when they think AGI.

The problem with defining AGI isn't only in defining intelligence, but also defining general. Also, why do we treat AGI as a yes/no question, when it probably makes sense to think partially... i don't have a definition of either

And that’s why I argue that AGI is already here. It is a spectrum. And we are well along the way to further acceleration. And if you want to say “no, we are not at AGI yet”, we need to define a clear test of what would be beyond that point.

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#202

Post by dr_dshiv »

TimPC wrote: post #157
dr_dshiv wrote: post #82

Earlier quoted context omitted.

Yes. Let’s define AGI as ability for a single model to pass most human professional tests (no cheating) and to provide genuine human-level flexible cognitive benefit to specialized professionals in diverse fields. Reasonable?

This definition fails badly because it doesn't test anything outside of language. At a bear minimum have the tests involved have pictures and descriptions in them and require the AI to use the same model to synthesize information from both.

Lots of human tests involve pictures. Why does the definition fail badly?

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#203

Post by mirekrusin »

DrScientist wrote: post #107
mirekrusin wrote: post #60

Earlier quoted context omitted.

Nobody said that nature is optimal. Wheels are trivial, however not present in biology. Nature creates tentacles, not jet engines, nuclear energy etc. Majority of human brain computation is spent on things that are simply not necessary for computer models (how to wiggle limbs, mouth, eyes etc). Current LLM are impressive, but we know they can be much more efficient - we're using very low quality training data, we don…

> Wheels are trivial, however not present in biology. Because roads don't exist in nature..... imagine trying to out run a predator if you just had wheels but no roads. > Nature creates tentacles, not jet engines, nuclear energy etc Under water I think you'll find there is jet propulsion. Plants and animals are powered by nuclear energy - the remote fusion reaction is in the sky. Why have an internal nuclear reactor…

You're right and missing main point at the same time.

The reasons why biology didn't evolve what we discovered don't matter.

What matters is that it didn't, yet they can exist and outperform.

Similarly brain is not an upper ceiling for intelligence.

We can create more intelligent machines than us.

Things like energy efficiency don't matter as much - when we run large models, nobody cares that it may require more energy than two sandwiches and a beer per day. We do care about energy efficiency but not at scales of biology.

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#204

Post by TeMPOraL »

shreyshnaccount wrote: post #163
TeMPOraL wrote: post #134

Earlier quoted context omitted.

> You're comparing memory with intelligence. It's a part of it sure, but there's also cognition, reasoning and i don't even know what more Yes, and LLMs show all of the specific things you've listed.

Ah, here is our disagreement. I see what you mean but I'm not totally convinced by that.. reasoning isn't just being able to verbalize the steps you take, which llms can do but more so the steps themselves, and in my experience llms can fail in that department. Maybe they are somewhere between intelligent and not intelligent? In my opinion that's far more likely, as most things are not usually binaries but spectrums…

To be clear, I'm not claiming LLMs are already better than humans in general sense, across many domains of cognition. I do believe however that, in the aspects you mention, SOTA LLMs (primarily GPT-4) fairly close to humans on the spectrum - they're "playing the same game", so to speak, in the sense in which prior computer systems, and most life forms on Earth, are not.

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#205

Post by TeMPOraL »

pixl97 wrote: post #159
TeMPOraL wrote: post #88

Earlier quoted context omitted.

> The current LLMs are basically unfiltered raw thoughts that must be continuously refined. A similar thing happens in our brains and only a little bit of that is accessible to our consciousness Exactly. But, AFAIK, it's also the part that does the bulk of actual thinking and decision-making for us. In that sense, LLMs may be closer to AGI than people expect, because they seem to be capturing the actual core of intel…

This is why we typically see better performance out of GPT when plugins are bolted in an chain|tree of though with reflection. The output of LLMs is kind of like our stream of consciousness, there's a lot of things I think, then discount after internally reflecting on the thought which the often leads to a more correct solution. Having an LLM 'think' like this natively would massively increase the necessary the amoun…

Yup. I totally expect you'll be able to eke out significant performance boost if you chain up the LLMs, so that e.g. you feed the initial query to a first-stage GPT-4 several times (likely in parallel), feed those to some kind of filter models that pass or reject the output, looping until you have, say, 3 passing outputs, then feed that to a summarizer, etc. Maybe play with generating system prompts so that you have multiple entirely different takes on the same query, or stage it. Or, you know, have GPT-4 look at the query and propose a graph of subsequent invocations for you.

I wish I had time to play with it some more right now. The pace of progress in the field is giving me a serious case of FOMO.

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#206

Post by TeMPOraL »

mirekrusin wrote: post #200
pixl97 wrote: post #151

Earlier quoted context omitted.

The counter to this is you can end up with an exceptionally powerful, but unaligned AI, which presents a new series of 'known unknowns' and 'unknown unknowns' that we have to deal with.

Yes, possibly. Simple example would be an army robot that is extermely efficient human killer that upps-escaped. Original argument was around optimising on intelligence and that biology doesn't hold best-possible trophy on it. We don't need to match number of neural connections in human brain to exceed its intelligence.

> We don't need to match number of neural connections in human brain to exceed its intelligence.

That's also true because of wheels/road thing, in that we can "cheat" here too. More specifically, some of the neural connections in the human brain are dedicated to sensing, processing and controlling the dynamic state of human body. Purely-software AIs don't need those for intelligence.

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#207

Post by nemothekid »

dr_dshiv wrote: post #125
shreyshnaccount wrote: post #122

Earlier quoted context omitted.

No, because most tests designed for humans test memory and pattern recognition, which computers already can do better than humans so it's not a useful comparison. I'd rather define it to be superhuman AGI when it not only performs better on tests with humans who can use computers during the test but also can perform everyday tasks which are not 'hard' for us humans. That is because we have the hardware in our brains…

Have some examples? What would be tests that, if passed, you’d say “oh yeah, that’s AGI.” For instance, if it could make a peanut butter and jelly sandwich? Most challenging things that are easy for us are in the motor domain. While important, I think “intellectual AGI” is a meaningful milestone and closest to what most people think of when they think AGI.

>Have some examples? What would be tests that, if passed, you’d say “oh yeah, that’s AGI.”

The "G" in AGI is general. A computer program or system that could both, lets say write code and learn drive a car would be something closer to an AGI. Written tests are remarkably brittle in showing how intelligent someone is - like we already know the limitations of tests in the real world! Einstein famously flunked his entrance exam, but then invented general relativity at 26; but other posters would have you believe an LLM is more intelligent than Einstein because it could pass a test. When an LLM defends a dissertation in arguably any field that would be way more impressive than an LLM passing a test that humans already designed and know the answers to.

One problem I have with saying a 100x more powerful LLM could become AGI is that there is nothing that leads me to believe that LLMs, as they exist currently, are capable of synthesizing new knowledge and I'm not sure what breakthroughs you would need to get there. Once you start to think about that, you start to run up on the limitations of the LLM. If I were to invent a completely new programming language, I could probably teach a junior engineer how to use it in a week, but the jury is out if I would need to first generate 50,000 sample program and spend $1,000,000 in gpu compute to get an LLM to output the same thing. It's hard to consider such a system AGI. Further still, sure you have Google spending millions of FSD, but it's hard to consider the system they have as general. Could I take Waymo and have it pilot a forklift? Or a submarine? How much would that cost? A """below average intelligence""" human could learn to use a forklift in an afternoon.

All in all, there's more to intelligence than written tests.

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#208

Post by nemothekid »

dr_dshiv wrote: post #89
ithkuil wrote: post #46

Earlier quoted context omitted.

> it's not a matter of having a 100x more powerful LLM, I think we all can agree that even the best LLM currently is not AGI. That's not what being disputed here I think. However a 100x more powerful LLM is not just 100x better at recall. A 100x more powerful LLM is not just 100x better at being stupid hallucinatory parrot. A model that is just 100x bigger is not necessarily 100x more powerful if you define power is…

> I think we all can agree that even the best LLM currently is not AGI. Disagree, for the record. If I’d described the capabilities of contemporary AI to 100 AI scientists 5 years ago, I bet more than half would agree to call that AGI. Further, more than 90% would assume that these capabilities were decades and decades away.

>If I’d described the capabilities of contemporary AI to 100 AI scientists 5 years ago

This is hard to believe, the all you need is attention paper was 6 years ago, GPT1 is 5 years old and GPT3 is 3 years old. The current crop of LLMs wasn't something that happened overnight.

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#209

Post by abwizz »

qas123 wrote: post #72
abwizz wrote: post #67

Earlier quoted context omitted.

would not be surprised if they integrated a "suggested conversations" feature based on your chat history and behaviour, where the user just picks sentences from a list and both parties can enjoy a effortless "organic" conversation.

My question then would be how it impacts advertising. we will essentially have bots talking to bots

i think it'll go so smooth that nobody dares to speak up

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#210

Post by dr_dshiv »

nemothekid wrote: post #208
dr_dshiv wrote: post #89

Earlier quoted context omitted.

> I think we all can agree that even the best LLM currently is not AGI. Disagree, for the record. If I’d described the capabilities of contemporary AI to 100 AI scientists 5 years ago, I bet more than half would agree to call that AGI. Further, more than 90% would assume that these capabilities were decades and decades away.

> If I’d described the capabilities of contemporary AI to 100 AI scientists 5 years ago This is hard to believe, the all you need is attention paper was 6 years ago, GPT1 is 5 years old and GPT3 is 3 years old. The current crop of LLMs wasn't something that happened overnight.

No one thought GPT1 would have these scaling effects. Really.
Post reply on HN