Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Meta AI Unleashes Megabyte, a Scalable Model Architecture

artisana.ai

Post reply on HN
Print view View original

191–200 of 213 posts

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#191

Post by twobitshifter »

DrScientist wrote: post #103
p-e-w wrote: post #4

Earlier quoted context omitted.

I don't buy the idea (with either architecture) that "10x"-type scaling is required for another breakthrough. Think of a human with below average intelligence. Then think of a human genius. Now consider how incredibly similar their brains are, despite the massive performance gap. It's not like one has 10x the number of neurons/synapses/connections etc. of the other. They're both healthy human brains, and you need pow…

> Considering this, it seems perfectly possible that a model like GPT-4 is just a hair's breadth away from vastly superhuman performance. Except that structurally the brain is clearly has vastly more capacity than the GPT-4 model. So sure one brain doesn't look that much different to the other - and it's in the details of the learning, wiring. But the brain, looks vastly different from a GPT-4 model in terms of capac…

I’ve heard that the human brain may have more connections but with AI we’ve made something that is more efficient. If we make it a hardware issue, if provided with the brains’ capacity, what would AI look like?

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#192

Post by lazzlazzlazz »

jacobn wrote: post #3

My main argument against the AI doomsayers has so far been that the current scaling laws simply make runaway singularity style scenarios algorithmically impossible (if for each step of improvement you need 10x parameters and 100x training, you quickly run into a brick wall). This is part of why I’m not worried about the current crop of generative AI. I am however both curious and concerned about what the tsunami of t…

10x scaling won't take long, even putting aside improvements in our understanding of training processes.

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#193

Post by lostmsu »

DrScientist wrote: post #133
lostmsu wrote: post #129

Earlier quoted context omitted.

> brain is clearly has vastly more capacity I would not be so sure about "clearly" bit. Brain has 100B neurons x 1000 connections. GPT3 has 175B connections, but to implement operations used in those connections Nvidia H100 uses about 5M transistors, which in human brain would have to be copied to each connection because nature didn't invent software. (assumes one needs one full CUDA core to implement necessary ops,…

Your argument only makes sense if you consider GPT3 equivalent to the human brain - yep the brain has more neurons - but it's also doing a heck of a lot more. Also each neuron can hold much more state than a transistor. For example they can respond to the timing of incoming events without having to build that capability with recurrent connections etc. In addition the neural connections themselves have properties. > b…

> but it's also doing a heck of a lot more

What are you referring to?

> Also each neuron can hold much more state than a transistor.

But can it hold much more state than 8 transistors? 16 transistors? 32 transistors?

> For example they can respond to the timing of incoming events without having to build that capability with recurrent connections etc.

But they can't fire 3B times a second. Transistors though can accumulate a number and fire when it reaches a threshold much faster (although that is not really used in transformers). There are differences, but it is unclear how to quantify them. At least I don't immediately see how that feature is a definite advantage, and not simply an implementation detail.

> In addition the neural connections themselves have properties.

This is very technically correct, which is great because in this form it shows to be a non-argument. Individual transistors also have properties. Even molecules do. Which of those properties are essential for the learning process is the important bit, and it is quite possible that none are.

> Eh? How do you explain the fact that brains can learn and aren't a fixed input, output engine?

Eh? Brains learn by physically growing parts of neurons. That growing can not be moved around to another set of neurons. That property is essential to be deemed "software".

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#194

Post by PoignardAzur »

mkaic wrote: post #138

Ugh. I just spent the past few days reading this exact paper and preparing a detailed presentation on it for my job as an AI researcher, and headlines like this make me roll my eyes very hard. MEGABYTE is indeed a cool new architecture, but it is still very much just a proof of concept at the moment. The paper shows that the model can compete with (but not decimate) vanilla Transformers on the scale of ~1B parameters…

> The paper shows that the model can compete with (but not decimate) vanilla Transformers on the scale of ~1B parameters in long-sequence prediction tasks. They did not "unleash" anything, and scalability to very large parameter counts and datasets still has not been tested.

One thing to keep in mind in particular is that the vast majority of transformer alternatives/improvements/optimizations that initially showed promise ultimately ended up scaling less well than the baseline transformer architecture. The transformer has been weirdly hard to improve/dethrone.

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#195

Post by vlovich123 »

woah wrote: post #187
vlovich123 wrote: post #12

Earlier quoted context omitted.

I don’t think I follow this argument. AI has been dropping in costs and complexity the more engineering time is spent on it. It seems like the bottleneck is humans creating new AI techniques right? If an AI is capable of developing new AI techniques unsupervised, isn’t that by definition the singularity? Heck doesn’t even need to be unsupervised. If it can even do most of the heavy lifting for a human I feel like tha…

> If an AI is capable of developing new AI techniques unsupervised, isn’t that by definition the singularity? AI is not capable of developing new AI techniques unsupervised. Honestly the lowest hanging fruit should be the ability for LLMs to generate their own textual training data that leads to an improvement in model abilities, since that is what they are supposed to be good at. Until then, it's still garbage in ga…

> AI is not capable of developing new AI techniques unsupervised.

Today. Hence we're not in the singularity.

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#196

Post by mkaic »

PoignardAzur wrote: post #194
mkaic wrote: post #138

Ugh. I just spent the past few days reading this exact paper and preparing a detailed presentation on it for my job as an AI researcher, and headlines like this make me roll my eyes very hard. MEGABYTE is indeed a cool new architecture, but it is still very much just a proof of concept at the moment. The paper shows that the model can compete with (but not decimate) vanilla Transformers on the scale of ~1B parameters…

> The paper shows that the model can compete with (but not decimate) vanilla Transformers on the scale of ~1B parameters in long-sequence prediction tasks. They did not "unleash" anything, and scalability to very large parameter counts and datasets still has not been tested. One thing to keep in mind in particular is that the vast majority of transformer alternatives/improvements/optimizations that initially showed p…

Yeah, it's had remarkable staying power. I think it says a lot that I can see "Vaswani, et al." cited in a paper and know exactly what the author is referring to lol. I don't usually memorize researcher names but the original Attention Is All You Need paper is just freaking ubiquitous.

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#197

Post by mkaic »

riwsky wrote: post #169
nwoli wrote: post #142

Earlier quoted context omitted.

I wish you had a link to your blog or twitter in your bio, you have the kind of nuanced tone I’d like to hear more from

they should consider doing this for a living

I'll consider doing it for a living when it can pay my rent, which is unlikely to be any time soon :P

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#198

Post by mkaic »

lordofgibbons wrote: post #149
mkaic wrote: post #138

Ugh. I just spent the past few days reading this exact paper and preparing a detailed presentation on it for my job as an AI researcher, and headlines like this make me roll my eyes very hard. MEGABYTE is indeed a cool new architecture, but it is still very much just a proof of concept at the moment. The paper shows that the model can compete with (but not decimate) vanilla Transformers on the scale of ~1B parameters…

I'd love to watch a recording of this presentation or write up, if possible!

Hmm, I may make a blog post using some of the graphics I made for the presentation. Can't make any guarantees though.

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#199

Post by NumberWangMan »

digging wrote: post #180
laratied wrote: post #117

Earlier quoted context omitted.

I would add to your amazing list that we are really good at denial as a coping mechanism with change. I am not a fan of the concept of AGI though. This means so many different things to people that it seems pointless to debate something when most likely we are not talking about the same thing. François Chollet has said that he believes all intelligence is specialized intelligence. From that perspective, whatever peop…

> The doomer argument though is coming from defending our highly affluent and privileged life It's not at all about that. Even if "truly general" intelligence is impossible, that's irrelevant to the actual concerns about AI apocalypse. There are multiple theories about what failure looks like, but they essentially come down to a loss of control. Now, obviously, that means something different for the owner class and f…

Nitpick: I think we can comprehend an intelligence greater than our own, up to some point, but that's different from being able to predict its actions.

And we could contain an intelligence greater than our own, up to a point. But if there are a lot of incentives not to, because letting that intelligence act on the world gains the "handler" money/power, then once there's one, there will likely be many, many more.

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#200

Post by mirekrusin »

pixl97 wrote: post #151
mirekrusin wrote: post #132

Earlier quoted context omitted.

Exactly, same with intelligence - it's polluted with emotions and all kind of "nonsense" - but it doesn't have to. We can create emotionless, super-intelligent machines exceeding human capability by far (and use them as hammers). No need to imitate every detail of the brain to extract intelligence.

The counter to this is you can end up with an exceptionally powerful, but unaligned AI, which presents a new series of 'known unknowns' and 'unknown unknowns' that we have to deal with.

Yes, possibly. Simple example would be an army robot that is extermely efficient human killer that upps-escaped.

Original argument was around optimising on intelligence and that biology doesn't hold best-possible trophy on it.

We don't need to match number of neural connections in human brain to exceed its intelligence.

Post reply on HN