Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Meta AI Unleashes Megabyte, a Scalable Model Architecture

artisana.ai

Post reply on HN
Print view View original

11–20 of 213 posts

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#11

Post by nemothekid »

p-e-w wrote: post #4
jacobn wrote: post #3

My main argument against the AI doomsayers has so far been that the current scaling laws simply make runaway singularity style scenarios algorithmically impossible (if for each step of improvement you need 10x parameters and 100x training, you quickly run into a brick wall). This is part of why I’m not worried about the current crop of generative AI. I am however both curious and concerned about what the tsunami of t…

I don't buy the idea (with either architecture) that "10x"-type scaling is required for another breakthrough. Think of a human with below average intelligence. Then think of a human genius. Now consider how incredibly similar their brains are, despite the massive performance gap. It's not like one has 10x the number of neurons/synapses/connections etc. of the other. They're both healthy human brains, and you need pow…

>Think of a human with below average intelligence. Then think of a human genius.

LLMs are not AGI. A human with below average intelligence is still a league above a chimpanzee. A chimpanzee will never be able to read, not because "it's too dumb", but because a chimp's brain lacks the actual hardware for reading. The LLM is the chimpanzee. The gap between an LLM and a "human with below average intelligence" is far more than 10x.

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#12

Post by vlovich123 »

jacobn wrote: post #3

My main argument against the AI doomsayers has so far been that the current scaling laws simply make runaway singularity style scenarios algorithmically impossible (if for each step of improvement you need 10x parameters and 100x training, you quickly run into a brick wall). This is part of why I’m not worried about the current crop of generative AI. I am however both curious and concerned about what the tsunami of t…

I don’t think I follow this argument. AI has been dropping in costs and complexity the more engineering time is spent on it. It seems like the bottleneck is humans creating new AI techniques right? If an AI is capable of developing new AI techniques unsupervised, isn’t that by definition the singularity? Heck doesn’t even need to be unsupervised. If it can even do most of the heavy lifting for a human I feel like that would put us into runaway territory.

Granted we’re a long way away from that and likely we’d need an AI that could come up with its own hypotheses for new AI models to make this truly minimal cost and that feels even farther away. But I don’t think I follow the claim that each improvement step change requires to much massive extra scaling as it seems not match what we’ve seen over the past few years (granted I could be misinformed - I’m applying a 10k view spectator and maybe my own mental model here is flawed).

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#13

Post by epups »

nemothekid wrote: post #11
p-e-w wrote: post #4

Earlier quoted context omitted.

I don't buy the idea (with either architecture) that "10x"-type scaling is required for another breakthrough. Think of a human with below average intelligence. Then think of a human genius. Now consider how incredibly similar their brains are, despite the massive performance gap. It's not like one has 10x the number of neurons/synapses/connections etc. of the other. They're both healthy human brains, and you need pow…

> Think of a human with below average intelligence. Then think of a human genius. LLMs are not AGI. A human with below average intelligence is still a league above a chimpanzee. A chimpanzee will never be able to read, not because "it's too dumb", but because a chimp's brain lacks the actual hardware for reading. The LLM is the chimpanzee. The gap between an LLM and a "human with below average intelligence" is far mo…

By what criteria? In several professional capacities, including coding, LLM's are far superior to an average human already.

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#15

Post by p-e-w »

nemothekid wrote: post #11
p-e-w wrote: post #4

Earlier quoted context omitted.

I don't buy the idea (with either architecture) that "10x"-type scaling is required for another breakthrough. Think of a human with below average intelligence. Then think of a human genius. Now consider how incredibly similar their brains are, despite the massive performance gap. It's not like one has 10x the number of neurons/synapses/connections etc. of the other. They're both healthy human brains, and you need pow…

> Think of a human with below average intelligence. Then think of a human genius. LLMs are not AGI. A human with below average intelligence is still a league above a chimpanzee. A chimpanzee will never be able to read, not because "it's too dumb", but because a chimp's brain lacks the actual hardware for reading. The LLM is the chimpanzee. The gap between an LLM and a "human with below average intelligence" is far mo…

> The gap between an LLM and a "human with below average intelligence" is far more than 10x.

In which direction?

GPT-4 passes the bar exam with a top 10% score. How do you think a human with below average intelligence (or even with average intelligence) would fare?

Copilot generates programming code that solves problems, and in most cases the code is correct. It outperforms many junior professional developers. Do you think a human with below average intelligence could do that?

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#16

Post by slowmovintarget »

epups wrote: post #13
nemothekid wrote: post #11

Earlier quoted context omitted.

> Think of a human with below average intelligence. Then think of a human genius. LLMs are not AGI. A human with below average intelligence is still a league above a chimpanzee. A chimpanzee will never be able to read, not because "it's too dumb", but because a chimp's brain lacks the actual hardware for reading. The LLM is the chimpanzee. The gap between an LLM and a "human with below average intelligence" is far mo…

By what criteria? In several professional capacities, including coding, LLM's are far superior to an average human already.

No they're not. They're superior at specific tests when prompted.

But asking for a complete application nearly always fails, when an average human that knows programming should be able to make one that works.

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#17

Post by radq »

This reminds me of the Octuple MIDI tokenization scheme introduced in the MusicBERT paper (https://arxiv.org/pdf/2106.05630.pdf). Would be interesting to see how much of a performance difference results from using a smaller decoder model (in Megabyte) instead of just doing a softmax (a la Octuple).

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#18

Post by CamperBob2 »

slowmovintarget wrote: post #16
epups wrote: post #13

Earlier quoted context omitted.

By what criteria? In several professional capacities, including coding, LLM's are far superior to an average human already.

No they're not. They're superior at specific tests when prompted. But asking for a complete application nearly always fails, when an average human that knows programming should be able to make one that works.

But asking for a complete application nearly always fails, when an average human that knows programming should be able to make one that works.

"This talking dog is a dumbass. The risotto recipe he gave me sucked, and the C++ code he wrote is full of security holes. I don't see what all the hype is about."

Hint: ML will get better. Humans will not. ("Slow moving target," indeed.) What we're learning is just how many aspects of our vaunted 'intelligence' are really just features of our languages and the graphs they encode.

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#19

Post by doctor_eval »

vlovich123 wrote: post #12
jacobn wrote: post #3

My main argument against the AI doomsayers has so far been that the current scaling laws simply make runaway singularity style scenarios algorithmically impossible (if for each step of improvement you need 10x parameters and 100x training, you quickly run into a brick wall). This is part of why I’m not worried about the current crop of generative AI. I am however both curious and concerned about what the tsunami of t…

I don’t think I follow this argument. AI has been dropping in costs and complexity the more engineering time is spent on it. It seems like the bottleneck is humans creating new AI techniques right? If an AI is capable of developing new AI techniques unsupervised, isn’t that by definition the singularity? Heck doesn’t even need to be unsupervised. If it can even do most of the heavy lifting for a human I feel like tha…

I consider the singularity to be the point at which the certainty of our predictions about our future becomes close to zero. By this definition I reckon we are already in the singularity.

It’s not necessarily bad. The problem with the singularity is that that we can’t tell if it’s bad or not.

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#20

Post by visarga »

p-e-w wrote: post #4
jacobn wrote: post #3

My main argument against the AI doomsayers has so far been that the current scaling laws simply make runaway singularity style scenarios algorithmically impossible (if for each step of improvement you need 10x parameters and 100x training, you quickly run into a brick wall). This is part of why I’m not worried about the current crop of generative AI. I am however both curious and concerned about what the tsunami of t…

I don't buy the idea (with either architecture) that "10x"-type scaling is required for another breakthrough. Think of a human with below average intelligence. Then think of a human genius. Now consider how incredibly similar their brains are, despite the massive performance gap. It's not like one has 10x the number of neurons/synapses/connections etc. of the other. They're both healthy human brains, and you need pow…

> I don't buy the idea (with either architecture) that "10x"-type scaling is required for another breakthrough.

Scaling can happen in two dimensions - model size and dataset size. What counts is the product of n_examples x n_parameters. That's why we have the super-Chinchilla laws, where n_examples >> 20*n_parameters. Scale the data, keep the model lean. Not to mention dataset quality - if you got clean diverse data you need less of it.

Another important trend is using LLMs to generate training sets. Example: ConstitutionalAI (pure RLAIF), TinyStories (scaling down LLMs), Alpaca (borrowed RLHF), AlpacaFarm - a recent paper promising fine-tuned models for $200 cost in 24h.

Post reply on HN