Meta AI Unleashes Megabyte, a Scalable Model Architecture
1–10 of 213 posts
Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture
#2Paper: https://arxiv.org/abs/2305.07185
Wow, seems like Meta AI is so ahead of the curve compared to even Google and OpenAI recently especially with their open sourcing pushes.
Great for the research community as a whole!
Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture
#3My main argument against the AI doomsayers has so far been that the current scaling laws simply make runaway singularity style scenarios algorithmically impossible (if for each step of improvement you need 10x parameters and 100x training, you quickly run into a brick wall).
This is part of why I’m not worried about the current crop of generative AI. I am however both curious and concerned about what the tsunami of talent and $$$ chasing the current trend will achieve.
If this n^(4/3) alt transformer compute scaling is real (and there’s been many a pretender, so it’s too early to tell), then that could fundamentally change the overall AI scaling law, substantially lowering the brick wall.
And that could be a game changer.
Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture
#4My main argument against the AI doomsayers has so far been that the current scaling laws simply make runaway singularity style scenarios algorithmically impossible (if for each step of improvement you need 10x parameters and 100x training, you quickly run into a brick wall). This is part of why I’m not worried about the current crop of generative AI. I am however both curious and concerned about what the tsunami of t…
I don't buy the idea (with either architecture) that "10x"-type scaling is required for another breakthrough.
Think of a human with below average intelligence. Then think of a human genius. Now consider how incredibly similar their brains are, despite the massive performance gap. It's not like one has 10x the number of neurons/synapses/connections etc. of the other. They're both healthy human brains, and you need powerful technology to even distinguish them structurally.
Considering this, it seems perfectly possible that a model like GPT-4 is just a hair's breadth away from vastly superhuman performance. It certainly beats the average human at many tasks already. The gap between a moron and a genius is a lot larger than that between GPT-4 and a superhuman.
Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture
#5Paper: https://arxiv.org/abs/2305.07185 Wow, seems like Meta AI is so ahead of the curve compared to even Google and OpenAI recently especially with their open sourcing pushes. Great for the research community as a whole!
great positioning for zuck, impressive
Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture
#6Paper: https://arxiv.org/abs/2305.07185 Wow, seems like Meta AI is so ahead of the curve compared to even Google and OpenAI recently especially with their open sourcing pushes. Great for the research community as a whole!
Commoditizing their complement?
Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture
#7Paper: https://arxiv.org/abs/2305.07185 Wow, seems like Meta AI is so ahead of the curve compared to even Google and OpenAI recently especially with their open sourcing pushes. Great for the research community as a whole!
Meta has no horse in the race (i.e. they don't have a search engine). So, they don't mind throwing random things out. Withholding it won't really make much of a difference for them, as they don't have a way to productionize the tech.
Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture
#8Megabyte sounds tiny.
Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture
#9My main argument against the AI doomsayers has so far been that the current scaling laws simply make runaway singularity style scenarios algorithmically impossible (if for each step of improvement you need 10x parameters and 100x training, you quickly run into a brick wall). This is part of why I’m not worried about the current crop of generative AI. I am however both curious and concerned about what the tsunami of t…
I don't buy the idea (with either architecture) that "10x"-type scaling is required for another breakthrough. Think of a human with below average intelligence. Then think of a human genius. Now consider how incredibly similar their brains are, despite the massive performance gap. It's not like one has 10x the number of neurons/synapses/connections etc. of the other. They're both healthy human brains, and you need pow…
> It certainly beats the average human at many tasks already
This is not the revolution you think it is.
```python > 338383*887282 ... ANSWER ... ```
Yet skynet never came.
Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture
#10Great paper. But wow, I really wish everyone would use more easily searchable names for their projects. In 6 months, there’s a high probability I’ll end up googling/ddging “megabyte model” trying to find this paper again.