Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Meta AI Unleashes Megabyte, a Scalable Model Architecture

artisana.ai

Post reply on HN
Print view View original

51–60 of 213 posts

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#51

Post by flangola7 »

shreyshnaccount wrote: post #47
p-e-w wrote: post #37

Earlier quoted context omitted.

> If AGI is a computer passing a test, then AGI was achieved a long time ago. AIs couldn't even pass a third-grade reading comprehension exam until about 5 years ago. Computers being able to pass tests designed for humans is a very new thing. > it's clear LLMs are not AGIs And the main argument for that is that "it's clear". They're beating lawyers, doctors, and software engineers, but obviously, that's not real inte…

You're assuming that everything needed to write code or make arguments is purely intelligence based and has nothing to do with patterns, structural repetition and things glorified autocomplete could do, and that's not true.

Who said anything about everything? AI could not write code AT ALL until very recently. We could invent a drug that kills 99% of cancers and the next day there would be people bemoaning that it isn't a "true" cure.

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#52

Post by hawk_ »

nemothekid wrote: post #34
p-e-w wrote: post #15

Earlier quoted context omitted.

> The gap between an LLM and a "human with below average intelligence" is far more than 10x. In which direction? GPT-4 passes the bar exam with a top 10% score. How do you think a human with below average intelligence (or even with average intelligence) would fare? Copilot generates programming code that solves problems, and in most cases the code is correct. It outperforms many junior professional developers. Do you…

> How do you think a human with below average intelligence (or even with average intelligence) would fare? I don't understand what rote memorization to pass a test has to do with intelligence. For the record Kim Kardashian passed the bar; I imagine anyone given the proper motivation and time to study could do it, it's not a hard test. If AGI is a computer passing a test, then AGI was achieved a long time ago. I don't…

> For the record Kim Kardashian passed the bar;

While I know the wider point you're trying to make, Kim Kardashian is a very smart business person. She may just not fit your narrow definition of "intelligent".

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#53

Post by flangola7 »

ithkuil wrote: post #46
nemothekid wrote: post #34

Earlier quoted context omitted.

> How do you think a human with below average intelligence (or even with average intelligence) would fare? I don't understand what rote memorization to pass a test has to do with intelligence. For the record Kim Kardashian passed the bar; I imagine anyone given the proper motivation and time to study could do it, it's not a hard test. If AGI is a computer passing a test, then AGI was achieved a long time ago. I don't…

> it's not a matter of having a 100x more powerful LLM, I think we all can agree that even the best LLM currently is not AGI. That's not what being disputed here I think. However a 100x more powerful LLM is not just 100x better at recall. A 100x more powerful LLM is not just 100x better at being stupid hallucinatory parrot. A model that is just 100x bigger is not necessarily 100x more powerful if you define power is…

Define grounding things in reality.

We only have our 5 senses to go off of. Meta has already put out one multimodal model incorporating multiple data types, openai is undoubtedly working on it too.

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#54

Post by ithkuil »

flangola7 wrote: post #53
ithkuil wrote: post #46

Earlier quoted context omitted.

> it's not a matter of having a 100x more powerful LLM, I think we all can agree that even the best LLM currently is not AGI. That's not what being disputed here I think. However a 100x more powerful LLM is not just 100x better at recall. A 100x more powerful LLM is not just 100x better at being stupid hallucinatory parrot. A model that is just 100x bigger is not necessarily 100x more powerful if you define power is…

Define grounding things in reality. We only have our 5 senses to go off of. Meta has already put out one multimodal model incorporating multiple data types, openai is undoubtedly working on it too.

Grounding in reality can be something as simple as what openai is experimenting with plugins or something much more integrated.

It's not a matter of which senses you have, but about being able to "continuously" use them.

The current LLMs are basically unfiltered raw thoughts that must be continuously refined. A similar thing happens in our brains and only a little bit of that is accessible to our consciousness

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#55

Post by raincole »

p-e-w wrote: post #4
jacobn wrote: post #3

My main argument against the AI doomsayers has so far been that the current scaling laws simply make runaway singularity style scenarios algorithmically impossible (if for each step of improvement you need 10x parameters and 100x training, you quickly run into a brick wall). This is part of why I’m not worried about the current crop of generative AI. I am however both curious and concerned about what the tsunami of t…

I don't buy the idea (with either architecture) that "10x"-type scaling is required for another breakthrough. Think of a human with below average intelligence. Then think of a human genius. Now consider how incredibly similar their brains are, despite the massive performance gap. It's not like one has 10x the number of neurons/synapses/connections etc. of the other. They're both healthy human brains, and you need pow…

> Now consider how incredibly similar their brains are, despite the massive performance gap.

A disk filled with random bits is "similar" to a disk that stores the whole Wikipedia's text content. So writing the whole Wikipedia is an effort of a hair's breadth...?

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#57

Post by chinaman425 »

joshxyz wrote: post #5
crakenzak wrote: post #2

Paper: https://arxiv.org/abs/2305.07185 Wow, seems like Meta AI is so ahead of the curve compared to even Google and OpenAI recently especially with their open sourcing pushes. Great for the research community as a whole!

great positioning for zuck, impressive

[dead]

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#58

Post by satellite2 »

p-e-w wrote: post #15
nemothekid wrote: post #11

Earlier quoted context omitted.

> Think of a human with below average intelligence. Then think of a human genius. LLMs are not AGI. A human with below average intelligence is still a league above a chimpanzee. A chimpanzee will never be able to read, not because "it's too dumb", but because a chimp's brain lacks the actual hardware for reading. The LLM is the chimpanzee. The gap between an LLM and a "human with below average intelligence" is far mo…

> The gap between an LLM and a "human with below average intelligence" is far more than 10x. In which direction? GPT-4 passes the bar exam with a top 10% score. How do you think a human with below average intelligence (or even with average intelligence) would fare? Copilot generates programming code that solves problems, and in most cases the code is correct. It outperforms many junior professional developers. Do you…

Maybe we are simply culturally inclined to believe that the bar exam is more sophisticated than say, learn how to drive. A lot of the compute power required to drive is simply hidden from us because it's inconsious. While intellectual tasks are mostly done consciously. So it remains to be shown which task is actually more complex. And in the meantime it's clear than most people, independantly of their academic background can learn how to drive relatively easily.

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#59

Post by [deleted] »

wudangmonk wrote: post #39
p-e-w wrote: post #4

Earlier quoted context omitted.

I don't buy the idea (with either architecture) that "10x"-type scaling is required for another breakthrough. Think of a human with below average intelligence. Then think of a human genius. Now consider how incredibly similar their brains are, despite the massive performance gap. It's not like one has 10x the number of neurons/synapses/connections etc. of the other. They're both healthy human brains, and you need pow…

You mean the average brain with about 100 billion neurons with about 1000 connections each bringing it to around 100 trillion connections. With an estimated 1000 "AI" neurons required per biologial neuron. I don't think you are givin these "below average" intelligence individuals enough credit. What we consider a genius is the equivalent of a dog show obstacle course. We measure intelligence/genius as whatever is har…

[deleted]

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#60

Post by mirekrusin »

wudangmonk wrote: post #39
p-e-w wrote: post #4

Earlier quoted context omitted.

I don't buy the idea (with either architecture) that "10x"-type scaling is required for another breakthrough. Think of a human with below average intelligence. Then think of a human genius. Now consider how incredibly similar their brains are, despite the massive performance gap. It's not like one has 10x the number of neurons/synapses/connections etc. of the other. They're both healthy human brains, and you need pow…

You mean the average brain with about 100 billion neurons with about 1000 connections each bringing it to around 100 trillion connections. With an estimated 1000 "AI" neurons required per biologial neuron. I don't think you are givin these "below average" intelligence individuals enough credit. What we consider a genius is the equivalent of a dog show obstacle course. We measure intelligence/genius as whatever is har…

Nobody said that nature is optimal. Wheels are trivial, however not present in biology. Nature creates tentacles, not jet engines, nuclear energy etc.

Majority of human brain computation is spent on things that are simply not necessary for computer models (how to wiggle limbs, mouth, eyes etc).

Current LLM are impressive, but we know they can be much more efficient - we're using very low quality training data, we don't use any methods to question training input/evaluate it against current knowledge etc. Our current language models are based on reciting/memorization/force-feeding, not true learning.

Computer models have massive underlying advantage of working on CPU/GPUs where they can be modeled, cloned, retrained, have binary accuracy, can integrate with specialized code instantly, have access to massive memory storage, they are insanely fast and precise etc.

Looking at it from first principles, there is no reason to think that near optimal runtime should not be more efficient than human brain on currently available computers.

We don't need to simulate full brain just like we didn't have to create super fancy legs to go to the Moon.

Post reply on HN