Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Meta AI Unleashes Megabyte, a Scalable Model Architecture

artisana.ai

Post reply on HN
Print view View original

101–110 of 213 posts

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#101

Post by somewhereoutth »

p-e-w wrote: post #4
jacobn wrote: post #3

My main argument against the AI doomsayers has so far been that the current scaling laws simply make runaway singularity style scenarios algorithmically impossible (if for each step of improvement you need 10x parameters and 100x training, you quickly run into a brick wall). This is part of why I’m not worried about the current crop of generative AI. I am however both curious and concerned about what the tsunami of t…

I don't buy the idea (with either architecture) that "10x"-type scaling is required for another breakthrough. Think of a human with below average intelligence. Then think of a human genius. Now consider how incredibly similar their brains are, despite the massive performance gap. It's not like one has 10x the number of neurons/synapses/connections etc. of the other. They're both healthy human brains, and you need pow…

However, that is to equivocate an LLM with a human brain. They both can be conceptualised with the idea of the 'neuron' (though with wildly different actual implementations of that term), but that is their only point of comparison. Thus your conjecture is invalid.

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#102

Post by pmoriarty »

baq wrote: post #76
mirekrusin wrote: post #60

Earlier quoted context omitted.

Nobody said that nature is optimal. Wheels are trivial, however not present in biology. Nature creates tentacles, not jet engines, nuclear energy etc. Majority of human brain computation is spent on things that are simply not necessary for computer models (how to wiggle limbs, mouth, eyes etc). Current LLM are impressive, but we know they can be much more efficient - we're using very low quality training data, we don…

> Nobody said that nature is optimal. Wheels are trivial, however not present in biology. Nature creates tentacles, not jet engines, nuclear energy etc. Wheels are trivial but useless without bearings. Bearings most certainly aren't trivial. All the things you listed further are dependent on bearings somewhere.

"Wheels are trivial but useless without bearings."

Carts, chariots, and wheelbarrows (to name but a few examples) have been useful for thousands of years without bearings.

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#103

Post by DrScientist »

p-e-w wrote: post #4
jacobn wrote: post #3

My main argument against the AI doomsayers has so far been that the current scaling laws simply make runaway singularity style scenarios algorithmically impossible (if for each step of improvement you need 10x parameters and 100x training, you quickly run into a brick wall). This is part of why I’m not worried about the current crop of generative AI. I am however both curious and concerned about what the tsunami of t…

I don't buy the idea (with either architecture) that "10x"-type scaling is required for another breakthrough. Think of a human with below average intelligence. Then think of a human genius. Now consider how incredibly similar their brains are, despite the massive performance gap. It's not like one has 10x the number of neurons/synapses/connections etc. of the other. They're both healthy human brains, and you need pow…

> Considering this, it seems perfectly possible that a model like GPT-4 is just a hair's breadth away from vastly superhuman performance.

Except that structurally the brain is clearly has vastly more capacity than the GPT-4 model.

So sure one brain doesn't look that much different to the other - and it's in the details of the learning, wiring.

But the brain, looks vastly different from a GPT-4 model in terms of capacity - with trillions of connections - with each connection and internal state being more subtle as well.

> vastly superhuman performance

In terms of specific tasks computers ( whether you write the program explicitly or it's learnt by tweaking params in a network ) have been there for decades.

So the question is really around which tasks can you apply computers successful to. Neural nets are allowing programs to be written that weren't possible be hand.

I find it amusing that people worry about ChatGPT etc al putting programmers out of a job, when it already has in the sense that CHapGPT is a program that was built by another program already.

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#104

Post by pmoriarty »

p-e-w wrote: post #15
nemothekid wrote: post #11

Earlier quoted context omitted.

> Think of a human with below average intelligence. Then think of a human genius. LLMs are not AGI. A human with below average intelligence is still a league above a chimpanzee. A chimpanzee will never be able to read, not because "it's too dumb", but because a chimp's brain lacks the actual hardware for reading. The LLM is the chimpanzee. The gap between an LLM and a "human with below average intelligence" is far mo…

> The gap between an LLM and a "human with below average intelligence" is far more than 10x. In which direction? GPT-4 passes the bar exam with a top 10% score. How do you think a human with below average intelligence (or even with average intelligence) would fare? Copilot generates programming code that solves problems, and in most cases the code is correct. It outperforms many junior professional developers. Do you…

"GPT-4 passes the bar exam with a top 10% score."

Turns out that may be as much marketing as truth.

According to this paper[1]:

"although GPT-4's UBE score nears the 90th percentile when examining approximate conversions from February administrations of the Illinois Bar Exam, these estimates are heavily skewed towards repeat test-takers who failed the July administration and score significantly lower than the general test-taking population. Second, data from a recent July administration of the same exam suggests GPT-4's overall UBE percentile was ~68th percentile, and ~48th percentile on essays. Third, examining official NCBE data and using several conservative statistical assumptions, GPT-4's performance against first-time test takers is estimated to be ~63rd percentile, including ~41st percentile on essays. Fourth, when examining only those who passed the exam (i.e. licensed or license-pending attorneys), GPT-4's performance is estimated to drop to ~48th percentile overall, and ~15th percentile on essays."

[1] - https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4441311

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#105

Post by fnordpiglet »

nemothekid wrote: post #11
p-e-w wrote: post #4

Earlier quoted context omitted.

I don't buy the idea (with either architecture) that "10x"-type scaling is required for another breakthrough. Think of a human with below average intelligence. Then think of a human genius. Now consider how incredibly similar their brains are, despite the massive performance gap. It's not like one has 10x the number of neurons/synapses/connections etc. of the other. They're both healthy human brains, and you need pow…

> Think of a human with below average intelligence. Then think of a human genius. LLMs are not AGI. A human with below average intelligence is still a league above a chimpanzee. A chimpanzee will never be able to read, not because "it's too dumb", but because a chimp's brain lacks the actual hardware for reading. The LLM is the chimpanzee. The gap between an LLM and a "human with below average intelligence" is far mo…

LLM are a single model. Multimodal LLM tightly integrated with classical AI techniques to inform, conform, compel, solve, optimize, etc in a tight feedback loop…. I think folks are too entranced by the abilities of LLM and too jaded to realize the flaws are in domains we’ve already achieved great success. the magic really comes when we combine them with classical AI techniques. I don’t think you need more than GPT4 level LLM, what you need is goal based agency, information retrieval, optimizers, solvers, etc. These are things we’ve already created. The magic comes not from bigger LLM but from integration.

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#106

Post by iinnPP »

jacobn wrote: post #3

My main argument against the AI doomsayers has so far been that the current scaling laws simply make runaway singularity style scenarios algorithmically impossible (if for each step of improvement you need 10x parameters and 100x training, you quickly run into a brick wall). This is part of why I’m not worried about the current crop of generative AI. I am however both curious and concerned about what the tsunami of t…

This assumes only the LLM matters. I don't think it does. I think GPT3.5 is plenty, if not overkill.

I can't imagine I am alone in my thought. I've even seen some other experiments in the same line of reasoning.

Meta seems to be thinking the same thing, or at least closer to it than the majority.

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#107

Post by DrScientist »

mirekrusin wrote: post #60
wudangmonk wrote: post #39

Earlier quoted context omitted.

You mean the average brain with about 100 billion neurons with about 1000 connections each bringing it to around 100 trillion connections. With an estimated 1000 "AI" neurons required per biologial neuron. I don't think you are givin these "below average" intelligence individuals enough credit. What we consider a genius is the equivalent of a dog show obstacle course. We measure intelligence/genius as whatever is har…

Nobody said that nature is optimal. Wheels are trivial, however not present in biology. Nature creates tentacles, not jet engines, nuclear energy etc. Majority of human brain computation is spent on things that are simply not necessary for computer models (how to wiggle limbs, mouth, eyes etc). Current LLM are impressive, but we know they can be much more efficient - we're using very low quality training data, we don…

> Wheels are trivial, however not present in biology.

Because roads don't exist in nature..... imagine trying to out run a predator if you just had wheels but no roads.

> Nature creates tentacles, not jet engines, nuclear energy etc

Under water I think you'll find there is jet propulsion.

Plants and animals are powered by nuclear energy - the remote fusion reaction is in the sky. Why have an internal nuclear reactor - fissile material is quite rare and fusion is quite hard to contain.

Human's are overly fond of their own children ( machines ) but fail to see the complexity of life. People are in awe of the latest Boston Dynamics robot - wow it can run - wow if can leap - and but don't post pictures of horses or gymnasts.

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#108

Post by jiggawatts »

dr_dshiv wrote: post #89
ithkuil wrote: post #46

Earlier quoted context omitted.

> it's not a matter of having a 100x more powerful LLM, I think we all can agree that even the best LLM currently is not AGI. That's not what being disputed here I think. However a 100x more powerful LLM is not just 100x better at recall. A 100x more powerful LLM is not just 100x better at being stupid hallucinatory parrot. A model that is just 100x bigger is not necessarily 100x more powerful if you define power is…

> I think we all can agree that even the best LLM currently is not AGI. Disagree, for the record. If I’d described the capabilities of contemporary AI to 100 AI scientists 5 years ago, I bet more than half would agree to call that AGI. Further, more than 90% would assume that these capabilities were decades and decades away.

“Oh those goalposts? We moved them over there because they were getting uncomfortably close.”

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#109

Post by DrScientist »

baq wrote: post #76
mirekrusin wrote: post #60

Earlier quoted context omitted.

Nobody said that nature is optimal. Wheels are trivial, however not present in biology. Nature creates tentacles, not jet engines, nuclear energy etc. Majority of human brain computation is spent on things that are simply not necessary for computer models (how to wiggle limbs, mouth, eyes etc). Current LLM are impressive, but we know they can be much more efficient - we're using very low quality training data, we don…

> Nobody said that nature is optimal. Wheels are trivial, however not present in biology. Nature creates tentacles, not jet engines, nuclear energy etc. Wheels are trivial but useless without bearings. Bearings most certainly aren't trivial. All the things you listed further are dependent on bearings somewhere.

> but useless without bearings.

Or more importantly - level ground.

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#110

Post by TeMPOraL »

mirekrusin wrote: post #98
TeMPOraL wrote: post #86

Earlier quoted context omitted.

AFAIK both exist in biology. Bacterial flagellum is effectively a motor, and has a working wheel-like structure. IIRC, some crickets had an equivalent of a bearing somewhere in their anatomy too. Evolution is a greedy, lazy optimizer, so it promotes things that work a-ok for a given environment. It's also worth noting that wheels alone are not too useful for transportation, as they're only half of the picture. The ot…

If that's the case why are we sending wheeled rovers to mars/moon and not something with legs? In any case this discussion is going sideways, the point is that nature doesn't have monopoly on being optimal. This also applies to intelligence/learning/modeling something better than brain.

> If that's the case why are we sending wheeled rovers to mars/moon and not something with legs?

Because Mars is a simple and boring environment. Most of its surface, and especially the parts we target with rover missions, are effectively flat sheets peppered with rocks - a decent set of wheels and suspension is close to optimal for navigating such terrain.

Now, if we were to send missions to a planet that's mostly forests and rivers, like Earth used to be, then wheels wouldn't cut it - not before cutting down some of the forests first.

> the point is that nature doesn't have monopoly on being optimal. This also applies to intelligence/learning/modeling something better than brain.

Fair enough. Nature doesn't do globally optimal - but it makes things heavily optimized for their environment. That's why our planes are nowhere near as energy-efficient in flying as birds are, but birds cannot travel as far and as fast as our planes can.

Post reply on HN