Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Meta AI Unleashes Megabyte, a Scalable Model Architecture

artisana.ai

Post reply on HN
Print view View original

121–130 of 213 posts

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#121

Post by pmoriarty »

laratied wrote: post #117
Henk0 wrote: post #87

Earlier quoted context omitted.

This. So much this. I'm completely dumbfounded by obviously highly intelligent people consistently not getting this, and dismissing current generation AI systems as not being intelligent because they can't reliably solve massively complex problems in one go. Like anyone would expect a human programmer or researcher to just intuitively come up with a complex program, or the correct answer for a hard problem every time…

I would add to your amazing list that we are really good at denial as a coping mechanism with change. I am not a fan of the concept of AGI though. This means so many different things to people that it seems pointless to debate something when most likely we are not talking about the same thing. François Chollet has said that he believes all intelligence is specialized intelligence. From that perspective, whatever peop…

"Humanity will benefit enormously from this huge increase in the availability of intelligence."

It's a near certainty that AI will be used to create more effective/destructive weapons (if it hasn't already), and will likely be used by terrorists, scammers, and others who wish to harm humans in some way.

As this technology becomes more powerful, easier, and cheaper to use, all sorts of harmful uses of it will be made. The effectiveness and scale of this harm will also increase.

And that's all before even considering what will happen if/when AI's become truly intelligent, self-motivating, indepent, and self-aware.

The jury is still out on whether the net harm will out weigh the net benefit, and if humanity will survive something that might be analogous to neanderthals encountering homo sapiens.

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#122

Post by shreyshnaccount »

dr_dshiv wrote: post #82
shreyshnaccount wrote: post #77

Earlier quoted context omitted.

Hey you're the one claiming intelligence and all, burden of proving it in squarely on you

Yes. Let’s define AGI as ability for a single model to pass most human professional tests (no cheating) and to provide genuine human-level flexible cognitive benefit to specialized professionals in diverse fields. Reasonable?

No, because most tests designed for humans test memory and pattern recognition, which computers already can do better than humans so it's not a useful comparison. I'd rather define it to be superhuman AGI when it not only performs better on tests with humans who can use computers during the test but also can perform everyday tasks which are not 'hard' for us humans. That is because we have the hardware in our brains to do these things, doesn't mean that it's easy in the least for a computer.

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#123

Post by shreyshnaccount »

flangola7 wrote: post #99
shreyshnaccount wrote: post #77

Earlier quoted context omitted.

Hey you're the one claiming intelligence and all, burden of proving it in squarely on you

Where have I claimed anything?

I'm sorry if I misunderstood your comment but your tone seemed to imply that

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#124

Post by shreyshnaccount »

TeMPOraL wrote: post #93
shreyshnaccount wrote: post #48

Earlier quoted context omitted.

Who is this average programmer you're talking about? We did primality testing in literally the first sem at uni

> We did primality testing in literally the first sem at uni There's a lot of things you did during university classes. I bet you don't even remember half of them, and of the half you do, you couldn't actually do most of them from memory right now. The advantage LLMs have over programmers is that they've seen much more code than any human ever would, remember pretty much all of it - not necessarily verbatim, but also…

Yeah, no contest, llms can remember things. Big deal. You're comparing memory with intelligence. It's a part of it sure, but there's also cognition, reasoning and i don't even know what more

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#125

Post by dr_dshiv »

shreyshnaccount wrote: post #122
dr_dshiv wrote: post #82

Earlier quoted context omitted.

Yes. Let’s define AGI as ability for a single model to pass most human professional tests (no cheating) and to provide genuine human-level flexible cognitive benefit to specialized professionals in diverse fields. Reasonable?

No, because most tests designed for humans test memory and pattern recognition, which computers already can do better than humans so it's not a useful comparison. I'd rather define it to be superhuman AGI when it not only performs better on tests with humans who can use computers during the test but also can perform everyday tasks which are not 'hard' for us humans. That is because we have the hardware in our brains…

Have some examples? What would be tests that, if passed, you’d say “oh yeah, that’s AGI.”

For instance, if it could make a peanut butter and jelly sandwich? Most challenging things that are easy for us are in the motor domain. While important, I think “intellectual AGI” is a meaningful milestone and closest to what most people think of when they think AGI.

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#126

Post by vlovich123 »

doctor_eval wrote: post #30
amoss wrote: post #26

Earlier quoted context omitted.

Certainty about predictions of the future has always been close to zero. If you take a person from an appropriate time and ask them what the future will look like in 1000, 100 or even 20 years then their predictions will bear little resemblence to what actually occurs. As humans we have a tendancy to make linear future predictions based on past observations. Over the timescales that matter significant effects occur f…

I would say that we are really saying the same thing. I did intend to imply “short term predictions”. However I feel that these LLMs are not like the internet or the release of the iPhone. We’ve gone in a very short time from LLMs in the lab to ChatGPT to passing the bar exam. Considering the rate of research that’s being published and the fact that what we see today is generally at least many months behind the state…

Any exponential growth in tech makes it hard for us to predict the outcome. By that metric we’ve been in that phase for some 70 years since the invention of the transistor. The world today looks nothing like the world in the 50s and the transistor is a huge part of that reason. And that is true - that was the technological singularity and we’ve been in that world ever since.

When people today talk about the AI singularity though it’s a slightly different definition from the more general technological singularity which is that the AI itself is delivering that improvement with no input from humans.

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#127

Post by Henk0 »

pmoriarty wrote: post #120
Henk0 wrote: post #87

Earlier quoted context omitted.

This. So much this. I'm completely dumbfounded by obviously highly intelligent people consistently not getting this, and dismissing current generation AI systems as not being intelligent because they can't reliably solve massively complex problems in one go. Like anyone would expect a human programmer or researcher to just intuitively come up with a complex program, or the correct answer for a hard problem every time…

"Processes that AI researchers are just now beginning to explore, with results like increasing reasoning ability by 900% in a recent paper" Would you happen to have a link to that paper?

Explanatory blog post with link to the paper:

https://www.aibloggs.com/post/tree-of-thoughts-supercharging...

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#128

Post by Henk0 »

pmoriarty wrote: post #121
laratied wrote: post #117

Earlier quoted context omitted.

I would add to your amazing list that we are really good at denial as a coping mechanism with change. I am not a fan of the concept of AGI though. This means so many different things to people that it seems pointless to debate something when most likely we are not talking about the same thing. François Chollet has said that he believes all intelligence is specialized intelligence. From that perspective, whatever peop…

"Humanity will benefit enormously from this huge increase in the availability of intelligence." It's a near certainty that AI will be used to create more effective/destructive weapons (if it hasn't already), and will likely be used by terrorists, scammers, and others who wish to harm humans in some way. As this technology becomes more powerful, easier, and cheaper to use, all sorts of harmful uses of it will be made.…

Yes, great point

So many of the people who opine about AI, its trajectory, and its possible effects on society, have latched on to one or two possible effects - like it overtaking jobs, or massively increasing misinformation. These are both very valid concerns, but they're only a tiny part of the big picture

The thinker who I perceive as having the best holistic (in the non-wooey sense of the word) understanding of how the rapid development of AI will affect this and a number of other social and existential risks is Daniel Schmachtenberger. He lays it out well in this episode of the Theories of Everything Podcast: https://www.youtube.com/watch?v=g7WtcTATa2U&t=2373s

Highly recommend watching it, even if it's long. Some main points though: - AI will increase the rate of development of every other technology it is applied to - In fields like biotech, this can lead to cancer cures, but also to increasingly dangerous bioweapons - Our current economic system is based on exponential economic growth in a limited resource world. AI applied in the service of profit will amplify this, leading us increasingly fast towards a number of tipping points. Of course, AI can also help steer us away from that path, but that is not the natural attractor - Game theoretic multipolar traps (aka Moloch) incentivise arms races and races to the bottom just like we see now. Those who are willing to move fast and break things have an advantage in these dynamics vs. those who prefer to move slowly and carefully - Cheaper and more efficient AI models will lead to increasing decentralisation of the technology, making it very hard to control - unlike current weapons of mass destruction

List goes on, but Daniel makes a much better case. Again, I would love to hear a good critique of his thinking, but haven't come across one yet

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#129

Post by lostmsu »

DrScientist wrote: post #103
p-e-w wrote: post #4

Earlier quoted context omitted.

I don't buy the idea (with either architecture) that "10x"-type scaling is required for another breakthrough. Think of a human with below average intelligence. Then think of a human genius. Now consider how incredibly similar their brains are, despite the massive performance gap. It's not like one has 10x the number of neurons/synapses/connections etc. of the other. They're both healthy human brains, and you need pow…

> Considering this, it seems perfectly possible that a model like GPT-4 is just a hair's breadth away from vastly superhuman performance. Except that structurally the brain is clearly has vastly more capacity than the GPT-4 model. So sure one brain doesn't look that much different to the other - and it's in the details of the learning, wiring. But the brain, looks vastly different from a GPT-4 model in terms of capac…

> brain is clearly has vastly more capacity

I would not be so sure about "clearly" bit. Brain has 100B neurons x 1000 connections. GPT3 has 175B connections, but to implement operations used in those connections Nvidia H100 uses about 5M transistors, which in human brain would have to be copied to each connection because nature didn't invent software. (assumes one needs one full CUDA core to implement necessary ops, and that H100's 80B transistors are evenly divided between its 16k cores)

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#130

Post by chaxor »

wudangmonk wrote: post #39
p-e-w wrote: post #4

Earlier quoted context omitted.

I don't buy the idea (with either architecture) that "10x"-type scaling is required for another breakthrough. Think of a human with below average intelligence. Then think of a human genius. Now consider how incredibly similar their brains are, despite the massive performance gap. It's not like one has 10x the number of neurons/synapses/connections etc. of the other. They're both healthy human brains, and you need pow…

You mean the average brain with about 100 billion neurons with about 1000 connections each bringing it to around 100 trillion connections. With an estimated 1000 "AI" neurons required per biologial neuron. I don't think you are givin these "below average" intelligence individuals enough credit. What we consider a genius is the equivalent of a dog show obstacle course. We measure intelligence/genius as whatever is har…

Listing the number of neurons in a brain has very little to do with these systems, so it's pretty meaningless. Also, the number of neurons in Wernicke's area and the PFC is quite a small fraction of the brain, making this even more meaningless.
Post reply on HN