Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Meta AI Unleashes Megabyte, a Scalable Model Architecture

artisana.ai

Post reply on HN
Print view View original

91–100 of 213 posts

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#91

Post by primax »

jacobn wrote: post #3

My main argument against the AI doomsayers has so far been that the current scaling laws simply make runaway singularity style scenarios algorithmically impossible (if for each step of improvement you need 10x parameters and 100x training, you quickly run into a brick wall). This is part of why I’m not worried about the current crop of generative AI. I am however both curious and concerned about what the tsunami of t…

Honestly, I welcome the talent working on AI now. They were working on how to make me spend 5 more seconds on Facebook, or how to click on a Google ad. AI has potentially huge positive productivity potential.

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#92

Post by danielbln »

wolfd wrote: post #74
1024core wrote: post #7

Earlier quoted context omitted.

Meta has no horse in the race (i.e. they don't have a search engine). So, they don't mind throwing random things out. Withholding it won't really make much of a difference for them, as they don't have a way to productionize the tech.

I would be _shocked_ if Zuck wasn't thinking 24/7 about how to capitalize on LLMs. I'm sure there are a thousand ideas (maybe even a few good ones?) being thrown around Meta at how to use LLMs to beat Google/Microsoft+OpenAI at the "search buddy" game.

Good, I couldn't give two craps about the Metaverse, but give me a powerful, useful AI and you have my attention.

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#93

Post by TeMPOraL »

shreyshnaccount wrote: post #48
p-e-w wrote: post #22

Earlier quoted context omitted.

> when an average human that knows programming should be able to make one that works. Very few people who "know programming" are actually capable of creating a complete application that serves a specific purpose. Many professional programmers struggle to implement basic algorithms like prime number testing. Coding AIs absolutely outperform the average programming professional, because the average programming professi…

Who is this average programmer you're talking about? We did primality testing in literally the first sem at uni

> We did primality testing in literally the first sem at uni

There's a lot of things you did during university classes. I bet you don't even remember half of them, and of the half you do, you couldn't actually do most of them from memory right now.

The advantage LLMs have over programmers is that they've seen much more code than any human ever would, remember pretty much all of it - not necessarily verbatim, but also as learned concepts, and that this knowledge does not decay. It's all equally accessible. Whether you ask LLM for basic or optimized primality testing algorithms, or how to use some obscure part of Microsoft's DCOM that approximately nobody on the planet wrote new code for in the last decade - they'll perform equally well. Not perfect, but not worse than an average software dev that does remember the things they're asked to work with.

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#94

Post by beefield »

p-e-w wrote: post #4
jacobn wrote: post #3

My main argument against the AI doomsayers has so far been that the current scaling laws simply make runaway singularity style scenarios algorithmically impossible (if for each step of improvement you need 10x parameters and 100x training, you quickly run into a brick wall). This is part of why I’m not worried about the current crop of generative AI. I am however both curious and concerned about what the tsunami of t…

I don't buy the idea (with either architecture) that "10x"-type scaling is required for another breakthrough. Think of a human with below average intelligence. Then think of a human genius. Now consider how incredibly similar their brains are, despite the massive performance gap. It's not like one has 10x the number of neurons/synapses/connections etc. of the other. They're both healthy human brains, and you need pow…

> seems perfectly possible that a model like GPT-4 is just a hair's breadth away from vastly superhuman performance

Sorry, can't help commenting the weirdness of high dimensional spaces. If you take an cube of a size of a hair breadth (100um) in a space of hindreds of billions of dimensions like gpt4, distances between random points in that cube are in the order of tens of meters...

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#95

Post by i2cmaster »

jacobn wrote: post #3

My main argument against the AI doomsayers has so far been that the current scaling laws simply make runaway singularity style scenarios algorithmically impossible (if for each step of improvement you need 10x parameters and 100x training, you quickly run into a brick wall). This is part of why I’m not worried about the current crop of generative AI. I am however both curious and concerned about what the tsunami of t…

[flagged]

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#96

Post by jerpint »

crakenzak wrote: post #2

Paper: https://arxiv.org/abs/2305.07185 Wow, seems like Meta AI is so ahead of the curve compared to even Google and OpenAI recently especially with their open sourcing pushes. Great for the research community as a whole!

Deepmind was behind this kind of thinking years ago with Perceiver models, but meta is crushing it with high quality publications lately

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#97

Post by baq »

TeMPOraL wrote: post #86
baq wrote: post #76

Earlier quoted context omitted.

> Nobody said that nature is optimal. Wheels are trivial, however not present in biology. Nature creates tentacles, not jet engines, nuclear energy etc. Wheels are trivial but useless without bearings. Bearings most certainly aren't trivial. All the things you listed further are dependent on bearings somewhere.

AFAIK both exist in biology. Bacterial flagellum is effectively a motor, and has a working wheel-like structure. IIRC, some crickets had an equivalent of a bearing somewhere in their anatomy too. Evolution is a greedy, lazy optimizer, so it promotes things that work a-ok for a given environment. It's also worth noting that wheels alone are not too useful for transportation, as they're only half of the picture. The ot…

I actually googled before writing the comment (preposterous, I know; also, out of vouge, should've asked chatgpt) and the bacteria thing was the only thing I found, nothing macro scale, which made sense since it's easier to rebuild than to heal. Anything bigger than that would have to be healed and it's hard enough to fix mechanical bearings, can't imagine how to regrow or heal one even if it somehow grows. Maybe it's an issue with my imagination ;)

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#98

Post by mirekrusin »

TeMPOraL wrote: post #86
baq wrote: post #76

Earlier quoted context omitted.

> Nobody said that nature is optimal. Wheels are trivial, however not present in biology. Nature creates tentacles, not jet engines, nuclear energy etc. Wheels are trivial but useless without bearings. Bearings most certainly aren't trivial. All the things you listed further are dependent on bearings somewhere.

AFAIK both exist in biology. Bacterial flagellum is effectively a motor, and has a working wheel-like structure. IIRC, some crickets had an equivalent of a bearing somewhere in their anatomy too. Evolution is a greedy, lazy optimizer, so it promotes things that work a-ok for a given environment. It's also worth noting that wheels alone are not too useful for transportation, as they're only half of the picture. The ot…

If that's the case why are we sending wheeled rovers to mars/moon and not something with legs?

In any case this discussion is going sideways, the point is that nature doesn't have monopoly on being optimal. This also applies to intelligence/learning/modeling something better than brain.

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#99

Post by flangola7 »

shreyshnaccount wrote: post #77
flangola7 wrote: post #51

Earlier quoted context omitted.

Who said anything about everything? AI could not write code AT ALL until very recently. We could invent a drug that kills 99% of cancers and the next day there would be people bemoaning that it isn't a "true" cure.

Hey you're the one claiming intelligence and all, burden of proving it in squarely on you

Where have I claimed anything?

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#100

Post by dontupvoteme »

p-e-w wrote: post #15
nemothekid wrote: post #11

Earlier quoted context omitted.

> Think of a human with below average intelligence. Then think of a human genius. LLMs are not AGI. A human with below average intelligence is still a league above a chimpanzee. A chimpanzee will never be able to read, not because "it's too dumb", but because a chimp's brain lacks the actual hardware for reading. The LLM is the chimpanzee. The gap between an LLM and a "human with below average intelligence" is far mo…

> The gap between an LLM and a "human with below average intelligence" is far more than 10x. In which direction? GPT-4 passes the bar exam with a top 10% score. How do you think a human with below average intelligence (or even with average intelligence) would fare? Copilot generates programming code that solves problems, and in most cases the code is correct. It outperforms many junior professional developers. Do you…

it's controversial but they're not allowed to sue us for it -- being a lawyer or doctor just requires a lot of rote memorization.
Post reply on HN