Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Meta Llama 3

llama.meta.com

Post reply on HN
Print view View original

941–950 of 965 posts

Re: Meta Llama 3

#941

Post by zingelshuher »

ktzar wrote: post #802
blackeyeblitzar wrote: post #136

Earlier quoted context omitted.

Llama is open weight, not open source. They don’t release all the things you need to reproduce their weights.

even if they released them, wouldn't it be prohibitively expensive to reproduce the weights?

It's impossible. Meta itself cannot reproduce the model. Because training is randomized and that info is lost. First samples a coming at random. Second there are often drop-out layers, they generate random pattern which exists only on GPU during training for the duration of a single sample. Nobody saves them, it would take much more than training data. If someone tries to re-train the patterns will be different, which results in different weight and divergence from the beginning. Model will converge to something completely different. With close behavior if training was stable. LLMs are stable.

So, no way to reproduce the model. This requirement for 'open source' is absurd. It cannot be reliably done even for small models due to GPU internal randomness. Only the smallest trained on CPU in single thread. Only academia will be interested.

Re: Meta Llama 3

#942

Post by herewego »

int_19h wrote: post #928
season2episode3 wrote: post #916

Earlier quoted context omitted.

Google’s play is not really in AI imo, it’s in the the fact that their custom silicon allows them to run models cheaply. Models are pretty much fungible at this point if you’re not trying to do any LoRAs or fine tunes.

There's still no other model on par with GPT-4. Not even close.

Many disagree. “Not even close” is a strong position to take on this.

Re: Meta Llama 3

#943

Post by zingelshuher »

exoverito wrote: post #879
endisneigh wrote: post #868

Earlier quoted context omitted.

Source?

Anecdotally speaking I use google search much less frequently and instead opt for GPT4. This is also what a number of my colleagues are doing as well.

I often use ChatGPT4 for technical info. It's easier then scrolling through pages whet it works. But.. the accuracy is inconsistent, to put it mildly. Sometimes it gets stuck on wrong idea.

Interesting how far LLMs can get? Looks like we are close to scale-up limit. It's technically difficult to get bigger models. The way to go probably is to add assisting sub-modules. Examples would be web search, have it already. Database of facts, similar to search. Compilers, image analyzers, etc. With this approach LLM is only responsible for generic decisions and doesn't need to be that big. No need to memorize all data. Even logic can be partially outsourced to sub-module.

Re: Meta Llama 3

#944

Post by HarHarVeryFunny »

akgerber wrote: post #357
HarHarVeryFunny wrote: post #281

Earlier quoted context omitted.

Pretty sure Yann is not involved in this - Meta has a separate GenAI group headed up by Ahmad Al-Dahle (@Ahmad_Al_Dahle).

Yann Lecun's name is listed in the model card: https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md

From the horse's mouth:

"I had some input on our open source policy but I had no direct technical input on Llama-3"

https://x.com/ylecun/status/1781749833981673741

Re: Meta Llama 3

#945

Post by zingelshuher »

cm2012 wrote: post #820
vineyardmike wrote: post #774

Earlier quoted context omitted.

I also disagree on Google... Google's business is largely not predicated on AI the way everyone else is. Sure they hope it's a driver of growth, but if the entire LLM industry disappeared, they'd be fine. Google doesn't need AI "Superiority", they need "good enough" to prevent the masses from product switching. If the entire world is saturated in AI, then it no longer becomes a differentiator to drive switching. And…

AI is taking marketshare from search slowly. More and more people will go to the AI to find things and not a search bar. It will be a crisis for Google in 5-10 years.

Only if it does nothing. In fact Google is one of the major players in LLM field. The winner is hard to predict, chip makers likely ;) Everybody jumped on bandwagon, Amazon is jumping...

Re: Meta Llama 3

#946

Post by zingelshuher »

HarHarVeryFunny wrote: post #251
nickthegreek wrote: post #17

Earlier quoted context omitted.

And they even allow you to use it without logging in. Didnt expect that from Meta.

Yeah, but not for image generation unfortunately I've never had a FaceBook account, and really don't trust them regarding privacy

had to upvote this

Re: Meta Llama 3

#947

Post by erhaetherth »

DresdenNick wrote: post #842
8n4vidtmkvmk wrote: post #655

Earlier quoted context omitted.

Can we stop referring to VS Code as "their IDE"? Do you support any other editors? If the list is small, just name them. Not everyone uses or likes VS Code.

Done. Anything else?

No, actually. Thank you for that.

Your "Double vs. Github Copilot" page is great.

I've signed up for the Jetbrains waitlist.

Re: Meta Llama 3

#948

Post by int_19h »

herewego wrote: post #942
int_19h wrote: post #928

Earlier quoted context omitted.

There's still no other model on par with GPT-4. Not even close.

Many disagree. “Not even close” is a strong position to take on this.

It takes less than an hour of conversation with either, giving them a few tasks requiring logical reasoning, to arrive at that conclusion. If that is a strong position, it's only because so many people seem to be buying the common scoreboards wholesale.

Re: Meta Llama 3

#949

Post by Breza »

dhon_ wrote: post #614
selcuka wrote: post #594

Earlier quoted context omitted.

I'm seeing the same behaviour. It's as if they have a post-processor that evaluates the quality of the response after a certain number of tokens have been generated, and reverts the response if it's below a threshold.

I've noticed Gemini exhibiting similar behaviour. It will start to answer, for example, a programming question - only to delete the answer and replace it with something along the lines of "I'm only a language model, I don't know how to do that"

I've seen the exact same thing! Gemini put together an impressive bash one liner then deleted it.

Re: Meta Llama 3

#950

Post by FuckButtons »

courseofaction wrote: post #911
eigenvalue wrote: post #248

Earlier quoted context omitted.

For sure. I just started watching the new Dwarkesh interview with Zuck that was just released ( https://t.co/f4h7ko0M7q ) and you can just tell from the first few minutes that he simply has a different level of enthusiasm and passion and level of engagement than 99% of big tech CEOs.

Who cares, listen to what he says. 38:30 Zuckerberg states that they won't release models once they're sufficiently powerful. It's OpenAI again, facebook has burnt all customer trust for years and the fact they changed their name to "Meta" actually worked.

I mean, he was pretty open with his motivations if you ask me, open source exists because it is a positive sum game, he gets something in return for being open, if that calculus is no longer true then he has no incentive to be open.
Post reply on HN