Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Meta Llama 3

llama.meta.com

Post reply on HN
Print view View original

791–800 of 965 posts

Re: Meta Llama 3

#791

Post by originalvichy »

cornholio wrote: post #786
paxys wrote: post #229

Earlier quoted context omitted.

Very interesting part around 5 mins in where Zuck says that they bought a shit ton of H100 GPUs a few years ago to build the recommendation engine for Reels to compete with TikTok (2x what they needed at the time, just to be safe), and now they are accidentally one of the very few companies out there with enough GPU capacity to train LLMs at this scale.

The only thing the Reels algorithm is showing me are videos of ladies with fat butts. Now, I must admit, I may have clicked once on such a video. Should I now be damned to spend an eternity in ass hell?

I could have saved them a lot of money by revealing to them that, yes, heterosexual men tend to gravitate towards ladies with fat butts.

I have a hunch that some of the more professional folks there game the algorithm. If you ever wanna find a place where people share algo optimization secrets, it’s OF creators on reddit.

Re: Meta Llama 3

#792

Post by baby »

tmalsburg2 wrote: post #324
noiseinvacuum wrote: post #254

Earlier quoted context omitted.

Good thing that he's only 39 years old and seems more energetic than ever to run his company. Having a passionate founder is, imo, a big advantage for Meta compared to other big tech companies.

Love how everyone is romanticizing his engineering mindset. But have we already forgotten that he was even more passionate about the metaverse which, as far as I can tell, was a 50B failure?

I get so annoyed by this every time I see it. It’s not because AI took over the news cycle that the idea of a Metaverse is a failure.

If you could have predicted that Internet was going to change our lives and that most people would spend most of their waking hours living their lives on the Internet people probably would have told you that you were a fool in the early days.

The same is true with this prediction of VR. If you think in the next decade that VR is not going to be the home for more and more people then you are wrong.

Re: Meta Llama 3

#793

Post by krisoft »

simonw wrote: post #696
RexM wrote: post #659

Earlier quoted context omitted.

It seems pretty clear cut that it’s monthly active users when Llama 3 is released. > If, on the Meta Llama 3 version release date, the monthly active users … is greater than 700 million monthly active users in the preceding calendar month … If that’s not true then the free license applies to you.

What happens if your startup is in negotiations to be acquired by a company that had more than 700m users before that date?

Now that I would argue puts you at a risk!

Re: Meta Llama 3

#794

Post by spi »

eggdaft wrote: post #739
mvkel wrote: post #732

Earlier quoted context omitted.

1. Free rlhf 2. They cookie the hell out of you to breadcrumb your journey around the web. They don't need you to login to get what they need, much like Google

Do they really need “free RLHF”? As I understand it, RLHF needs relatively little data to work and its quality matters - I would expect paid and trained labellers to do a much better job than Joey Keyboard clicking past a “which helped you more” prompt whilst trying to generate an email.

Variety matters a lot. If you pay 1000 trained labellers, you get 1000 POVs for a good amount of money, and likely can't even think of 1000 good questions to have them ask. If you let 1000000 people give you feedback on random topics for free, and then pay 100 trained people to go through all of that and only retain the most useful 1%, you get much ten times more variety for a tenth of the cost.

Of course numbers are pretty random, but it's just to give an idea of how these things scale. This is my experience from my company's own internal -deep learning but not LLM- models to train which we had to buy data instead of collecting it. If you can't tap into data "from the wild" -in our case, for legal reason- you can still get enough data (if measured in GB), but it's depressingly more repetitive, and that's not quite the same thing when you want to generalize.

Re: Meta Llama 3

#795

Post by jiggawatts »

littlestymaar wrote: post #771
jiggawatts wrote: post #667

Earlier quoted context omitted.

> the same way you can edit the binary of a proprietary game to disable DRMs, that doesn't make it open-source either This is where I have to disagree. Continuing the training of an open model is the same process as the original training run. It's not a fundamentally different operation.

> Continuing the training of an open model is the same process as the original training run. It's not a fundamentally different operation. In practice it's not (because LoRA) but that doesn't matter: continuing the training is just a patch on top of the initial training, it doesn't matter if this patch is applied through gradient descent as well, you are completely dependent on how the previous training was done, and…

> (because LoRA)

That's one of many techniques, and is popular because it's cheap to implement. The training of a full model can be continued with full updates, the same as the original training run.

> completely dependent on how the previous training was done, and your ability to overwrite the model's behavior is limited.

Not necessarily. You can even alter the architecture! There have been many papers about various approaches such as extending token window sizes, or adding additional skip connections, quantization, sparsity, or whatever.

> specially crafted group of rare tokens

The analogy here is that some Linux kernel developer could have left a back door in the Linux kernel source. You're arguing that Linux would only be open source if you could personally go back to the time when it was an empty folder on Linus Torvald's computer and then reproduce every step it took to get to today's tarball of the source, including every Google search done, every book referenced, every email read, etc...

That's not what open source is. The code is open, not the process that it took to get there.

Linux development may have used information from copyrighted textbooks. The source code doesn't contain the text of those textbooks, and in some sense could not be "reproduced" without the copyrighted text.

Similarly, AIs are often trained on copyrighted textbooks but the end result is open source.

Re: Meta Llama 3

#796

Post by baby »

aerialfish wrote: post #706
tmalsburg2 wrote: post #352

Earlier quoted context omitted.

His engineering mindset made him blind to the fact the metaverse was a product that nobody wanted or needed. In one of the Fridman interviews, he goes on and on about all the cool technical challenges involved in making the metaverse work. But when Fridman asked him what he likes to do in his spare time, it was all things that you could precisely not do in the metaverse. It was baffling to me that he failed to connec…

Yes, I thought the same exact thing. Seemed so odd to hear him gush over his foiling and MMA while simultaneously expecting everyone else to migrate to the metaverse.

He wants to see MMA fights from VR, pretty good usecase.

Re: Meta Llama 3

#797

Post by baby »

asadotzler wrote: post #679
brigadier132 wrote: post #453

Earlier quoted context omitted.

was a failure? they are still building it, when they shut down or sell off the division then you can call it a failure

10 years, $50 billion, fewer than 10 million MAU. It's a failure today, right this minute it's a failure.

Disagree from VR

Re: Meta Llama 3

#798

Post by zzzzzzzzzz10 »

ben_w wrote: post #757
eigenvalue wrote: post #438

Earlier quoted context omitted.

Yes, I completely agree with every point you made. It’s going to be so satisfying when all the AI safety people realize that their attempts to cram this protectionist/alarmist control down our throats are all for nothing, because there is an even stronger model that is totally open weights, and you can never put the genie back in the bottle!

> you can never put the genie back in the bottle That's specifically why OpenAI don't release weights, and why everyone who cares about safety talks about laws, and why Yud says the laws only matter if you're willing to enforce them internationally via air strikes. > It’s going to be so satisfying I won't be feeling Schadenfreude if a low budget group or individual takes an open weights model, does a white-box analys…

If some unhinged psycho want to build nerve agents and bombs I think it's laughable to believe an LLM will be the tool that makes a difference in enabling them to do so.

As you said the information is already out there - getting info on how to do this stuff is not the barrier you think it is.

Re: Meta Llama 3

#799

Post by littlestymaar »

jiggawatts wrote: post #795
littlestymaar wrote: post #771

Earlier quoted context omitted.

> Continuing the training of an open model is the same process as the original training run. It's not a fundamentally different operation. In practice it's not (because LoRA) but that doesn't matter: continuing the training is just a patch on top of the initial training, it doesn't matter if this patch is applied through gradient descent as well, you are completely dependent on how the previous training was done, and…

> (because LoRA) That's one of many techniques, and is popular because it's cheap to implement. The training of a full model can be continued with full updates, the same as the original training run. > completely dependent on how the previous training was done, and your ability to overwrite the model's behavior is limited. Not necessarily. You can even alter the architecture! There have been many papers about various…

> Not necessarily. You can even alter the architecture!

You can alter the architecture, but you're still playing with an opaque blob of binary *you don't know what it's made of*.

> The analogy here is that some Linux kernel developer could have left a back door in the Linux kernel source. You're arguing that Linux would only be open source if you could personally go back to the time when it was an empty folder on Linus Torvald's computer and then reproduce every step it took to get to today's tarball of the source, including every Google search done, every book referenced, every email read, etc...

No, it is just a bad analogy. To be sure that there's no backdoor in the Linux kernel, the code itself suffice. That doesn't mean there can be no backdoor since it's complex enough to hide things in it, but it's not the same thing as a backdoor hidden in a binary blob you cannot inspect even if you had a trillion dollar to spend on a million of developers.

> The code is open, not the process that it took to get there.

The code is by definition a part of a process that gets you a piece of software (which is the actually useful binary), and it's the part of the process that contains most of the value. Model weights are binary, and they are akin to the compiled binary of the software (training from data being a compute-intensive like compilation from source code, but orders of magnitude more intensive).

> Similarly, AIs are often trained on copyrighted textbooks but the end result is open source.

Court decisions are pending on the mere legality of such training, and it has nothing to do with being open-source, what's at stake is whether or not these models can be open-weight or if it is copyright infringement to publish the models.

Re: Meta Llama 3

#800

Post by bionhoward »

paxys wrote: post #240
eigenvalue wrote: post #231

I just want to express how grateful I am that Zuck and Yann and the rest of the Meta team have adopted an open approach and are sharing the model weights, the tokenizer, information about the training data, etc. They, more than anyone else, are responsible for the explosion of open research and improvement that has happened with things like llama.cpp that now allow you to run quite decent models locally on consumer h…

You can see from Zuck's interviews that he is still an engineer at heart. Every other big tech company has lost that kind of leadership.

Are you joking? “ v. You will not use the Llama Materials or any output or results of the Llama Materials to improve any other large language model (excluding Llama 2 or derivative works thereof). “ is no sign of a strong engineering culture, it’s a sign of greed.
Post reply on HN