Meta Llama 3 • Hacker News Board

Re: Meta Llama 3

#301

Post by tinyspacewizard » Thu, Apr 18, 2024, 6:31 PM UTC

swalsh wrote: post #279

eigenvalue wrote: post #231

I just want to express how grateful I am that Zuck and Yann and the rest of the Meta team have adopted an open approach and are sharing the model weights, the tokenizer, information about the training data, etc. They, more than anyone else, are responsible for the explosion of open research and improvement that has happened with things like llama.cpp that now allow you to run quite decent models locally on consumer h…

Why is Meta doing it though? This is an astronomical investment. What do they gain from it?

I think what Meta is doing is really smart.

We don't really know where AI will be useful in a business sense yet (the apps with users are losing money) but a good bet is that incumbent platforms stand to benefit the most once these uses are discovered. What Meta is doing is making it easier for other orgs to find those use-cases (and take on the risk) whilst keeping the ability to jump in and capitalize on it when it materializes.

As for X-Risk? I don't think any of the big tech leadsership actually beleive in that. I also think that deep down a lot of the AI safety crowd love solving hard problems and collecting stock options.

On cost, the AI hype raises Met's valuation by more than the cost of engineers and server farms.

Re: Meta Llama 3

#302

Post by schleck8 » Thu, Apr 18, 2024, 6:32 PM UTC

swalsh wrote: post #279

eigenvalue wrote: post #231

I just want to express how grateful I am that Zuck and Yann and the rest of the Meta team have adopted an open approach and are sharing the model weights, the tokenizer, information about the training data, etc. They, more than anyone else, are responsible for the explosion of open research and improvement that has happened with things like llama.cpp that now allow you to run quite decent models locally on consumer h…

Why is Meta doing it though? This is an astronomical investment. What do they gain from it?

Zuck equated the current point in AI to iOS vs Android and MacOS vs Windows. He thinks there will be an open ecosystem and a closed one coexisting if I got that correctly, and thinks he can make the former.

Re: Meta Llama 3

#303

Post by ativzzz » Thu, Apr 18, 2024, 6:32 PM UTC

swalsh wrote: post #279

eigenvalue wrote: post #231

I just want to express how grateful I am that Zuck and Yann and the rest of the Meta team have adopted an open approach and are sharing the model weights, the tokenizer, information about the training data, etc. They, more than anyone else, are responsible for the explosion of open research and improvement that has happened with things like llama.cpp that now allow you to run quite decent models locally on consumer h…

Why is Meta doing it though? This is an astronomical investment. What do they gain from it?

Meta is an advertising company that is primarily driven by user generated content. If they can empower more people to create more content more quickly, they make more money. Particularly the metaverse, if they ever get there, because making content for 3d VR is very resource intensive.

Making AI as open as possible so more people can use it accelerates the rate of content creation

Re: Meta Llama 3

#304

Post by chaoz_ » Thu, Apr 18, 2024, 6:34 PM UTC

aster0id wrote: post #188

lordswork wrote: post #132

Earlier quoted context omitted.

Lex walked so that Dwarkesh could run. He runs the best AI podcast around right now, by a long shot.

I agree that it is the best AI podcast. I do have a few gripes though, which might just be from personal preference. A lot of the time the language used by both the host and the guests is unnecessarily obtuse. Also the host is biased towards being optimistic about LLMs leading to AGI, and so he doesn't probe guests deep enough about that, more than just asking something along the lines of "Do you think next token pre…

but do you think "next token prediction is enough for AGI" though?

Re: Meta Llama 3

#305

Post by aster0id » Thu, Apr 18, 2024, 6:37 PM UTC

chaoz_ wrote: post #304

aster0id wrote: post #188

Earlier quoted context omitted.

I agree that it is the best AI podcast. I do have a few gripes though, which might just be from personal preference. A lot of the time the language used by both the host and the guests is unnecessarily obtuse. Also the host is biased towards being optimistic about LLMs leading to AGI, and so he doesn't probe guests deep enough about that, more than just asking something along the lines of "Do you think next token pre…

but do you think "next token prediction is enough for AGI" though?

I think AGI is less a "generation" problem and more a "context retrieval" problem. I am an outsider looking in to the field, though, so I might be completely wrong.

Re: Meta Llama 3

#306

Post by graeme » Thu, Apr 18, 2024, 6:37 PM UTC

Solvency wrote: post #298

paxys wrote: post #240

Earlier quoted context omitted.

You can see from Zuck's interviews that he is still an engineer at heart. Every other big tech company has lost that kind of leadership.

[flagged]

If you combine engineer mindset, business acumen, relentless drive and do so over decades, you can get outsized results.

It's a thing to admire, *even if you dislike the products*. Much the same as you can be awed by Ray Kroc's execution regardless of whether you like McDonald's or what you think of him personally.

It simply isn't that common to have that combination of talents at work on one thing at such scale for so long. Steve Jobs and Bill Gates had the same combo of really being down in the details despite reaching such heights.

You can contrast to Google, a company whose founders had similar traits but who got tired of it. Totally understandable, but it makes a difference in terms of the focus of google today.

Again this is true regardless of what you think of Meta on, say, privacy vs. Google's original "Don't be Evil" idea.

Saying "wow they still have engineering leadership" is hardly worship. It's a statement of fact.

Re: Meta Llama 3

#307

Post by aster0id » Thu, Apr 18, 2024, 6:37 PM UTC

mrtranscendence wrote: post #291

HarHarVeryFunny wrote: post #270

Earlier quoted context omitted.

There's a difference to being a good chatshow/podcast host and a journalist holding someone's feet to the fire! Dwarkesh is excellent at what he does - lots of research beforehand (which is how he lands these great guests), but then lets the guest do most of the talking, and encourages them to expand on what they are saying. It you are critisizing the guest or giving them too much push back, then they are going to cl…

I haven't listened to Dwarkesh, but I take the complaint to mean that he doesn't probe his guests in interesting ways, not so much that he doesn't criticize his guests. If you aren't guiding the conversation into interesting corners then that seems like a problem.

Agree

Re: Meta Llama 3

#308

Post by farco12 » Thu, Apr 18, 2024, 6:37 PM UTC

swalsh wrote: post #279

eigenvalue wrote: post #231

I just want to express how grateful I am that Zuck and Yann and the rest of the Meta team have adopted an open approach and are sharing the model weights, the tokenizer, information about the training data, etc. They, more than anyone else, are responsible for the explosion of open research and improvement that has happened with things like llama.cpp that now allow you to run quite decent models locally on consumer h…

Why is Meta doing it though? This is an astronomical investment. What do they gain from it?

Mark probably figured Meta would gain knowledge and experience more rapidly if they threw Llama out in the wild while they caught up to the performance of the bigger & better closed source models. It helps that unlike their competition, these models aren't a threat to Meta's revenue streams and they don't have an existing enterprise software business that would seek to immediately monetize this work.

Re: Meta Llama 3

#309

Post by evnc » Thu, Apr 18, 2024, 6:37 PM UTC

swalsh wrote: post #279

eigenvalue wrote: post #231

I just want to express how grateful I am that Zuck and Yann and the rest of the Meta team have adopted an open approach and are sharing the model weights, the tokenizer, information about the training data, etc. They, more than anyone else, are responsible for the explosion of open research and improvement that has happened with things like llama.cpp that now allow you to run quite decent models locally on consumer h…

Why is Meta doing it though? This is an astronomical investment. What do they gain from it?

They're commoditizing their complement [0][1], inasmuch as LLMs are a complement of social media and advertising (which I think they are).

They've made it harder for competitors like Google or TikTok to compete with Meta on the basis of "we have a super secret proprietary AI that no one else has that's leagues better than anything else". If everyone has access to a high quality AI (perhaps not the world's best, but competitive), then no one -- including their competitors -- has a competitive advantage from having exclusive access to high quality AI.

[0]: https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/

[1]: https://gwern.net/complement

Re: Meta Llama 3

#310

Post by hnav » Thu, Apr 18, 2024, 6:39 PM UTC

pellucide wrote: post #288

From the article >We made several new observations on scaling behavior during the development of Llama 3. For example, while the Chinchilla-optimal amount of training compute for an 8B parameter model corresponds to ~200B tokens, we found that model performance continues to improve even after the model is trained on two orders of magnitude more data. Both our 8B and 70B parameter models continued to improve log-linea…

They're saying with this architecture there's a tradeoff between training and inference cost where a 10x smaller model (much cheaper to run inference) can match a bigger model if the smaller is trained on 100x data (much more expensive to train) and that the improvement continues log-linearly.