Live data from Hacker News

Meta Llama 3

llama.meta.com

541–550 of 965 posts

Re: Meta Llama 3

#541

I just want to express how grateful I am that Zuck and Yann and the rest of the Meta team have adopted an open approach and are sharing the model weights, the tokenizer, information about the training data, etc. They, more than anyone else, are responsible for the explosion of open research and improvement that has happened with things like llama.cpp that now allow you to run quite decent models locally on consumer h…

It's crazy how the managerial executive class seems to resent the vital essence of their own companies. Based on the behavior, nature, stated beliefs and interviews I've seen of most tech CEOs and CEOs in general, there seems to be almost a natural aversion to talking about things in non hyper-abstracted terms.

I get the feeling that the nature of the corporate world is often better understood as a series of rituals to create the illusion of the necessity of the capitalist hierarchy itself. (not that this is exclusive to capitalism, this exists in politics and any system that becomes somewhat self-sustaining) More important than a company doing well is the capacity to use the company as an image/lifestyle enhancement tool for those at the top. So many companies run almost mindlessly as somewhat autonomous machines, allowing pretense and personal egoic myth-making to win over the purpose of the company in the first place.

I think this is why Elon, Mark, Jensen, etc. have done so well. They don't perceive their position as founder/CEOs as a class position: a level above the normal lot that requires a lack of caring for tangible matters. They see their companies as ways of making things happen, for better or for worse.

Re: Meta Llama 3

#542

Earlier quoted context omitted.

They also stated that they are still training larger variants that will be more competitive: > Our largest models are over 400B parameters and, while these models are still training, our team is excited about how they’re trending. Over the coming months, we’ll release multiple models with new capabilities including multimodality, the ability to converse in multiple languages, a much longer context window, and stronge…

Anyone have any informed guesstimations as to where we might expect a 400b parameter model for llama 3 to land benchmark wise and performance wise, relative to this current llama 3 and relative to GPT-4? I understand that parameters mean different things for different models, and llama two had 70 b parameters, so I'm wondering if anyone can contribute some guesstimation as to what might be expected with the larger mo…

They are aiming to beat the current GPT4 and stand a fair chance, they are unlikly to hold the crown for long.

Re: Meta Llama 3

#543

Earlier quoted context omitted.

Where did you find this number? Not doubting it, just want to get a better idea of how precise the estimate may be.

I'm not OP, but George Hotz said in his lex friedman podcast a while back that it was an MoE of 8 250B. subtract out duplication of attention nodes, and you get something right around 1.8T

I'm pretty sure he suggested it was a 16 way 110 MoE

Re: Meta Llama 3

#544

I was curious how the numbers compare to GPT-4 in the paid ChatGPT Plus, since they don't compare directly themselves. Llama 3 8B Llama 3 70B GPT-4 MMLU 68.4 82.0 86.5 GPQA 34.2 39.5 49.1 MATH 30.0 50.4 72.2 HumanEval 62.2 81.7 87.6 DROP 58.4 79.7 85.4 Note that the free version of ChatGPT that most people use is based on GPT-3.5 which is much worse than GPT-4. I haven't found comprehensive eval numbers for the lates…

Wild considering, GPT-4 is 1.8T.

The original GPT4 may have been around that size (16x 110B).

But it's pretty clear GPT4 Turbo is a smaller and heavily quantized model.

Re: Meta Llama 3

#545

I downloaded llama3:8b-instruct-q4_0 in ollama and said "hi" and it answered with 10 screen long rant. This is an exerpt. > You're welcome! It was a pleasure chatting with you. Bye for now!assistant > Bye for now!assistant > Bye!assistant

Sorry about this. It should be fixed now. There was an issue with the vocabulary we had to fix and re-push!

  ollama pull llama3:8b-instruct-q4_0
should update it.

Re: Meta Llama 3

#546
post #240

Earlier quoted context omitted.

You can see from Zuck's interviews that he is still an engineer at heart. Every other big tech company has lost that kind of leadership.

Also, being open source adds phenomenal value for Meta: 1. It attracts the world's best academic talent, who deeply want their work shared. AI experts can join any company, so ones which commit to open AI have a huge advantage. 2. Having armies of SWEs contributing millions of free labor hours to test/fix/improve/expand your stuff is incredible. 3. The industry standardizes around their tech, driving down costs and d…

How does that work? Nobody will be able to run the big models who doesn't have a big data center or lots of rent money to burn. How is it going to matter to most of us?

It seems similar to open chip designs - irrelevant to people who are going to buy whatever chips they use anyway. Maybe I'll design a circuit board, but no deeper than that.

Modern civilization means depending on supply chains.

Re: Meta Llama 3

#547

Open weight models do more for AI safety than any other measure by far, as the most serious threath is never going to be misuse, but abuse of unequal access.

Agreed. Still not open data though, is it? i.e., we don't have access to the same data they used to train, which is useful for both finetuning and studying the model.

To be fair, this is part of the reason it is so valuable. Meta has access to absolutely stupendous amounts of private training data. If you forced them to make the data available you would have to settle for a tiny training set by comparison. The value of this is exactly that you are getting access to the result of training on data that you couldn't otherwise access at all.

Re: Meta Llama 3

#549

Just a quick observation: it seems to not mention commercial companies (or at least be biased against it). I tried executing "what are popular design tools with an infinite canvas" against both meta.ai and OpenAI. OpenAI returned what you would expect, Figma Sketch etc. But MetaAI only returned free/open-source software https://x.com/seveibar/status/1781042926430437404

Interesting, I'm playing with it and I asked it what SIEMs are and it gave examples of companies/solutions, including Splunk and RSA Security Analytics.
Post reply on HN