571–580 of 965 posts
"You’ll also soon be able to test multimodal Meta AI on our Ray-Ban Meta smart glasses." Now this is interesting. I've been thinking for some time now that traditional computer/smartphone interfaces are on the way out for all but a few niche applications. Instead, everyone will have their own AI assistant, which you'll interact with naturally the same way as you interact with other people. Need something visual? Just…
I was curious how the numbers compare to GPT-4 in the paid ChatGPT Plus, since they don't compare directly themselves. Llama 3 8B Llama 3 70B GPT-4 MMLU 68.4 82.0 86.5 GPQA 34.2 39.5 49.1 MATH 30.0 50.4 72.2 HumanEval 62.2 81.7 87.6 DROP 58.4 79.7 85.4 Note that the free version of ChatGPT that most people use is based on GPT-3.5 which is much worse than GPT-4. I haven't found comprehensive eval numbers for the lates…
Earlier quoted context omitted.
Also, being open source adds phenomenal value for Meta: 1. It attracts the world's best academic talent, who deeply want their work shared. AI experts can join any company, so ones which commit to open AI have a huge advantage. 2. Having armies of SWEs contributing millions of free labor hours to test/fix/improve/expand your stuff is incredible. 3. The industry standardizes around their tech, driving down costs and d…
How does that work? Nobody will be able to run the big models who doesn't have a big data center or lots of rent money to burn. How is it going to matter to most of us? It seems similar to open chip designs - irrelevant to people who are going to buy whatever chips they use anyway. Maybe I'll design a circuit board, but no deeper than that. Modern civilization means depending on supply chains.
Earlier quoted context omitted.
Anyone have any informed guesstimations as to where we might expect a 400b parameter model for llama 3 to land benchmark wise and performance wise, relative to this current llama 3 and relative to GPT-4? I understand that parameters mean different things for different models, and llama two had 70 b parameters, so I'm wondering if anyone can contribute some guesstimation as to what might be expected with the larger mo…
They are aiming to beat the current GPT4 and stand a fair chance, they are unlikly to hold the crown for long.
They've got a console for it as well, https://www.meta.ai/ And announcing a lot of integration across the Meta product suite, https://about.fb.com/news/2024/04/meta-ai-assistant-built-wi... Neglected to include comparisons against GPT-4-Turbo or Claude Opus, so I guess it's far from being a frontier model. We'll see how it fares in the LLM Arena.
Earlier quoted context omitted.
> in general Open Source doesn't matter for almost everyone? Most of the qualities that come with open source (which also come with llama 3), matter a lot. But no, it is not a binary, yes or no thing, where something is either open source and useful or not. Instead, there is a very wide spectrum is licensing agreements. And even if something does not fit the very specific and exact definition of open source, it can s…
If I build a train, put it into service, and say to the passengers “this has 99.9% of the required parts from the design”, would you ride on that train? Would you consider that train 99.9% as good at being a train? Or is it all-or-nothing? I don’t necessarily disagree with your point about there still being value in mostly-open software, but I want to challenge your notion that you still get most of the benefit. I th…
Well if the missing piece is a cup holder on the train, yes absolutely! It would absolutely be as good as the binary "contains a cup holder" train design.
So the point stands. For almost everyone, these almost open source licenses are good enough for their usecase and the limitations apply to almost noone.
And you have chosen a wonderful example that exactly proves my point. In your example, the incorrect people are claiming that "99.9%" of a train is dangerous to ride in, while ignoring the fact that the missing .1% is the cup holders.
> You can imagine a big legacy bank
Fortunately, most people aren't running a big legacy bank. So the point stands, once again.
> It’s a giant barrier to their adoption
Only if you are at a big legacy bank, in your example, or similar. If you aren't in that very small percentage of the market, you are fine.
Earlier quoted context omitted.
[flagged]
I haven't tried Llama 3 yet, but Llama 2 is indeed extremely "safe." (I'm old enough to remember when AI safety was about not having AI take over the world and kill all humans, not when it might offend a Puritan's sexual sensibilities or hurt somebody's feelings, so I hate using the word "safe" for it, but I can't think of a better word that others would understand). It's not quite as bad as Gemini, but in the same c…
Earlier quoted context omitted.
[flagged]
I haven't tried Llama 3 yet, but Llama 2 is indeed extremely "safe." (I'm old enough to remember when AI safety was about not having AI take over the world and kill all humans, not when it might offend a Puritan's sexual sensibilities or hurt somebody's feelings, so I hate using the word "safe" for it, but I can't think of a better word that others would understand). It's not quite as bad as Gemini, but in the same c…
Earlier quoted context omitted.
Claude has the same restriction [0], the whole of Europe (except Albania) is excluded. Somehow I don't think it is a retaliation against Europe for fining Meta and Google. I could be wrong, but a business decision seems more likely, like keeping usage down to a manageable level in an initial phase. Still, curious to understand why, should anyone here know more. [0] https://www.anthropic.com/claude-ai-locations
It's because of regulations! The same reason that Threads was launched with a delay in EU. It simply takes a lot of work to comply with EU regulations, and by no surprise will we see these launches happen outside of EU first.
But if you say "It's because of regulations!" I hope you have a source to back that up.
https://github.com/meta-llama/llama3/blob/main/LICENSE Llama is not open source. It's corporate freeware with some generous allowances. Open source licenses are a well defined thing. Meta marketing saying otherwise doesn't mean they get to usurp the meaning of a well understood and commonly used understanding of the term "open source." https://opensource.org/license Nothing about Meta's license is open source. It's a…
What is "source" regarding an LLM? Public training data and initial parameters?
The starting point is the ability to run the LLM as you wish, for any purpose - so if a license prohibits some uses and you have to start any usage with thinking whether it's permitted or not, that's a fail.
Then the freedom where "source" matters is the practical freedom to change the behavior so it does your computing as you wish. And that's a bit tricky - since one interpretation would require having the training data, training code and parameters; but for current LLMs the training hardware and cost of running it is a major practical limitation, so much that one could argue that the ability to change the behavior (which is the core freedom that we'd like) is separate from the ability to recreate the model, and would be more relevant in the context of the "instruction training" which happens after the main training, is the main determiner of behavior (as opposed to capability), and so the main "source would be the data for that (instruct training data, and the model weights before that finetuning) so that you can fine-tune the model on different instructions, which requires much less resources than training it from scratch, and don't have to start with the instructions and values imposed on the LLM by someone else.