The instant generation of pictures as you type in meta.ai is really impressive!
Meta Llama 3
111–120 of 965 posts
Re: Meta Llama 3
#112I was curious how the numbers compare to GPT-4 in the paid ChatGPT Plus, since they don't compare directly themselves. Llama 3 8B Llama 3 70B GPT-4 MMLU 68.4 82.0 86.5 GPQA 34.2 39.5 49.1 MATH 30.0 50.4 72.2 HumanEval 62.2 81.7 87.6 DROP 58.4 79.7 85.4 Note that the free version of ChatGPT that most people use is based on GPT-3.5 which is much worse than GPT-4. I haven't found comprehensive eval numbers for the lates…
The bottom of https://ai.meta.com/blog/meta-llama-3/ has in-progress results for the 400B model as well. Looks like it's not quite there yet.
Llama 3 400B Base / Instruct
MMLU 84.8 86.1
GPQA - 48.0
MATH - 57.8
HumanEval - 84.1
DROP 83.5 -Re: Meta Llama 3
#113Earlier quoted context omitted.
Why do people keep saying that Claude3 has been nerfed? Their CTO has said on Twitter multiple times that not a single byte has been changed since its launch, so I'm curious why I keep hearing this. edit: having trouble finding the tweet I saw recently, it might have been from their lead engineer and not the CTO.
I suspect that there is some psychological effect going on where people adjust their expectations and start to be more open to noticing flaws after working with it for a while. Seems to be a recurring thing with most models.
The code it writes is getting worse eg. lazy and not updating the function, not following prompts etc. So we can objectively say its getting worse.
Re: Meta Llama 3
#114I was curious how the numbers compare to GPT-4 in the paid ChatGPT Plus, since they don't compare directly themselves. Llama 3 8B Llama 3 70B GPT-4 MMLU 68.4 82.0 86.5 GPQA 34.2 39.5 49.1 MATH 30.0 50.4 72.2 HumanEval 62.2 81.7 87.6 DROP 58.4 79.7 85.4 Note that the free version of ChatGPT that most people use is based on GPT-3.5 which is much worse than GPT-4. I haven't found comprehensive eval numbers for the lates…
Wild considering, GPT-4 is 1.8T.
[deleted]
Re: Meta Llama 3
#115They've got a console for it as well, https://www.meta.ai/ And announcing a lot of integration across the Meta product suite, https://about.fb.com/news/2024/04/meta-ai-assistant-built-wi... Neglected to include comparisons against GPT-4-Turbo or Claude Opus, so I guess it's far from being a frontier model. We'll see how it fares in the LLM Arena.
[flagged]
[deleted]
Re: Meta Llama 3
#116I was curious how the numbers compare to GPT-4 in the paid ChatGPT Plus, since they don't compare directly themselves. Llama 3 8B Llama 3 70B GPT-4 MMLU 68.4 82.0 86.5 GPQA 34.2 39.5 49.1 MATH 30.0 50.4 72.2 HumanEval 62.2 81.7 87.6 DROP 58.4 79.7 85.4 Note that the free version of ChatGPT that most people use is based on GPT-3.5 which is much worse than GPT-4. I haven't found comprehensive eval numbers for the lates…
Wild considering, GPT-4 is 1.8T.
Where did you find this number? Not doubting it, just want to get a better idea of how precise the estimate may be.
Re: Meta Llama 3
#117Earlier quoted context omitted.
Honestly, I swear to god, been working 12 hours a day with these for a year now, llama.cpp, Claude, OpenAI, Mistral, Gemini: The long context window isn't worth much and is currently creating more problems than it's worth for the bigs, with their "unlimited" use pricing models. Let's take Claude 3's web UI as an example. We build it, and go the obvious route: we simply use as much of the context as possible, given ch…
As a user, I've been putting in some long mathematical research papers and asking detailed questions about them in order to understand certain parts better. I feel some benefit from it because it can access the full context of the paper so it is less likely to misunderstand notation that was defined earlier etc.
Same, that's super useful.
Re: Meta Llama 3
#118I am always excited to see these Open Weight models released, I think its very good for the ecosystem and definitely has its place in many situations. However since I use LLMs as a coding assistant (mostly via "rubber duck" debugging and new library exploration) I really don't want to use anything other than the absolutely best in class available now. That continues to be GPT4-turbo (or maybe Claude 3). Does anyone k…
Do you mind my asking, if you're working on private codebases, how you go about using GPT/Claude as a code assistant? I'm just removing IP and pasting into their website's chat interface. I feel like there's got to be something better out there but I don't really know anyone else that's using AI code assistance at all.
Personally I don't paste anything. I ask for code examples that demonstrate what I want, and then I adapt it to my needs. It's definitely less powerful than directly sharing code, but it is what it is.
I also run a personal language model server, but that is far less capable than the models available as services. It can still be better than nothing for code O can't share with APIs.
I also use gpt.el a but for editor integration, but I honestly haven't workeded that into my workflow very much yet.
Re: Meta Llama 3
#119Does it support function/tool calling ?
Re: Meta Llama 3
#120I was curious how the numbers compare to GPT-4 in the paid ChatGPT Plus, since they don't compare directly themselves. Llama 3 8B Llama 3 70B GPT-4 MMLU 68.4 82.0 86.5 GPQA 34.2 39.5 49.1 MATH 30.0 50.4 72.2 HumanEval 62.2 81.7 87.6 DROP 58.4 79.7 85.4 Note that the free version of ChatGPT that most people use is based on GPT-3.5 which is much worse than GPT-4. I haven't found comprehensive eval numbers for the lates…
Has anyone prepared a comparison to Mixtral 8x22B? (Life sure moves fast.)