Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Meta Llama 3

llama.meta.com

Post reply on HN
Print view View original

101–110 of 965 posts

Re: Meta Llama 3

#101

Post by vacuumcl »

refulgentis wrote: post #49
Anuiran wrote: post #14

Awesome, but I am surprised by the constrained context window as it balloons everywhere else. Am I missing something? 8k seems quite low in current landscape.

Honestly, I swear to god, been working 12 hours a day with these for a year now, llama.cpp, Claude, OpenAI, Mistral, Gemini: The long context window isn't worth much and is currently creating more problems than it's worth for the bigs, with their "unlimited" use pricing models. Let's take Claude 3's web UI as an example. We build it, and go the obvious route: we simply use as much of the context as possible, given ch…

As a user, I've been putting in some long mathematical research papers and asking detailed questions about them in order to understand certain parts better. I feel some benefit from it because it can access the full context of the paper so it is less likely to misunderstand notation that was defined earlier etc.

Re: Meta Llama 3

#102

Post by BoorishBears »

fnordlord wrote: post #95
doctoboggan wrote: post #40

I am always excited to see these Open Weight models released, I think its very good for the ecosystem and definitely has its place in many situations. However since I use LLMs as a coding assistant (mostly via "rubber duck" debugging and new library exploration) I really don't want to use anything other than the absolutely best in class available now. That continues to be GPT4-turbo (or maybe Claude 3). Does anyone k…

Do you mind my asking, if you're working on private codebases, how you go about using GPT/Claude as a code assistant? I'm just removing IP and pasting into their website's chat interface. I feel like there's got to be something better out there but I don't really know anyone else that's using AI code assistance at all.

Use the API (or a product that uses the API)

If you can trust Azure or AWS or GCP with your IP, you can get Claude 3 and GPT-4 Turbo through at least one of them

If your IP is so secret you can't do that, then I wouldn't imagine you'd be using the chat interface

Re: Meta Llama 3

#104

Post by oersted »

rileyphone wrote: post #97
oersted wrote: post #19

Earlier quoted context omitted.

For the instruction tuned models, Llama 3 8B is even significantly better than Llama 2 70B!

To be fair, the Llama 2 instruction tuning was notably bad.

I see it more as an indirect signal for how good Llama 3 8B can get after proper fine-tuning by the community.

Re: Meta Llama 3

#105

Post by gliched_robot »

modeless wrote: post #89

I was curious how the numbers compare to GPT-4 in the paid ChatGPT Plus, since they don't compare directly themselves. Llama 3 8B Llama 3 70B GPT-4 MMLU 68.4 82.0 86.5 GPQA 34.2 39.5 49.1 MATH 30.0 50.4 72.2 HumanEval 62.2 81.7 87.6 DROP 58.4 79.7 85.4 Note that the free version of ChatGPT that most people use is based on GPT-3.5 which is much worse than GPT-4. I haven't found comprehensive eval numbers for the lates…

Wild considering, GPT-4 is 1.8T.

Re: Meta Llama 3

#106

Post by erichocean »

trevor-e wrote: post #11
namanyayg wrote: post #7

I'm so surprised that Meta is actually leading the open source AI landscape?! I've used llama2 extensively and can't wait to try out llama3 now. I can't believe that it does better than Claude 3 in benchmarks (though admittedly claude 3 seems to have been nerfed recently) I sure do wish there was more info about how its trained and its training data.

Why do people keep saying that Claude3 has been nerfed? Their CTO has said on Twitter multiple times that not a single byte has been changed since its launch, so I'm curious why I keep hearing this. edit: having trouble finding the tweet I saw recently, it might have been from their lead engineer and not the CTO.

They can change the prompt without changing the model, since the prompt only affects current "attention."

And they do.

Re: Meta Llama 3

#107

Post by wincy »

fnordlord wrote: post #95
doctoboggan wrote: post #40

I am always excited to see these Open Weight models released, I think its very good for the ecosystem and definitely has its place in many situations. However since I use LLMs as a coding assistant (mostly via "rubber duck" debugging and new library exploration) I really don't want to use anything other than the absolutely best in class available now. That continues to be GPT4-turbo (or maybe Claude 3). Does anyone k…

Do you mind my asking, if you're working on private codebases, how you go about using GPT/Claude as a code assistant? I'm just removing IP and pasting into their website's chat interface. I feel like there's got to be something better out there but I don't really know anyone else that's using AI code assistance at all.

I haven’t used it but I’ve heard https://cursor.sh/ might work?

Re: Meta Llama 3

#108

Post by seydor »

oersted wrote: post #19
minimaxir wrote: post #5

The model card has the benchmark results relative to other Llama models including Llama 2: https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md... The dramatic performance increase of Llama 3 relative to Llama 2 (even Llama 2 13B!) is very impressive. Doubling the context window to 8k will open a lot of new oppertunities too.

For the instruction tuned models, Llama 3 8B is even significantly better than Llama 2 70B!

how much vram does the 8B model use?

Re: Meta Llama 3

#109

Post by colesantiago »

namanyayg wrote: post #7

I'm so surprised that Meta is actually leading the open source AI landscape?! I've used llama2 extensively and can't wait to try out llama3 now. I can't believe that it does better than Claude 3 in benchmarks (though admittedly claude 3 seems to have been nerfed recently) I sure do wish there was more info about how its trained and its training data.

No surprise, Meta AI has been there since 2013 as FAIR and Meta has been doing open source ever since Facebook was made.

I am surprised that this is a surprise to some, it just that some have not been paying attention.

https://en.wikipedia.org/wiki/Meta_AI

Re: Meta Llama 3

#110

Post by observationist »

robertlagrant wrote: post #80
observationist wrote: post #67

https://github.com/meta-llama/llama3/blob/main/LICENSE Llama is not open source. It's corporate freeware with some generous allowances. Open source licenses are a well defined thing. Meta marketing saying otherwise doesn't mean they get to usurp the meaning of a well understood and commonly used understanding of the term "open source." https://opensource.org/license Nothing about Meta's license is open source. It's a…

What is "source" regarding an LLM? Public training data and initial parameters?

The parameters and the license. Mistral uses Apache 2.0, a neatly permissive open source license. As such, it's an open source model.

Models are similar to code you might run on a compiled vm or native operating system. Llama.cpp is to a model as Python is to a python script. The license lays out the rights and responsibilities of the users of the software, or the model, in this case. The training data, process, pipeline to build the model in the first place is a distinct and separate thing from the models themselves. It'd be nice if those were open, too, but when dealing with just the model:

If it uses an OSI recognized open source license, it is an open source model. If it doesn't use an OSI recognized open source license, it's not.

Llama is not open source. It's corporate freeware.

Post reply on HN