Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama 3.1

llama.meta.com

Post reply on HN
Print view View original

241–250 of 279 posts

Re: Llama 3.1

#241

Post by Alifatisk »

cubefox wrote: post #126
primaprashant wrote: post #50

I have found Claude 3.5 Sonnet really good for coding tasks along with the artifacts feature and seems like it's still the king on the coding benchmarks

I have found it to be better than GPT-4o at math too, despite the latter being better at several math benchmarks.

Yeah same experience here aswell, I found Sonnet 3.5 to fulfill my task much better than 4o even though 4o scores higher on benchmarks.

Re: Llama 3.1

#242

Post by j_maffe »

ninjin wrote: post #232
j_maffe wrote: post #164

Earlier quoted context omitted.

I think it's great that you're fighting to maintain the term's fundamental meaning. I do, however, think that we need to give credit where credit is due to companies who take actions in the right direction to encourage more companies to do the same. If we blindly protest any positive-impact action by corporations for not being perfect, they'll get the hint and stop trying to appease the community entirely.

I am in agreement. However, I do believe that a large portion of the community here is also missing a key point: Facebook was more open five years ago with their AI research than they are today. I suspect this perspective is because of the massive influx of people into AI around the time of the ChatGPT release. From their point of view, Facebook's move (although dishonestly labelled as something it is not) is a step…

Totally agree. Your suggested response is perfect IMO.

Re: Llama 3.1

#243

Post by weberer »

listic wrote: post #199
foundval wrote: post #74

You can chat with these new models at ultra-low latency at groq.com. 8B and 70B API access is available at console.groq.com. 405B API access for select customers only – GA and 3rd party speed benchmarks soon. If you want to learn more, there is a writeup at https://wow.groq.com/now-available-on-groq-the-largest-and-m... . (disclaimer, I am a Groq employee)

Just checked it out. Is pay-as-you-go API access available at all? It says 'Coming Soon' https://console.groq.com/settings/billing

I've found Bedrock to be nice with pay-as-you-go, but they take a long time to adopt new models.

Re: Llama 3.1

#244

Post by j_maffe »

anotherpaulg wrote: post #228

Llama 3.1 405B instruct is #7 on aider's leaderboard, well behind Claude 3.5 Sonnet & GPT-4o. When using SEARCH/REPLACE to efficiently edit code, it drops to #11. https://aider.chat/docs/leaderboards/ 77.4% claude-3.5-sonnet 75.2% DeepSeek Coder V2 (whole) 72.9% gpt-4o 69.9% DeepSeek Chat V2 0628 68.4% claude-3-opus-20240229 67.7% gpt-4-0613 66.2% llama-3.1-405b-instruct (whole)

Ordinal value doesn't really matter in this case, especially when it's a categorically different option, access-wise. A 10% difference isn't bad at all.

Re: Llama 3.1

#245

Post by lelag »

mi_lk wrote: post #191
lelag wrote: post #58

The 405b model is actually competitive against closed source frontier models. Quick comparison with GPT-4o: +----------------+-------+-------+ | Metric | GPT-4o| Llama | | | | 3.1 | | | | 405B | +----------------+-------+-------+ | MMLU | 88.7 | 88.6 | | GPQA | 53.6 | 51.1 | | MATH | 76.6 | 73.8 | | HumanEval | 90.2 | 89.0 | | MGSM | 90.5 | 91.6 | +----------------+-------+-------+

How do you draw/generate such ascii table?

In the past, I might have used a python library like asciitable to do that.

This time, I just copy pasted the raw metrics I found and asked an LLM to format it as an ASCII table.

Re: Llama 3.1

#246

Post by TeMPOraL »

OrangeMusic wrote: post #237
TeMPOraL wrote: post #227

Earlier quoted context omitted.

In other words, it's everything except the one thing that actually matters .

Maybe, but it doesn't mean it's not open source.

The things that don't matter are, the thing that does isn't. Together, they can hardly be called open source.

Re: Llama 3.1

#247

Post by fishermanbill »

lolinder wrote: post #206
nomel wrote: post #154

Earlier quoted context omitted.

I really wish people would use "open weights" rather than "open source". It's precise and obvious, and leaves an accurate descriptor for actual "open source" models, where the source and methods that that generate the artifact , that is the weights, is open.

It's not precise. People who want to use "open weights" instead of "open source" are focusing on the wrong thing. The weights are, for all practical purposes, source code in their own right. The GPL defines "source code" as "the preferred form of the work for making modifications to it". Almost no one would be capable of reproducing them even if given the source + data. At the same time, the weights are exactly what…

Its not open source. Your definition would make most video games open source - we modify them all the time. The small runtime framework IS open source but that's not much benefit as you cant really modify it hugely because the weights fix it to an implementation.

Re: Llama 3.1

#248

Post by fishermanbill »

nomel wrote: post #166
fngjdflmdflg wrote: post #160

Earlier quoted context omitted.

As far as I know it's not just the weights. it's everything but the dataset. So the code used to generate the weights is also open source.

Is there any other case where "open source" is used for something that can't be reproduced? Seems like a new term is required, in the concept of "open source, non-reproducible artifacts". I suppose language changes. I just prefer it changes towards being more precise, not less.

Yes its "freeware" or any one of the similar terms we've used to refer to free software.

Re: Llama 3.1

#250

Post by dunefox »

dunefox wrote: post #47
netsec_burn wrote: post #41

Today appears to be the day you can run an LLM that is competitive with GPT-4o at home with the right hardware. Incredible for progress and advancement of the technology. Statement from Mark: https://about.fb.com/news/2024/07/open-source-ai-is-the-path...

It's not really competitive though, is it? I tested it and 4o is just better.

Disclaimer: I tested llama3-8B, 3.1 might even as a small model be better, but I so far I have not seen a single small model approach 4o, ime.
Post reply on HN