Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama 3.1

llama.meta.com

Post reply on HN
Print view View original

271–279 of 279 posts

Re: Llama 3.1

#271

Post by drdaeman »

michaelt wrote: post #162
drdaeman wrote: post #122

Earlier quoted context omitted.

I thought home use is whatever fits in 24GB (a single 3090 GPU, which is pretty affordable), not 8 or 16. 30B models fit.

While some home users do indeed have 24GB of vram, the fact is a 4090 costs $1700 Such models will never top the number of downloads charts, or the community hype, as there’s just loads more people who can use the smaller models. And if you can afford one 4090 you can probably afford two.

Why 4090, though? I read (and agree) that 3090 is generally considered to be the best bang for the buck: 24GB, priced at $800-1000 range, and giving decent TPS for LLMs.

Re: Llama 3.1

#272

Post by hrpnk »

jiriro wrote: post #270

Can this Llama process ~1GB of custom XML data? And answer queries like: Give all which refer to which refer to an Indo-European .

The model's context is 128k tokens, so you'd have to split the data and analyze in chunks.

Re: Llama 3.1

#273

Post by kromem »

diimdeep wrote: post #52

This 405B seriously need quantization solution like 1.625 bpw ternary packing for BitNet b1.58 https://github.com/ggerganov/llama.cpp/pull/8151

In general this needs to be done across the board.

The perplexity per parameter is higher and the delta grows as it scales.

Not per bit, but per parameter.

Why this is happening really needs more attention and more consideration for pretrained model development right now.

A sleeping giant of a difference in a space where even marginal gains make headlines.

Re: Llama 3.1

#275

Post by noble-lombax »

geepytee wrote: post #94
foundval wrote: post #74

You can chat with these new models at ultra-low latency at groq.com. 8B and 70B API access is available at console.groq.com. 405B API access for select customers only – GA and 3rd party speed benchmarks soon. If you want to learn more, there is a writeup at https://wow.groq.com/now-available-on-groq-the-largest-and-m... . (disclaimer, I am a Groq employee)

We also added Llama 3.1 405B to our VSCode copilot extension for anyone to try coding with it. Free trial gets you 50 messages, no credit card required - https://double.bot (disclaimer, I am the co-founder)

would be great if there was a page showing benchmarks compared to other auto completion tools

Re: Llama 3.1

#276

Post by niutech »

monkeydust wrote: post #95
cchance wrote: post #59

Earlier quoted context omitted.

Super cool, though sadly 405b will be outside most personal usage without cloud providers which sorta defeats the purpose of opensource to some extent atleast sadly, because .. nvidia's rampup of consumer VRAM is glacial

Great for Groq whos already hosting it but at what cost I guess.

Groq provides a limited free tier for now: https://wow.groq.com/now-available-on-groq-the-largest-and-m...

Re: Llama 3.1

#277

Post by niutech »

petercooper wrote: post #53
lolinder wrote: post #44

Earlier quoted context omitted.

> at home with the right hardware Where the right hardware is 10x4090s even at 4 bits quantization. I'm hoping we'll see these models get smaller, but the GPT-4-competitive one isn't really accessible for home use yet. Still amazing that it's available at all, of course!

It's hardly cheap starting at about $10k of hardware, but another potential option appears to be using Exo to spread the model across a few MBPs or Mac Studios: https://x.com/exolabs_/status/1814913116704288870

Or maybe using Distributed Llama? https://github.com/b4rtaz/distributed-llama

Re: Llama 3.1

#278

Post by serverlessmania »

foundval wrote: post #74

You can chat with these new models at ultra-low latency at groq.com. 8B and 70B API access is available at console.groq.com. 405B API access for select customers only – GA and 3rd party speed benchmarks soon. If you want to learn more, there is a writeup at https://wow.groq.com/now-available-on-groq-the-largest-and-m... . (disclaimer, I am a Groq employee)

You can chat with all these models for free and ultra-low latency using this hosted website https://nat.dev/chat for free by GitHub Founder
Post reply on HN