Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

github.com

Post reply on HN
Print view View original

201–210 of 241 posts

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#201

Post by toofy »

razster wrote: post #108
Tepix wrote: post #50

Earlier quoted context omitted.

It's insane how much traffic HF must be pushing out of the door. I routinely download models that are hundreds of gigabytes in size from them. A fantastic service to the sovererign AI community.

My fear is that these large "AI" companies will lobby to have these open source options removed or banned, growing concern. I'm not sure how else to explain how much I enjoy using what HF provides, I religiously browse their site for new and exciting models to try.

it’s only a matter of time. we have all seen first hand how … wrong … these companies behave, almost on a regular basis.

there’s a small tinfoil hat part of me that suspects part of their obscene investments and cornering the hardware market is driven by an conscious attempt to stop open source local from taking off. they want it all, the money, the control, and to be the only source of information to us.

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#202

Post by zozbot234 »

zargon wrote: post #183
zozbot234 wrote: post #28

Earlier quoted context omitted.

The general rule of thumb is that you should feel free to quantize even as low as 2 bits average if this helps you run a model with more active parameters. Quantized models are not perfect at all, but they're preferable to the models with fewer, bigger parameters. With 8GB usable, you could run models with up to 32B active at heavy quantization.

A large model (100B+, the more the better) may be acceptable at 2-bit quantization, depending on the task. But not a small model. Especially not for technical tasks. On top of that, one still needs room for OS, software and KV cache. 8GB is just not very useful for local LLMs. That said, it can still be entertaining to try out a 4-bit 8B model for the fun of it.

100B+ is the amount of total parameters, whereas what matters here is active - very different for sparse MoE models. You're right that there's some overhead for the OS/software stack but it's not that much. KV-cache is a good candidate for being swapped out, since it only gets a limited amount of writes per emitted token.

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#203

Post by chucksmash »

RussianCow wrote: post #137
5o1ecist wrote: post #93

Earlier quoted context omitted.

[flagged]

> By law they are required to do whatever they can to maximize profits. I know it's a nit-pick, but I hate that this always gets brought up when it's not actually true. Public corporations face pressure from investors to maximize returns, sure, but there is no law stating that they have to maximize profits at all costs. Public companies can (and often do) act against the interest of immediate profits for some other g…

It's a worthwhile point to make because if people believe that misconception then it lets companies wash their hands of flagrantly bad behavior. "Gosh, we should really get around to changing the law that makes them act that way."

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#204

Post by illusive4080 »

danielhanchen wrote: post #197
jychang wrote: post #193

Earlier quoted context omitted.

> How hard would it be to systematically evaluate the different quantizations? E.g. on the Aider benchmark that you used in the past? Very hard. $$$ The benchmarks are not cheap to run. It'll cost a lot to run them for each quant of each model.

Yes sadly very expensive :( Maybe a select few quants could happen - we're still figuring out what is the most economical and most efficient way to benchmark!

Roughly how much does it cost to run one of the popular benchmarks? Are we talking $1,000, $10,000, or $100k?

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#206

Post by magicalhippo »

ushakov wrote: post #95
simonw wrote: post #84

It's hard to overstate the impact Georgi Gerganov and llama.cpp have had on the local model space. He pretty much kicked off the revolution in March 2023, making LLaMA work on consumer laptops. Here's that README from March 10th 2023 https://github.com/ggml-org/llama.cpp/blob/775328064e69db1eb... > The main goal is to run the model using 4-bit quantization on a MacBook. [...] This was hacked in an evening - I have no…

i am curious, why are your comments always pinned to the top?

New comments get a boost, and as such are frequently near the top just due to that. Frequent upvotes also boosts. There might be other factors.

However these things are dynamic and change over time. As I read the discussion just now, the GP comment was the ~5th top-level comment.

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#207

Post by noisy_boy »

car wrote: post #186

So great to see my two favorite Open Source AI projects/companies joining forces. Since I don't see it mentioned here, LlamaBarn is an awesome little—but mighty—MacOS menubar program, making access to llama.cpp's great web UI and downloading of tastefully curated models easy as pie. It automatically determines the available model- and context-sizes based on available RAM. https://github.com/ggml-org/LlamaBarn Downloa…

Github is showing me unicorn - is there an Linux equivalent? I have a old Thinkpad with a puny Nvidia GPU, can I hope to find anything useful to run on that?

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#208

Post by cpburns2009 »

culi wrote: post #188
Eupolemos wrote: post #163

Earlier quoted context omitted.

Why are you talking price when we are talking local AI? That doesn't make any sense to me. Am I missing something?

Your electricity is free?

If you have the hardware to run expensive models, is the cost of electricity much of a factor? According to Google, the average price in the Silicon Valley Area is $0.448 per kWh. An RTX 5090 costs about $4,000 and has a peak power consumption of 1000 W. Maxing out that GPU for a whole year would cost $3,925 at that rate. It's not particularly more expensive than that hardware itself.

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#210

Post by zargon »

zozbot234 wrote: post #202
zargon wrote: post #183

Earlier quoted context omitted.

A large model (100B+, the more the better) may be acceptable at 2-bit quantization, depending on the task. But not a small model. Especially not for technical tasks. On top of that, one still needs room for OS, software and KV cache. 8GB is just not very useful for local LLMs. That said, it can still be entertaining to try out a 4-bit 8B model for the fun of it.

100B+ is the amount of total parameters, whereas what matters here is active - very different for sparse MoE models. You're right that there's some overhead for the OS/software stack but it's not that much. KV-cache is a good candidate for being swapped out, since it only gets a limited amount of writes per emitted token.

Total parameters, not active parameters, is the property that matters for model robustness under extreme quantization.

Once you're swapping from disk, the performance will be quite unusable for most people. And for local inference, KV cache is the worst possible choice to put on disk.

Post reply on HN