Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

github.com

Post reply on HN
Print view View original

91–100 of 241 posts

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#91

Post by r_lee »

zozbot234 wrote: post #13
r_lee wrote: post #9

Earlier quoted context omitted.

I think that's because there's less local AI usage now since there's all kinds of image models by the big labs, so there's really no rush of people self hosting stable diffusion etc anymore the space moved from Consumer to Enterprise pretty fast due to models getting bigger

Today's free models are not really bigger when you account for the use of MoE (with ever increasing sparsity, meaning a smaller fraction of active parameters), and better ways of managing KV caching. You can do useful things with very little RAM/VRAM, it just gets slower and slower the more you try to squeeze it where it doesn't quite belong. But that's not a problem if you're willing to wait for every answer.

yeah, but I mean more like the old setups where you'd just load a model on a 4090 or something, even with MoE it's a lot more complex and takes more VRAM, right? like it just seems not justifiable for most hobbyists

but maybe I'm just slightly out of the loop

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#92

Post by taminka »

embedding-shape wrote: post #47
sowbug wrote: post #39

Earlier quoted context omitted.

Why doesn't HF support BitTorrent? I know about hf-torrent and hf_transfer, but those aren't nearly as accessible as a link in the web UI.

> Why doesn't HF support BitTorrent? Harder to track downloads then. Only when clients hit the tracker would they be able to get download states, and forget about private repositories or the "gated" ones that Meta/Facebook does for their "open" models. Still, if vanity metrics wasn't so important, it'd be a great option. I've even thought of creating my own torrent mirror of HF to provide as a public service, as even…

most of the traffic is probably from open weights, just seed those, host private ones as is

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#93

Post by 5o1ecist »

zozbot234 wrote: post #59
heliumtera wrote: post #56

Earlier quoted context omitted.

>Will they ever "sell out"? Oh no, never. Don't worry, the usual investors are very well known for fighting for user autonomy (AMD, Nvidia, Intel,IBM, Qualcomm) They are all very pro consumers and all backers are certainly here for your enjoyment only

These are all big hardware firms, which makes a lot of sense as a classic 'commoditize the complement' play. Not exactly pro-consumer, but not quite anti-consumer either!

[flagged]

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#94

Post by HanClinto »

5o1ecist wrote: post #93
zozbot234 wrote: post #59

Earlier quoted context omitted.

These are all big hardware firms, which makes a lot of sense as a classic 'commoditize the complement' play. Not exactly pro-consumer, but not quite anti-consumer either!

[flagged]

Great points.

Valve is one of my top favorite companies right now. Love the work they're doing, and their products are amazing.

Can hardly wait for the Steam Frame.

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#95

Post by ushakov »

simonw wrote: post #84

It's hard to overstate the impact Georgi Gerganov and llama.cpp have had on the local model space. He pretty much kicked off the revolution in March 2023, making LLaMA work on consumer laptops. Here's that README from March 10th 2023 https://github.com/ggml-org/llama.cpp/blob/775328064e69db1eb... > The main goal is to run the model using 4-bit quantization on a MacBook. [...] This was hacked in an evening - I have no…

i am curious, why are your comments always pinned to the top?

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#96

Post by arcanemachiner »

disiplus wrote: post #35
data-ottawa wrote: post #22

Earlier quoted context omitted.

Can we toss in the work unsloth does too as an unsung hero? They provide excellent documentation and they’re often very quick to get high quality quants up in major formats. They’re a very trustworthy brand.

Yeah, they're the good guys. I suspect the open source work is mostly advertisements for them to sell consulting and services to enterprises. Otherwise, the work they do doesn't make sense to offer for free.

I hope that is exactly what is happening. It benefits them, and it benefits us.

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#97

Post by carbocation »

ushakov wrote: post #95
simonw wrote: post #84

It's hard to overstate the impact Georgi Gerganov and llama.cpp have had on the local model space. He pretty much kicked off the revolution in March 2023, making LLaMA work on consumer laptops. Here's that README from March 10th 2023 https://github.com/ggml-org/llama.cpp/blob/775328064e69db1eb... > The main goal is to run the model using 4-bit quantization on a MacBook. [...] This was hacked in an evening - I have no…

i am curious, why are your comments always pinned to the top?

Because many of us think simonw has discerning taste on this topic and like to read what he has to say about it, so we upvote his comments.

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#98

Post by homarp »

embedding-shape wrote: post #47
sowbug wrote: post #39

Earlier quoted context omitted.

Why doesn't HF support BitTorrent? I know about hf-torrent and hf_transfer, but those aren't nearly as accessible as a link in the web UI.

> Why doesn't HF support BitTorrent? Harder to track downloads then. Only when clients hit the tracker would they be able to get download states, and forget about private repositories or the "gated" ones that Meta/Facebook does for their "open" models. Still, if vanity metrics wasn't so important, it'd be a great option. I've even thought of creating my own torrent mirror of HF to provide as a public service, as even…

how are all the private trackers tracking ratios?

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#99

Post by zozbot234 »

r_lee wrote: post #91
zozbot234 wrote: post #13

Earlier quoted context omitted.

Today's free models are not really bigger when you account for the use of MoE (with ever increasing sparsity, meaning a smaller fraction of active parameters), and better ways of managing KV caching. You can do useful things with very little RAM/VRAM, it just gets slower and slower the more you try to squeeze it where it doesn't quite belong. But that's not a problem if you're willing to wait for every answer.

yeah, but I mean more like the old setups where you'd just load a model on a 4090 or something, even with MoE it's a lot more complex and takes more VRAM, right? like it just seems not justifiable for most hobbyists but maybe I'm just slightly out of the loop

With sparse MoE it's worth running the experts in system RAM since that allows you to transparently use mmap and inactive experts can stay on disk. Of course that's also a slowdown unless you have enough RAM for the full set, but it lets you run much larger models on smaller systems.

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#100

Post by ushakov »

carbocation wrote: post #97
ushakov wrote: post #95

Earlier quoted context omitted.

i am curious, why are your comments always pinned to the top?

Because many of us think simonw has discerning taste on this topic and like to read what he has to say about it, so we upvote his comments.

i don't doubt this. i just find it questionable that one particular poster always gets in the spotlight when AI is the topic - while other conversations in my opinion offer more interesting angles.
Post reply on HN