Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

github.com

Post reply on HN
Print view View original

191–200 of 241 posts

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#191

Post by BoredomIsFun »

culi wrote: post #152
thot_experiment wrote: post #150

Earlier quoted context omitted.

Not true! Mistral is really really good, but I agree that there isn't a single decent open model from the USA.

Mistral is cool and I wish them success but it consistently ranks extremely low on benchmarks while still being expensive. Chinese models like DeepSeek might rank almost as low as Mistral but they are significantly cheaper. And Kimi is the best of both worlds with incredible benchmark results while still being incredibly cheap I know things change rapidly so I'm not counting them out quite yet but I don't see them as…

> it consistently ranks extremely low on benchmarks

As general purpose chatbots small Mistral models are better than comparably sized Chiniese models, as they have better SimpleQA scores and general knowledge of Western culture.

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#193

Post by jychang »

WanderPanda wrote: post #179
danielhanchen wrote: post #157

Earlier quoted context omitted.

Haha for now our primary goal is to expand the market for local AI and educate people on how to do RL, fine-tuning and running quants :)

Amazing work and people should really appreciate that the opportunity costs of your work are immense (given the hype). On another note: I'm a bit paranoid about quantization. I know people are not good at discerning model quality at these levels of "intelligence" anymore, I don't think a vibe check really catches the nuances. How hard would it be to systematically evaluate the different quantizations? E.g. on the Aid…

> How hard would it be to systematically evaluate the different quantizations? E.g. on the Aider benchmark that you used in the past?

Very hard. $$$

The benchmarks are not cheap to run. It'll cost a lot to run them for each quant of each model.

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#195

Post by intrasight »

jimbob45 wrote: post #119
embedding-shape wrote: post #47

Earlier quoted context omitted.

> Why doesn't HF support BitTorrent? Harder to track downloads then. Only when clients hit the tracker would they be able to get download states, and forget about private repositories or the "gated" ones that Meta/Facebook does for their "open" models. Still, if vanity metrics wasn't so important, it'd be a great option. I've even thought of creating my own torrent mirror of HF to provide as a public service, as even…

Wouldn’t it still provide massive benefits if they could convince/coerce their most popular downloaded models to move to torrenting?

Benefit to you, but great downside to the three letter agencies that inject their goods into these models.

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#196

Post by danielhanchen »

WanderPanda wrote: post #179
danielhanchen wrote: post #157

Earlier quoted context omitted.

Haha for now our primary goal is to expand the market for local AI and educate people on how to do RL, fine-tuning and running quants :)

Amazing work and people should really appreciate that the opportunity costs of your work are immense (given the hype). On another note: I'm a bit paranoid about quantization. I know people are not good at discerning model quality at these levels of "intelligence" anymore, I don't think a vibe check really catches the nuances. How hard would it be to systematically evaluate the different quantizations? E.g. on the Aid…

Thanks! Yes we actually did think about that - it can get quite expensive sadly - perplexity benchmarks over short context lengths with small datasets are doable, but it's not an accurate measure sadly. We're actually investigating currently what would be the best efficient course of action on evaluating quants - will keep you posted!

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#197

Post by danielhanchen »

jychang wrote: post #193
WanderPanda wrote: post #179

Earlier quoted context omitted.

Amazing work and people should really appreciate that the opportunity costs of your work are immense (given the hype). On another note: I'm a bit paranoid about quantization. I know people are not good at discerning model quality at these levels of "intelligence" anymore, I don't think a vibe check really catches the nuances. How hard would it be to systematically evaluate the different quantizations? E.g. on the Aid…

> How hard would it be to systematically evaluate the different quantizations? E.g. on the Aider benchmark that you used in the past? Very hard. $$$ The benchmarks are not cheap to run. It'll cost a lot to run them for each quant of each model.

Yes sadly very expensive :( Maybe a select few quants could happen - we're still figuring out what is the most economical and most efficient way to benchmark!

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#198

Post by danielhanchen »

Zetaphor wrote: post #185
WanderPanda wrote: post #179

Earlier quoted context omitted.

Amazing work and people should really appreciate that the opportunity costs of your work are immense (given the hype). On another note: I'm a bit paranoid about quantization. I know people are not good at discerning model quality at these levels of "intelligence" anymore, I don't think a vibe check really catches the nuances. How hard would it be to systematically evaluate the different quantizations? E.g. on the Aid…

This would be amazing

Working on it! :)

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#199

Post by danielhanchen »

swyx wrote: post #175
data-ottawa wrote: post #22

Earlier quoted context omitted.

Can we toss in the work unsloth does too as an unsung hero? They provide excellent documentation and they’re often very quick to get high quality quants up in major formats. They’re a very trustworthy brand.

not that unsung! we've given them our biggest workshop spot every single year we've been able to and will do until they are tired of us https://www.youtube.com/@aiDotEngineer/search?query=unsloth

Appreciate it immensely haha :) Never tired - always excited and pumped for this year!

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#200

Post by zozbot234 »

Aurornis wrote: post #106
zozbot234 wrote: post #21

Earlier quoted context omitted.

> We still need good value hardware to run Kimi/GLM in-house If you stream weights in from SSD storage and freely use swap to extend your KV cache it will be really slow (multiple seconds per token!) but run on basically anything. And that's still really good for stuff that can be computed overnight, perhaps even by batching many requests simultaneously. It gets progressively better as you add more compute, of course…

> it will be really slow (multiple seconds per token!) This is fun for proving that it can be done, but that's 100X slower than hosted models and 1000X slower than GPT-Codex-Spark. That's like going from real time conversation to e-mailing someone who only checks their inbox twice a day if you're lucky.

You'd need real rack-scale/datacenter infrastructure to properly match the hosted models that are keeping everything in fast VRAM at all times, and then you only get reasonable utilization on that by serving requests from many users. The ~100X slower tier is totally okay for experimentation and non-conversational use cases (including some that are more agentic-like!), and you'd reach ~10X (quite usable for conversation) by running something like a good homelab.
Post reply on HN