Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

github.com

Post reply on HN
Print view View original

31–40 of 241 posts

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#31

Post by HPsquared »

zozbot234 wrote: post #21
mythz wrote: post #18

I consider HuggingFace more "Open AI" than OpenAI - one of the few quiet heroes (along with Chinese OSS) helping bring on-premise AI to the masses. I'm old enough to remember when traffic was expensive, so I've no idea how they've managed to offer free hosting for so many models. Hopefully it's backed by a sustainable business model, as the ecosystem would be meaningfully worse without them. We still need good value…

> We still need good value hardware to run Kimi/GLM in-house If you stream weights in from SSD storage and freely use swap to extend your KV cache it will be really slow (multiple seconds per token!) but run on basically anything. And that's still really good for stuff that can be computed overnight, perhaps even by batching many requests simultaneously. It gets progressively better as you add more compute, of course…

At a certain point the energy starts to cost more than renting some GPUs.

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#32

Post by woadwarrior01 »

option wrote: post #29

Isn't HF banned in China? Also, how are many Chinese labs on Twitter all the time? In either case - huge thanks to them for keeping AI open!

HF is indeed banned in China. The Chinese equivalent of HF is ModelScope[1].

[1]: https://modelscope.cn/

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#34

Post by segmondy »

geooff_ wrote: post #5

As someone who's been in the "AI" space for a while its strange how Hugging Face went from one of the biggest name to not a part of the discussion at all.

part of what discussion? anyone in the AI space knows and uses HF, but the public doesn't give a care and why should they? It's just an advanced site were nerds download AI stuff. HF is super valuable with their transformers library, their code, tutorials, smol-models, etc, but how does it translate to investor dollars?

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#35

Post by disiplus »

data-ottawa wrote: post #22
mythz wrote: post #18

I consider HuggingFace more "Open AI" than OpenAI - one of the few quiet heroes (along with Chinese OSS) helping bring on-premise AI to the masses. I'm old enough to remember when traffic was expensive, so I've no idea how they've managed to offer free hosting for so many models. Hopefully it's backed by a sustainable business model, as the ecosystem would be meaningfully worse without them. We still need good value…

Can we toss in the work unsloth does too as an unsung hero? They provide excellent documentation and they’re often very quick to get high quality quants up in major formats. They’re a very trustworthy brand.

Yeah, they're the good guys. I suspect the open source work is mostly advertisements for them to sell consulting and services to enterprises. Otherwise, the work they do doesn't make sense to offer for free.

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#36

Post by disiplus »

option wrote: post #29

Isn't HF banned in China? Also, how are many Chinese labs on Twitter all the time? In either case - huge thanks to them for keeping AI open!

I think in the West we think everything is blocked. But for example, if you book an eSIM, when you visit you already get direct access to Western services because they route it to some other server. Hong Kong is totally different: they basically use WhatsApp and Google Maps, and everything worked when I was there.

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#37

Post by andsoitis »

rvz wrote: post #2

This acquisition is almost the same as the acquisition of Bun by Anthropic. Both $0 revenue "companies", but have created software that is essential to the wider ecosystem and has mindshare value; Bun for Javascript and Ggml for AI models. But of course the VCs needed an exit sooner or later. That was inevitable.

I believe ggml.ai was funded by angel investors, not VC.

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#38

Post by mft_ »

tkp-415 wrote: post #25

Can anyone point me in the direction of getting a model to run locally and efficiently inside something like a Docker container on a system with not so strong computing power (aka a Macbook M1 with 8gb of memory)? Is my only option to invest in a system with more computing power? These local models look great, especially something like https://huggingface.co/AlicanKiraz0/Cybersecurity-BaronLLM_O... for assisting in p…

There’s no way around needing a powerful-enough system to run the model. So you either choose a model that can fit on what you have —i.e. via a small model, or a quantised slightly larger model— or you access more powerful hardware, either by buying it or renting it. (IME you don’t need Docker. For an easy start just install LM Studio and have a play.)

I picked up a second-hand 64GB M1 Max MacBook Pro a while back for not too much money for such experimentation. It’s sufficiently fast at running any LLM models that it can fit in memory, but the gap between those models and Claude is considerable. However, this might be a path for you? It can also run all manner of diffusion models, but there the performance suffers (vs. an older discrete GPU) and you’re waiting sometimes many minutes for an edit or an image.

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#39

Post by sowbug »

mythz wrote: post #18

I consider HuggingFace more "Open AI" than OpenAI - one of the few quiet heroes (along with Chinese OSS) helping bring on-premise AI to the masses. I'm old enough to remember when traffic was expensive, so I've no idea how they've managed to offer free hosting for so many models. Hopefully it's backed by a sustainable business model, as the ecosystem would be meaningfully worse without them. We still need good value…

Why doesn't HF support BitTorrent? I know about hf-torrent and hf_transfer, but those aren't nearly as accessible as a link in the web UI.

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#40

Post by sigbottle »

mft_ wrote: post #38
tkp-415 wrote: post #25

Can anyone point me in the direction of getting a model to run locally and efficiently inside something like a Docker container on a system with not so strong computing power (aka a Macbook M1 with 8gb of memory)? Is my only option to invest in a system with more computing power? These local models look great, especially something like https://huggingface.co/AlicanKiraz0/Cybersecurity-BaronLLM_O... for assisting in p…

There’s no way around needing a powerful-enough system to run the model. So you either choose a model that can fit on what you have —i.e. via a small model, or a quantised slightly larger model— or you access more powerful hardware, either by buying it or renting it. (IME you don’t need Docker. For an easy start just install LM Studio and have a play.) I picked up a second-hand 64GB M1 Max MacBook Pro a while back fo…

Are mac kernels optimized compared to CUDA kernels? I know that the unified GPU approach is inherently slower, but I thought a ton of optimizations were at the kernel level too (CUDA itself is a moat)
Post reply on HN