Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

github.com

Post reply on HN
Print view View original

81–90 of 241 posts

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#81

Post by liuliu »

logicallee wrote: post #70
liuliu wrote: post #49

Earlier quoted context omitted.

> We have a local model we would like to distribute but don't have a good CDN. That is not true. I am serving models off Cloudflare R2. It is 1 petabyte per month in egress use and I basically pay peanuts (~$200 everything included).

1 petabyte per month is 1 million downloads of a 1 GB file. We intend to scale to more than 1 million downloads per month. We have a specific scaling architecture in mind. We're qualified to say this because we've ported a billion parameter model to run in your browser - fast - on either webgpu or wasm. (You can see us doing it live at the youtube link in my comment above.) There is a lot of demand for that.

The bandwidth is free on Cloudflare R2. I paid money for storage (~10TiB storage of different models). If you only host 1GiB file there, you are only paying $0.01 per month I believe.

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#82

Post by Hamuko »

tkp-415 wrote: post #25

Can anyone point me in the direction of getting a model to run locally and efficiently inside something like a Docker container on a system with not so strong computing power (aka a Macbook M1 with 8gb of memory)? Is my only option to invest in a system with more computing power? These local models look great, especially something like https://huggingface.co/AlicanKiraz0/Cybersecurity-BaronLLM_O... for assisting in p…

I tried to run some models on my M1 Max (32 GB) Mac Studio and it was a pretty miserable experience. Slow performance and awful results.

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#83

Post by bityard »

HanClinto wrote: post #7

I'm regularly amazed that HuggingFace is able to make money. It does so much good for the world. How solid is its business model? Is it long-term viable? Will they ever "sell out"?

Their business model is essentially the same as GitHub. Host lots of stuff for free and build a community around it, sell the upscaled/private version to businesses. They are already profitable.

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#84

Post by simonw »

It's hard to overstate the impact Georgi Gerganov and llama.cpp have had on the local model space. He pretty much kicked off the revolution in March 2023, making LLaMA work on consumer laptops.

Here's that README from March 10th 2023 https://github.com/ggml-org/llama.cpp/blob/775328064e69db1eb...

> The main goal is to run the model using 4-bit quantization on a MacBook. [...] This was hacked in an evening - I have no idea if it works correctly.

Hugging Face have been a great open source steward of Transformers, I'm optimistic the same will be true for GGML.

I wrote a bit about this here: https://simonwillison.net/2026/Feb/20/ggmlai-joins-hugging-f...

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#85

Post by zozbot234 »

0xbadcafebee wrote: post #76

> The community will continue to operate fully autonomously and make technical and architectural decisions as usual. Hugging Face is providing the project with long-term sustainable resources, improving the chances of the project to grow and thrive. The project will continue to be 100% open-source and community driven as it is now. I want this to be true, but business interests win out in the end. Llama.cpp is now th…

Llama.cpp is an open source project that anyone can fork as needed, so any "control" over it really only extends to facilitating development of certain features.

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#86

Post by [deleted] »

data-ottawa wrote: post #22
mythz wrote: post #18

I consider HuggingFace more "Open AI" than OpenAI - one of the few quiet heroes (along with Chinese OSS) helping bring on-premise AI to the masses. I'm old enough to remember when traffic was expensive, so I've no idea how they've managed to offer free hosting for so many models. Hopefully it's backed by a sustainable business model, as the ecosystem would be meaningfully worse without them. We still need good value…

Can we toss in the work unsloth does too as an unsung hero? They provide excellent documentation and they’re often very quick to get high quality quants up in major formats. They’re a very trustworthy brand.

[deleted]

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#87

Post by HanClinto »

bityard wrote: post #83
HanClinto wrote: post #7

I'm regularly amazed that HuggingFace is able to make money. It does so much good for the world. How solid is its business model? Is it long-term viable? Will they ever "sell out"?

Their business model is essentially the same as GitHub. Host lots of stuff for free and build a community around it, sell the upscaled/private version to businesses. They are already profitable.

This is what Sourceforge did too, and they still had the DevShare adware thing didn't they?

GitHub is great -- huge fan. To some degree they "sold out" to Microsoft and things could have gone more south, but thankfully Microsoft has ruled them with a very kind hand, and overall I'm extremely happy with the way they've handled it.

I guess I always retain a bit of skepticism with such things, and the long-term viability and goodness of such things never feels totally sure.

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#88

Post by 0xbadcafebee »

tkp-415 wrote: post #25

Can anyone point me in the direction of getting a model to run locally and efficiently inside something like a Docker container on a system with not so strong computing power (aka a Macbook M1 with 8gb of memory)? Is my only option to invest in a system with more computing power? These local models look great, especially something like https://huggingface.co/AlicanKiraz0/Cybersecurity-BaronLLM_O... for assisting in p…

8GB is not enough to do complex reasoning, but you could do very small simple things. Models like Whisper, SmolVLM, Quen2.5-0.5B, Phi-3-mini, Granite-4.0-micro, Mistral-7B, Gemma3, Llama-3.2 all work on very little memory. Tiny models can do a lot if you tune/train them. They also need to be used differently: system prompt preloaded with information, few-shot examples, reasoning guidance, single-task purpose, strict output guidelines. See https://github.com/acon96/home-llm for an example. For each small model, check if Unsloth has a tuned version of it; it reduces your memory footprint and makes inference faster.

For your Mac, you can use Ollama, or MLX (Mac ARM specific, requires different engine and different model disk format, but is faster). Ramalama may help fix bugs or ease the process w/MLX. Use either Docker Desktop or Colima for the VM + Docker.

For today's coding & reasoning models, you need a minimum of 32GB VRAM combined (graphics + system), the more in GPU the better. Copying memory between CPU and GPU is too slow so the model needs to "live" in GPU space. If it can't fit all in GPU space, your CPU has to work hard, and you get a space heater. That Mac M1 will do 5-10 tokens/s with 8GB (and CPU on full blast), or 50 token/s with 32GB RAM (CPU idling). And now you know why there's a RAM shortage.

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#89

Post by vardalab »

Tepix wrote: post #50
mythz wrote: post #18

I consider HuggingFace more "Open AI" than OpenAI - one of the few quiet heroes (along with Chinese OSS) helping bring on-premise AI to the masses. I'm old enough to remember when traffic was expensive, so I've no idea how they've managed to offer free hosting for so many models. Hopefully it's backed by a sustainable business model, as the ecosystem would be meaningfully worse without them. We still need good value…

It's insane how much traffic HF must be pushing out of the door. I routinely download models that are hundreds of gigabytes in size from them. A fantastic service to the sovererign AI community.

Yup, I have downloaded probably a terabyte in the last week, especially with the Step 3.5 model being released and Minimax quants. I wonder what my ISP thinks. I hope they don't cut me off. They gave me a fast lane, they better let me use it, lol

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#90

Post by vardalab »

HPsquared wrote: post #31
zozbot234 wrote: post #21

Earlier quoted context omitted.

> We still need good value hardware to run Kimi/GLM in-house If you stream weights in from SSD storage and freely use swap to extend your KV cache it will be really slow (multiple seconds per token!) but run on basically anything. And that's still really good for stuff that can be computed overnight, perhaps even by batching many requests simultaneously. It gets progressively better as you add more compute, of course…

At a certain point the energy starts to cost more than renting some GPUs.

Yeah, that is hard to argue with because I just go to OpenRouter and play around with a lot of models before I decide which ones I like. But there's something special about running it locally in your basement
Post reply on HN