Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

github.com

Post reply on HN
Print view View original

171–180 of 241 posts

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#171

Post by ttoinou »

sigbottle wrote: post #40
mft_ wrote: post #38

Earlier quoted context omitted.

There’s no way around needing a powerful-enough system to run the model. So you either choose a model that can fit on what you have —i.e. via a small model, or a quantised slightly larger model— or you access more powerful hardware, either by buying it or renting it. (IME you don’t need Docker. For an easy start just install LM Studio and have a play.) I picked up a second-hand 64GB M1 Max MacBook Pro a while back fo…

Are mac kernels optimized compared to CUDA kernels? I know that the unified GPU approach is inherently slower, but I thought a ton of optimizations were at the kernel level too (CUDA itself is a moat)

There’s this developer called nightmedia who converts a lot of models to apple MLX. I can run Qwen3 coder next at 60 tps on my m4 max. It works

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#172

Post by ttoinou »

dust42 wrote: post #144
karmasimida wrote: post #129

Does local AI have a future? The models are getting ridiculously big and any storage hardware is hoarded by few companies for next 2 years and nvidia has stopped making consumer GPU for this year. It seems to me there is no chance local ML is going to be anywhere out of the toy status comparing to closed source ones in short term

I am actually doing now a good part of dev with Qwen3-Coder-Next on an M1 64GB with Qwen Code CLI (a fork of Gemini CLI). I very much like a) to have an idea how much tokens I use and b) be independent of VC financed token machines and c) I can use it on a plane/train Also I never have to wait in a queue, nor will I be told to wait for a few hours. And I get many answers in a second. I don't do full vibe coding with…

How much tokens per seconds are you getting ?

Whats the advantage of qwen code cli over opencode ?

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#173

Post by 3eb7988a1663 »

fc417fc802 wrote: post #123
vardalab wrote: post #89

Earlier quoted context omitted.

Yup, I have downloaded probably a terabyte in the last week, especially with the Step 3.5 model being released and Minimax quants. I wonder what my ISP thinks. I hope they don't cut me off. They gave me a fast lane, they better let me use it, lol

Even fairly restrictive data caps are in the range of 6 Tb per month. P2P at a mere 100 Mb works out to 1 TiB per 24 hours. Hypothetically my ISP will sell me unmetered 10 Gb service but I wonder if they would actually make good on their word ...

I have a 1.2TB cap before you start getting charged extra, so you might need to recalibrate your restrictive level.

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#174

Post by fc417fc802 »

3eb7988a1663 wrote: post #173
fc417fc802 wrote: post #123

Earlier quoted context omitted.

Even fairly restrictive data caps are in the range of 6 Tb per month. P2P at a mere 100 Mb works out to 1 TiB per 24 hours. Hypothetically my ISP will sell me unmetered 10 Gb service but I wonder if they would actually make good on their word ...

I have a 1.2TB cap before you start getting charged extra, so you might need to recalibrate your restrictive level.

Is that with a WISP by chance? Or in a developing country? Or are there really wired providers with such low caps in the western world in this day and age?

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#175

Post by swyx »

data-ottawa wrote: post #22
mythz wrote: post #18

I consider HuggingFace more "Open AI" than OpenAI - one of the few quiet heroes (along with Chinese OSS) helping bring on-premise AI to the masses. I'm old enough to remember when traffic was expensive, so I've no idea how they've managed to offer free hosting for so many models. Hopefully it's backed by a sustainable business model, as the ecosystem would be meaningfully worse without them. We still need good value…

Can we toss in the work unsloth does too as an unsung hero? They provide excellent documentation and they’re often very quick to get high quality quants up in major formats. They’re a very trustworthy brand.

not that unsung! we've given them our biggest workshop spot every single year we've been able to and will do until they are tired of us https://www.youtube.com/@aiDotEngineer/search?query=unsloth

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#176

Post by swyx »

mattfrommars wrote: post #111

I don’t know if this warrants a separate thread here but I have to ask… How can I realistically get involved the AI development space? I feel left out with what’s going on and living in a bubble where AI is forced into by my employer to make use of it (GitHub Copilot), what is a realistic road map to kinda slowly get into AI development, whatever that means My background is full stack development in Java and React, a…

go thru workshops here https://www.youtube.com/@aiDotEngineer/

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#177

Post by thot_experiment »

culi wrote: post #152
thot_experiment wrote: post #150

Earlier quoted context omitted.

Not true! Mistral is really really good, but I agree that there isn't a single decent open model from the USA.

Mistral is cool and I wish them success but it consistently ranks extremely low on benchmarks while still being expensive. Chinese models like DeepSeek might rank almost as low as Mistral but they are significantly cheaper. And Kimi is the best of both worlds with incredible benchmark results while still being incredibly cheap I know things change rapidly so I'm not counting them out quite yet but I don't see them as…

Sure, benchmarks are fake and I use Mistral over equivalently sized models most of the time because it's better in real life. It runs plenty fast for me, I don't pay for inference.

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#178

Post by dust42 »

ttoinou wrote: post #172
dust42 wrote: post #144

Earlier quoted context omitted.

I am actually doing now a good part of dev with Qwen3-Coder-Next on an M1 64GB with Qwen Code CLI (a fork of Gemini CLI). I very much like a) to have an idea how much tokens I use and b) be independent of VC financed token machines and c) I can use it on a plane/train Also I never have to wait in a queue, nor will I be told to wait for a few hours. And I get many answers in a second. I don't do full vibe coding with…

How much tokens per seconds are you getting ? Whats the advantage of qwen code cli over opencode ?

320 tok/s PP and 42 tok/s TG with 4bit quant and MLX. Llama.cpp was half for this model but afaik has improved a few days ago, I haven't yet tested though.

I have tried many tools locally and was never really happy with any. I tried finally Qwen Code CLI assuming that it would run well with a Qwen model and it does. YMMV, I mostly do javascript and Python. Most important setting was to set the max context size, it then auto compacts before reaching it. I run with 65536 but may raise this a bit.

Last not least OpenCode is VC funded, at some point they will have to make money while Gemini CLI / Qwen CLI are not the primary products of the companies but definitely dog-fooded.

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#179

Post by WanderPanda »

danielhanchen wrote: post #157
disiplus wrote: post #35

Earlier quoted context omitted.

Yeah, they're the good guys. I suspect the open source work is mostly advertisements for them to sell consulting and services to enterprises. Otherwise, the work they do doesn't make sense to offer for free.

Haha for now our primary goal is to expand the market for local AI and educate people on how to do RL, fine-tuning and running quants :)

Amazing work and people should really appreciate that the opportunity costs of your work are immense (given the hype).

On another note: I'm a bit paranoid about quantization. I know people are not good at discerning model quality at these levels of "intelligence" anymore, I don't think a vibe check really catches the nuances. How hard would it be to systematically evaluate the different quantizations? E.g. on the Aider benchmark that you used in the past?

I was recently trying Qwen 3 Coder Next and there are benchmark numbers in your article but they seem to be for the official checkpoint, not the quantized ones. But it is not even really clear (and chatbots confuse them for benchmarks of the quantized versions btw.)

I think systematic/automated benchmarks would really bring the whole effort to the next level. Basically something like the bar chart from the Dynamic Quantization 2.0 article but always updated with all kinds of recent models.

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#180

Post by nagaiaida »

fc417fc802 wrote: post #174
3eb7988a1663 wrote: post #173

Earlier quoted context omitted.

I have a 1.2TB cap before you start getting charged extra, so you might need to recalibrate your restrictive level.

Is that with a WISP by chance? Or in a developing country? Or are there really wired providers with such low caps in the western world in this day and age?

well it's my wired cap a stone's throw from buildings with google cloud logos on the side in a major us city, so...
Post reply on HN