Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DeepSeek 4 Flash local inference engine for Metal

github.com

Post reply on HN
Print view View original

91–100 of 171 posts

Re: DeepSeek 4 Flash local inference engine for Metal

#91

Post by Havoc »

zozbot234 wrote: post #85
Havoc wrote: post #74

Was excited until I realized DS flash is still enormous. Oh well...glad it exists anyway & happy to see antirez still doing fun stuff

It could run viably with SSD offload on Macs with very little memory. You could even exploit batching to make the model almost compute limited even in that challenging setting, seeing as the KV cache is so extremely small (for non-humongous context). In fact, if that approach can be made to work I'd like to see a comparison between DS4 Flash and Pro on the same (Mac) hardware.

>It could run viably with SSD offload on Macs with very little memory

Not really. That's going to land you somewhere in the 0.2-0.5 tokens a second range

Lovely as modern nvmes are they're not memory

Re: DeepSeek 4 Flash local inference engine for Metal

#92

Post by zozbot234 »

Havoc wrote: post #91
zozbot234 wrote: post #85

Earlier quoted context omitted.

It could run viably with SSD offload on Macs with very little memory. You could even exploit batching to make the model almost compute limited even in that challenging setting, seeing as the KV cache is so extremely small (for non-humongous context). In fact, if that approach can be made to work I'd like to see a comparison between DS4 Flash and Pro on the same (Mac) hardware.

>It could run viably with SSD offload on Macs with very little memory Not really. That's going to land you somewhere in the 0.2-0.5 tokens a second range Lovely as modern nvmes are they're not memory

You can run multiple inferences in parallel on the same set of weights, that's what batching is. Given enough parallelization it can be almost entirely compute-limited, at least for small context (max ~10GB per request apparently, but that's for 1M tokens!)

Re: DeepSeek 4 Flash local inference engine for Metal

#93

Post by lhl »

I think especially with the ability for SOTA AI to optimize kernels more people should try their hand at making better inference for their specific hardware.

I have an older W7900 (RDNA3) which, besides 48GB of VRAM, has some pretty decent roofline specs - 123 FP16 TFLOPS/INT8 TOPS, 864 GB/s MBW, but has had notoriously bad support both from AMD (ROCm) as well as llama.cpp.

Recently I decided I'd like to turn the card into a dedicated agentic/coder endpoint and I started tuning a W8A8-INT8 model. Over the course of a few days of autolooping (about 800 iterations using a variety of frontier/SOTA models, Kimi K2.6 did surprisingly well), and I ended up with prefill +20% and decode +50% faster than the best llama.cpp numbers for Qwen3.6 MoE.

I'm currently grinding MTP and DFlash optimization on it, but I've been pretty pleased with the results, and will probably try Gemma 4 next.

Re: DeepSeek 4 Flash local inference engine for Metal

#94

Post by crazygringo »

cortesoft wrote: post #54
minimaxir wrote: post #17

Earlier quoted context omitted.

"Data centers for LLMs are technically more energy efficient per-user than self-hosting LLM models due to economies-of-scale" is a data point the internet isn't ready for.

I thought this is a pretty generally accepted fact?

I've seen plenty of people on HN claim that LLM's running on their phones is the obvious future in terms of not just privacy but also efficiency, i.e. better along every possible metric.

They don't usually go into much detail, but the impression I get is that they think data centers are energy monsters full of overheated GPU's that need to be constantly replaced, while your phone is full of mostly unused compute capacity and will barely break a sweat if it's only serving queries for a single user at a time.

They don't seem to give much thought to the energy usage per user (or what this will potentially do to your phone battery), or how different phone-sized vs data center-sized models are in terms of capability.

Re: DeepSeek 4 Flash local inference engine for Metal

#95

Post by p_stuart82 »

kgeist wrote: post #36

Heh, I made something very similar for the Qwen3 models a while back. It only runs Qwen3, supports only some quants, loads from GGUF, and has inference optimized by Claude (in a loop). The whole thing is compact (just a couple of files) and easy to reason about. I made it for my students so they could tinker with it and learn (add different decoding strategies, add abliteration, etc.). Popular frameworks are large, c…

this feels closer to ATLAS/FFTW than a model runner. the generated kernel ages out, the tuning harness is the bit you actually want to keep.

Re: DeepSeek 4 Flash local inference engine for Metal

#96

Post by someone13 »

antirez wrote: post #66
bel8 wrote: post #42

Earlier quoted context omitted.

And unless I'm mistaken, the repo is about running it with 2bit quantization. This is probably far from the raw intelligence provided by cloud providers. Still, this shines more light on local LLMs for agentic workflows.

It runs both q2 and original (4 bit routed experts). At the same speed more or less. The q2 quants are not what you could expect: it works extremely well for a few reasons. For the full model you need a Mac with 256GB.

Out of curiosity, do you have any theories of why it works so well at such aggressive quantization levels?

Re: DeepSeek 4 Flash local inference engine for Metal

#98

Post by bensyverson »

dakolli wrote: post #86
bensyverson wrote: post #83

Earlier quoted context omitted.

Correct, the progress is not perfectly linear. But do you believe technological progress has stalled forever? If so, I'd get out of tech and start selling bomb shelters.

Do you really think the trend of consumer hardware is heading towards more memory and better specs? Apple's most popular product this year is an 8gb of RAM laptop.. The trend is heading in the opposite direction, less options for strong consumer hardware and towards cloud based products. This is a memory issue more than anything. Nvidia is done selling their ddr7 to gamers and people with AI girlfriends.

Just so that I have your position straight: you actually believe that over the long term, like 10, 20 years, that the amount of RAM in a laptop is going to go down?

It's not out of the realm of possibility, but I just want to make you aware that this would be a very surprising development in computing history.

Re: DeepSeek 4 Flash local inference engine for Metal

#99

Post by iuffxguy »

dakolli wrote: post #86
bensyverson wrote: post #83

Earlier quoted context omitted.

Correct, the progress is not perfectly linear. But do you believe technological progress has stalled forever? If so, I'd get out of tech and start selling bomb shelters.

Do you really think the trend of consumer hardware is heading towards more memory and better specs? Apple's most popular product this year is an 8gb of RAM laptop.. The trend is heading in the opposite direction, less options for strong consumer hardware and towards cloud based products. This is a memory issue more than anything. Nvidia is done selling their ddr7 to gamers and people with AI girlfriends.

This is more then just the hardware evolving over time but we also are seeing big improvements in quantization and efficiency improvements.

Re: DeepSeek 4 Flash local inference engine for Metal

#100

Post by fgfarben »

actsasbuffoon wrote: post #59
unshavedyak wrote: post #22

Earlier quoted context omitted.

What do you use it for? I tend to just stick to SOTA (Claude 4.7 Max thinking), and put up with the slow req/response. I'm not sure what type of work i'd trust a less thinking model, as my intuition is built around what Claude vSOTA Max can handle. Nonetheless eventually i want to build an at-home system. I imagine some smaller local model could handle metadata assignment quite well. edit : Though TIL Mac Studio does…

Apple just dropped the 128GB option as well.

It is still available for the M5 Max Macbook Pro, but yes, the Mac Studio is now only offered with up to 96 GB.
Post reply on HN