Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DeepSeek 4 Flash local inference engine for Metal

github.com

Post reply on HN
Print view View original

141–150 of 171 posts

Re: DeepSeek 4 Flash local inference engine for Metal

#141

Post by segmondy »

antirez wrote: post #109
layoric wrote: post #87

Very impressive. One thing that seems odd to me is that is at like 4 minutes before it starts a response for large input? I don't use mac hardware for LLMs, but that is quite surprising and would seem to be a pretty large stumbling block for practical usage. Edit: Caching story makes a lot more sense for regular usage: > Claude Code may send a large initial prompt, often around 25k tokens, before it starts doing usef…

Yep that happens with coding agents sending a very large system prompt. And also when later tool calling feed it large files or diffs. But with the M3 ultra the prefill speed is almost 500 t/s that is quite into the very usable zone. With M3 max you need a bit more patience but it works well and as it emits the think process if you use the pi agent you don't wait: you read non censored chain of though. I posted a vid…

Curious why you went this route, don't you think you could have achieved near this performance 80%+ or more within llama.cpp?

Re: DeepSeek 4 Flash local inference engine for Metal

#143

Post by GeekyBear »

andsoitis wrote: post #135
nopurpose wrote: post #134

Earlier quoted context omitted.

Ultra-optimized HW-specific engines is what Mojo lang seems to be targeting, but I rarely hear about it here.

> Mojo lang seems to be targeting, but I rarely hear about it here Momentum over at Mojo lang seems very very slow. According to their roadmap, they're still busy on Phase 1 ("High performance CPU + GPU coding"), and haven't touched Phase 2 ("Systems application programming") and Phase 3 ("Dynamic object-oriented programming"). So perhaps there isn't much to talk about?

They've got a lot of work yet to do to be a general purpose language, but for GPU programming they have already demonstrated that they can outperform CUDA on Nvidia GPUs.

That's pretty compelling.

Re: DeepSeek 4 Flash local inference engine for Metal

#144

Post by Terretta »

kristianp wrote: post #124

Hmm, I'm unable to order more than 96GB RAM for a Mac studio, even with the M3 ultra or M4 Max. Is this au specific? However with the MacBook Pro I can specify 128GB with the M5 Mac. https://www.apple.com/au/shop/buy-mac/mac-studio

Difficult to believe this memory is made of unobtanium.

Maybe Apple would rather not price it at all than experience blowback for either gouging or lack of inventory.

Re: DeepSeek 4 Flash local inference engine for Metal

#145

Post by Terretta »

smcleod wrote: post #126
kristianp wrote: post #124

Hmm, I'm unable to order more than 96GB RAM for a Mac studio, even with the M3 ultra or M4 Max. Is this au specific? However with the MacBook Pro I can specify 128GB with the M5 Mac. https://www.apple.com/au/shop/buy-mac/mac-studio

The studio is really old now. The new one will drop at some point no doubt with more memory options. the 128GB M5 max MBP is great though

And yet, aside from offering 512GB, that really old Studio Ultra M3 LLMs faster (especially sustained) than the new M5 Max.

Re: DeepSeek 4 Flash local inference engine for Metal

#146

Post by happyPersonR »

antirez wrote: post #110
someone13 wrote: post #96

Earlier quoted context omitted.

Out of curiosity, do you have any theories of why it works so well at such aggressive quantization levels?

It's a mix of extreme sparsity but with the routed expert doing a non trivial amount of work (and it is q8), and projections and routing not being quantized as well. Also the fact it's a QAT model must have a role I guess, and I quantized routed experts out layers with Q2 instead of IQ2_XXS to retain quality.

Not trying to give anyone homework thinking out loud :

One thing I would love to see is if this dogfoods itself

Like would dsv4 with q2 be able to do this task itself on this hardware ?

Sidenote: I wish I had a M4-m3 … thinking about getting a ASUS ROG Flow Z13 Gaming Laptop (Model GZ302EA-XS99) uses pcie 4.0 so disk might be a little slower, but I want to see how this does on like Vulcan :)

Re: DeepSeek 4 Flash local inference engine for Metal

#147

Post by sev_verso »

I've tried it out with Claude Code on my existing codebase and it seemed to hold its weight (despite being the 2-bit quant). Takes minutes on prompt processing, the actual edits are reasonably quick at above 20 tks.

The good: It succeeded with discovering, applying edits and writing a test for a small task I gave it. The bad: It could not address a small nitpick I had. The ugly: It hallucinated a conversation about "The Duck" that I had with it simultaneously while trying to solve another problem. I can only imagine it's one of examples in the initial Claude Code prompt:

--cut-- However, the user's query is "Can you track these 3 videos here?" which seems unrelated. Perhaps the user is asking if I can track the progress of three videos they are working on?

Let me re-read the user's message. The user said "Source Code" and "The Agent" and "The Duck", it could be video titles. And they are asking if I can track these 3 videos.

?? That doesn't make sense in the context. Could there be two different conversations? --cut--

Re: DeepSeek 4 Flash local inference engine for Metal

#149

Post by marci »

dakolli wrote: post #101
bensyverson wrote: post #98

Earlier quoted context omitted.

Just so that I have your position straight: you actually believe that over the long term, like 10, 20 years, that the amount of RAM in a laptop is going to go down? It's not out of the realm of possibility, but I just want to make you aware that this would be a very surprising development in computing history.

A future with less RAM is possible with more applications using computational storage with ssd/nvme. But that's not my main argument is that its delusional for OP thinks its reasonable to expect that soon we'll be able to run models on consumer hardware that will be able to build basically most things, But I do think there will be many compromises made for consumer electronics, I don't think the powers that be are ea…

Did they modify their post? I can't see who claimed that consumer hardware will be able to build most things?

Re: DeepSeek 4 Flash local inference engine for Metal

#150

Post by srdjanr »

wlesieutre wrote: post #69
minimaxir wrote: post #17

Earlier quoted context omitted.

"Data centers for LLMs are technically more energy efficient per-user than self-hosting LLM models due to economies-of-scale" is a data point the internet isn't ready for.

But if you're running it on your own hardware you might only generate tokens when you have something useful to do with them, instead of every time you load a Google search results page because Google decided the future is stuffing Gemini-generated answers down your eyeballs instead of letting you read it yourself from the primary source for 0.1 watts.

Whether I'm using Google or not is completely unrelated to whether I use OpenAI (for example) API or run LLM locally
Post reply on HN