Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A few words on DS4

antirez.com

Post reply on HN
Print view View original

81–90 of 197 posts

Re: A few words on DS4

#81

Post by simpaticoder »

simonw wrote: post #59
chatmasta wrote: post #45

Earlier quoted context omitted.

So you’re saying I should buy the M5? :) I’ve been resisting, thinking I’ll never use it… it’ll be better in a year… I’ll wait for the Studio (do we still think that’s coming in June?)… etc.

I expect this to be my main machine for the next 3-4 years (which is how I justified the 128GB one). It's a beast of a machine - I love that I can run an 80GB model and still have 48GB left for everything else. Can't say that it wouldn't be a better idea to spend that cash on tokens from the frontier hosted models though. I'm an LLM nerd so running local models is worth it from a research perspective.

An M5 Max MBP with 128G of RAM costs ~$5k. An Nvidia RTX 5090 with 32G RAM is $4-5k, and RTX PRO 6000 with 96GB RAM $10k. Do you have any data on which is the best price/performance for local inference? Do you know what the big OpenAI/Anthropic/Google datacenters are running?

Re: A few words on DS4

#82

Post by [deleted] »

xienze wrote: post #13
aiscoming wrote: post #9

Earlier quoted context omitted.

if it's just the coding agent system prompt and tools, you can cache that

Yeah the problem is that's just the start of the context. There's, you know, all the tool call results and file reads and stuff.

[deleted]

Re: A few words on DS4

#83

Post by thomasm6m6 »

DeathArrow wrote: post #74
gcr wrote: post #46

DwarfStar4 is a small LLM inference runtime that can run DeepSeek 4. The blog post implies that it currently requires 96GB of VRAM. For others who are lacking context :-)

>The blog post implies that it currently requires 96GB of VRAM. From the Github page it seems it only supports Apple and DGX Spark. I have 128 GB of RAM and a 3090 but it probably won't work.

FYI, llama.cpp (which antirez/ds4 is inspired by) supports system ram. E.g. [1] is a good guide for running a similar-sized model with 128gb ram and a 3090-sized GPU.

[1] https://unsloth.ai/docs/models/tutorials/minimax-m27

(Unsloth's deepseek-v4 support is still WIP)

Re: A few words on DS4

#84

Post by shric »

keyle wrote: post #79
karmakaze wrote: post #42

Great to find this narrow focused thing: > We support the following backends: Metal is our primary target. Starting from MacBooks with 96GB of RAM. NVIDIA CUDA with special care for the DGX Spark. AMD ROCm is only supported in the rocm branch. It is kept separate from main since I (antirez) don't have direct hardware access, so the community rebases the branch as needed. > This project would not exist without llama.c…

If only you could still buy Mac's with that much RAM

You can buy 128GB M5 MacBook Pros?

Configured one just now, delivers in 2 weeks

Re: A few words on DS4

#85

Post by trollbridge »

DeathArrow wrote: post #75
stavros wrote: post #37

Earlier quoted context omitted.

In my (obviously anecdotal) testing, DeepseekV4 Pro was better than Sonnet at coding. However, it is much slower, but also many times cheaper, especially with the promotion right now.

Do they have a coding plan or you only pay per API call?

It’s just per token, but burning up 100 million+ tokens is a $3 transaction with their pricing right now

Re: A few words on DS4

#86

Post by vlovich123 »

davnicwil wrote: post #65
loeg wrote: post #62

Earlier quoted context omitted.

> At some point, you can let a less smart model hammer at a problem for longer and get to the same result, and as long as you are not involved it comes to the same thing. Is that true? I find the smarter models can just be effective when smaller models can't. It isn't a matter of just waiting longer.

it's almost certainly not true yet but at some point there might be an equilibrium reached of speed Vs quality (and let's not forget, cost) where it's true for most of what you do. Perhaps you'd still turn to hosted models for the hardest tasks, but most tasks go local. It does seem like that would make demand go down significantly. Of course that's all predicated on model advances plateauing, or at least getting inc…

Why is this presumed to be de facto inevitable:

* local compute isn’t scaling as before, so algorithmic improvements are the only ways models get meaningfully faster and smarter

* all those same algorithmic improvements would also be true for larger models

* hardware manufacturers have an incentive against local LLMs because cloud LLMs are so much more lucrative (+ corps would by desktop variants if they were good enough)

So no it’s not clear quality will ever be comparable. It may be good enough for what you want but there will always be a harder problem that you need to throw more compute and more memory at.

Re: A few words on DS4

#87

Post by linkregister »

dakolli wrote: post #76
linkregister wrote: post #72

Earlier quoted context omitted.

I'm having trouble finding dictionaries or other references that add the qualifier that it needs to be self-tested and not relying on the research of others. Can you point me to one?

I don't think comments on the internet count as "empirical" evidence, but sure.

If you think antirez's benchmarks in the blog post are false, you should make the claim. Continue to move the goal posts.

Re: A few words on DS4

#88

Post by Riany »

I think local models need to be good enough that privacy, latency, and control become worth the tradeoff, instead of beat the best cloud models

Re: A few words on DS4

#89

Post by aiscoming »

simpaticoder wrote: post #81
simonw wrote: post #59

Earlier quoted context omitted.

I expect this to be my main machine for the next 3-4 years (which is how I justified the 128GB one). It's a beast of a machine - I love that I can run an 80GB model and still have 48GB left for everything else. Can't say that it wouldn't be a better idea to spend that cash on tokens from the frontier hosted models though. I'm an LLM nerd so running local models is worth it from a research perspective.

An M5 Max MBP with 128G of RAM costs ~$5k. An Nvidia RTX 5090 with 32G RAM is $4-5k, and RTX PRO 6000 with 96GB RAM $10k. Do you have any data on which is the best price/performance for local inference? Do you know what the big OpenAI/Anthropic/Google datacenters are running?

[dead]

Re: A few words on DS4

#90

Post by manmal »

DeathArrow wrote: post #74
gcr wrote: post #46

DwarfStar4 is a small LLM inference runtime that can run DeepSeek 4. The blog post implies that it currently requires 96GB of VRAM. For others who are lacking context :-)

>The blog post implies that it currently requires 96GB of VRAM. From the Github page it seems it only supports Apple and DGX Spark. I have 128 GB of RAM and a 3090 but it probably won't work.

It wouldn’t be useful with your setup, probably 3-4 token per second.
Post reply on HN