Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Run DeepSeek R1 Dynamic 1.58-bit

unsloth.ai

Post reply on HN
Print view View original

311–320 of 346 posts

Re: Run DeepSeek R1 Dynamic 1.58-bit

#311

Post by benjiro »

tucnak wrote: post #277
DogRunner wrote: post #267

>For optimal performance, we recommend the sum of VRAM + RAM to be at least 80GB+. Oh nice! So I can try it in my local "low power/low cost" server at home. My homesystem does run in a ryzen 5500 + 64gb RAM + 7x RTX 3060 12gb So 64gb RAM plus 84gb VRAM I dont want to brag around, but point to solutions for us tinkerers with a small budget and high energy costs. such system can be build for around 1600 euro. The power…

How are you arriving at those numbers? ryzen 5500 + 7x3060 + cooling ~= 1.6 kW off the wall, at 360 GB/s memory bandwidth, and considering your lane budget, most of it will be wasted in single PCIe lanes. After-market unit price of 3060's is 200 eur, so 1600 is not good-faith cost estimate. From the looks of it, your setup is neither low-power, nor low-cost. You'd be better served with a refurbished mac studio (2022)…

The issue is that you are taking max GPU power draw, as a given. Running a LLM does not tax a GPU the same way a game does. There is a rather know Youtuber, that ran LLMs on a 4090, and the actual power draw was only 130W on the GPU.

Now add that this guy has 7x3060 = 100% miner. So you know that he is running a optimized profile (underclocked).

Fyi, my gaming 6800 draws 230W, but with a bit of undervolting and sacrificing 7% performance, it runs at 110W for the exact same load. And that is 100% taxed. This is just a simple example to show that a lot of PC hardware runs very much overclocked/unoptimized out of the box.

Somebody getting down to 520W sounds perfectly normal, for a undervolted card that gives up maybe 10% performance, for big gains in power draw.

And no, old hardware can be extreme useful in the right hands. Add to this, its the main factor that influences the speed tends to be more memory usage (the more you can fit and the interconnects), then actual processing performance for running a LLM.

Being able to run a large model for 1600 sounds like a bargain to me. Also, remember, when your not querying the models, the power will be mostly the memory wakes + power regulators. Coming back to that youtuber, he was not constantly drawing that 130W, it was only with spikes when he ran prompts or did activity.

Yes, running from home will be more expensive then a 10$ copilot plan but ... nobody is also looking at your data ;)

Re: Run DeepSeek R1 Dynamic 1.58-bit

#312

Post by emseetech »

TheTaytay wrote: post #310
emseetech wrote: post #243

Earlier quoted context omitted.

I'm downloading it now and will report back. (I've been using the 32B and while it could always be better, I'm not unhappy with it)

How'd it go, and which client are you using? :)

Pretty rough.

Using LM Studio, trying to load the model throws an error of "insufficient system resources."

I disabled this error, set the context length to 1024 and was able to get 0.24 tokens per second. Comparatively, the 32B distill model gets about 20 tokens per second.

And it became incredibly flaky, using up all available ram, and crashing the whole system a few times.

While the M4 Max 128GB handles the 32B well, it seems to choke on this. Here's to hoping someone works on something in-between (or works out what the ideal settings are because nothing I fiddled with helped much).

Re: Run DeepSeek R1 Dynamic 1.58-bit

#313

Post by DogRunner »

benjiro wrote: post #311
tucnak wrote: post #277

Earlier quoted context omitted.

How are you arriving at those numbers? ryzen 5500 + 7x3060 + cooling ~= 1.6 kW off the wall, at 360 GB/s memory bandwidth, and considering your lane budget, most of it will be wasted in single PCIe lanes. After-market unit price of 3060's is 200 eur, so 1600 is not good-faith cost estimate. From the looks of it, your setup is neither low-power, nor low-cost. You'd be better served with a refurbished mac studio (2022)…

The issue is that you are taking max GPU power draw, as a given. Running a LLM does not tax a GPU the same way a game does. There is a rather know Youtuber, that ran LLMs on a 4090, and the actual power draw was only 130W on the GPU. Now add that this guy has 7x3060 = 100% miner. So you know that he is running a optimized profile (underclocked). Fyi, my gaming 6800 draws 230W, but with a bit of undervolting and sacri…

Thanks for the clarification. Surely, If I run hashcat benchmark the power consumption goes nearly to 1400 Watt, but I also limited the max power consumption for each card to 100 Watt, which worked out better than limiting the max gpu frequency. To be fair, the most speed comes from the RAM frequency - as long as this is not limited, it works out great.

I took a fair amount of time to get everything to a reduced power level and measured several llm models (and hashcat for the extreme) to find the best speed per watt, which is usally around 1700-1900 mhz or limiting 3060 to 100 to 115 watt.

If I planned it in the first run, I may got away with a used mac studio, thats right. However, I incrementally added more cards as I moved further into exploration.

I didn't wanted to confront someone, but it looks like you either show of 4x 4090 or you keep silent

Re: Run DeepSeek R1 Dynamic 1.58-bit

#314

Post by stevenhuang »

ggm wrote: post #298

If I invested in a 100x machine because I needed 100 of x to run, and somebody shows how 10x can work, why have I not just become the holder of 10 10x machines, and therefore have already achieved capex to exploit this new market? I cannot understand why "openai is dead" has legs: repurpose the hardware and data and it can be multiple instances of the more efficient model.

because of discounted cash flow/valuation models.

you invest in a 100x machine expecting a revenue of X, but now you can only charge X/100 because R1 shows that AI inference can be done much more efficiently. see the price decrease of ChatGPT and addition of free O3 etc.

this reduction of future cash flows, ceteris paribus, implies that the present value of these cash flows decrease. this then results in massive repricing to the downside as market participants update their forecasts.

what you are missing is that to assume as you do, you must make the additional assumption that demand for additional compute is infinite. Which may very well be the case, but it is not guaranteed compared to the present realized fact that R1 means lower revenues for AI inference providers -> changes the capex justification for even more hardware -> NVDA receives less revenue.

Re: Run DeepSeek R1 Dynamic 1.58-bit

#315

Post by sureglymop »

nicce wrote: post #268
yobid20 wrote: post #91

Earlier quoted context omitted.

I canceled my OpenAI subscription last night, as did many many others. There were some threads in reddit with everyone chiming in they all just canceled too. imo OpenAI is done, and will go through massive cuts and probably acquired by the end of the year for a very tiny fraction of its current value.

1. You can get all the models by buying Kagi subscription (excluding o1). Includes DeepSeek models. You can also feed the assistant with search data that you can filter. 2. If you have GitHub Copilot, you get o1 chat also there. I haven't seen much value with OpenAI subscription for ages.

I have Kagi Ultimate and it is nice for this. But a cheaper suggestion would be to use OpenRouter and then use these models via Fireworks or TogetherAI. It also integrates into much more applications. AFAIK Kagi doesn't document a user facing API for the assistant feature.

Re: Run DeepSeek R1 Dynamic 1.58-bit

#316

Post by benjiro »

DogRunner wrote: post #313
benjiro wrote: post #311

Earlier quoted context omitted.

The issue is that you are taking max GPU power draw, as a given. Running a LLM does not tax a GPU the same way a game does. There is a rather know Youtuber, that ran LLMs on a 4090, and the actual power draw was only 130W on the GPU. Now add that this guy has 7x3060 = 100% miner. So you know that he is running a optimized profile (underclocked). Fyi, my gaming 6800 draws 230W, but with a bit of undervolting and sacri…

Thanks for the clarification. Surely, If I run hashcat benchmark the power consumption goes nearly to 1400 Watt, but I also limited the max power consumption for each card to 100 Watt, which worked out better than limiting the max gpu frequency. To be fair, the most speed comes from the RAM frequency - as long as this is not limited, it works out great. I took a fair amount of time to get everything to a reduced powe…

np ;)

I am amazed these days people lacking knowledge about hardware, and the mass benefits of undervolting/power limiting hardware. Its like people do not realize that what is sold, is often overclocked/too high vcore. The amount of people i see buying insane overspec PSUs, and go O_o ...

How is your performance with the different models on your setup?

Re: Run DeepSeek R1 Dynamic 1.58-bit

#317

Post by ggm »

stevenhuang wrote: post #314
ggm wrote: post #298

If I invested in a 100x machine because I needed 100 of x to run, and somebody shows how 10x can work, why have I not just become the holder of 10 10x machines, and therefore have already achieved capex to exploit this new market? I cannot understand why "openai is dead" has legs: repurpose the hardware and data and it can be multiple instances of the more efficient model.

because of discounted cash flow/valuation models. you invest in a 100x machine expecting a revenue of X, but now you can only charge X/100 because R1 shows that AI inference can be done much more efficiently. see the price decrease of ChatGPT and addition of free O3 etc. this reduction of future cash flows, ceteris paribus, implies that the present value of these cash flows decrease. this then results in massive repr…

Thanks. Always a mistake to assume the price of something is bound to your own cost of doing it: the price is bound to the other guys cheaper price!

Re: Run DeepSeek R1 Dynamic 1.58-bit

#318

Post by htk »

emseetech wrote: post #312
TheTaytay wrote: post #310

Earlier quoted context omitted.

How'd it go, and which client are you using? :)

Pretty rough. Using LM Studio, trying to load the model throws an error of "insufficient system resources." I disabled this error, set the context length to 1024 and was able to get 0.24 tokens per second. Comparatively, the 32B distill model gets about 20 tokens per second. And it became incredibly flaky, using up all available ram, and crashing the whole system a few times. While the M4 Max 128GB handles the 32B we…

There's a terminal command to increase the maximum vram MacOS can use, you can try that as you're probably going over the limit and the system is resorting to treat as system ram. (I ran into this problem a couple of times using ollama).

Re: Run DeepSeek R1 Dynamic 1.58-bit

#319

Post by bluesounddirect »

lucb1e wrote: post #237
bluesounddirect wrote: post #134

Hi small comment, please remember in china many things are sponsored by or subsidized by the government. "We[china] can do it for less.." , "it's cheaper in china.." only means the government gave us a pile of cash and help to get here . I 100% expect some downvotes from the ccp.

> I 100% expect some downvotes from the ccp. Always happy to oblige when someone insinuates that any critics must be government agents

Naa just wanted to see who would bite.

Re: Run DeepSeek R1 Dynamic 1.58-bit

#320

Post by ddingus »

dbtc wrote: post #249
ddingus wrote: post #229

Earlier quoted context omitted.

I fired the model up at 8:39, ready to chat just a bit before 8:40, during which I input the following prompt: Please define "lark" for me, as in "I did it on a lark." Output began a bit before 8:41, and was complete before 8:43: ---Begin Model Output "On a lark" means without much planning or reason, impulsively or unexpectedly. Here's how it works: "I did it on a lark." This means the person acted without a specifi…

I opened Dictionary.app and typed in lark and got the definition with etymology in under 30 seconds. > British English informal used to suggest that an activity is foolish or a waste of time: he's serious about this music lark.

You know that's not the point. You can't even ask the dictionary app a question. You more or less just input and it'll give you what it has.

I can ask this model some pretty intense things and I get meaningful output back and it doesn't take very long and I don't need an internet connection. Like I said it's spiffy.

I will, have, and will use it again sans network connection and be happy I had the option.

Better will come. That is just fine. I have something very useful today, right now.

Post reply on HN