Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Run DeepSeek R1 Dynamic 1.58-bit

unsloth.ai

Post reply on HN
Print view View original

271–280 of 346 posts

Re: Run DeepSeek R1 Dynamic 1.58-bit

#271

Post by rafaelmn »

sliken wrote: post #235
rafaelmn wrote: post #156

Earlier quoted context omitted.

>Like, I get that shared memory architectures like a 192GB Mac Ultra are a big deal, but who’s dropping $6,000+ on that setup? AMD strix halo APU will have quad channel memory and will launch soon so expect these kinds of setups available for much less. Apple is charging an arm and a leg for memory upgrades, hopefully we get competition soon. From what I saw at CES OEMs are paying attention to this use case as well -…

Keep in mind the strix halo APU has a 256 bit wide memory bus and the Mac Ultra has a 1024 bit wide memory bus. Here's hoping the Nvidia Digit (GB10 chip) has a 512 bit or 1024 bit wide interface, otherwise the Strix Halo will be the best you can do if you don't get the Mac Ultra.

I mean it remains to be seen if it will be compute or bandwidth bound, I am sure mac ultra will also have double or triple compute as well.

But in either case its going to do much better than currently available CPUs with easily upgradeable ram. I would not be surprised to see 128gb configurations for around 3k (going of the ASUS g13 announced pricing of arround 2k for 32gb version and them saying it will go up to 128gb).

At that point sure it might not compete with max but its at a much more acceptable price point, it will not be a device you get just for the AI, but a mobile workstation that you can also run some local models on for normal money. Will need to wait and see. I know I am not buying anything from ASUS either way.

Re: Run DeepSeek R1 Dynamic 1.58-bit

#272

Post by ryao »

smcleod wrote: post #67
Jasondells wrote: post #44

An 80% size reduction is no joke, and the fact that the 1.58-bit version runs on dual H100s at 140 tokens/s is kind of mind-blowing. That said, I’m still skeptical about how practical this really is for most people. Like, yeah, you can run it on 24GB VRAM or even with just 20GB RAM, but "slow" is an understatement—those speeds would make even the most patient person throw their hands up. And then there’s the whole re…

I just ran it up on 48gb (2x 3090) + overflow into CPU RAM and it runs at around 4tk/s (only a little 8k context size though) which while absolutely not something I'd personally use daily - it is actually usable.

Which one did you run?

Re: Run DeepSeek R1 Dynamic 1.58-bit

#273

Post by danielhanchen »

Der_Einzige wrote: post #265
smcleod wrote: post #69

Earlier quoted context omitted.

min_p is great, do you apply a small amount of temperate as well?

Btw, min_p (the paper about the sampler) got accepted to ICLR! As 4th author it warms my heart to so it used so much in the wild.

Oh hi!! Congratulations on ICLR!!! min_p = 0.1 and temp = 1.5 is my default goto settings!!

Re: Run DeepSeek R1 Dynamic 1.58-bit

#274

Post by ryao »

thenickdude wrote: post #59
karamanolev wrote: post #42

Earlier quoted context omitted.

For personal usage, does it matter though? In most places residential electricity is cheap compared to everything else. In a DC context I feel it matters a lot more compared to the capex.

1x 3090 (350W power limit) already makes it feel like I'm running a fan heater under my desk, 5x would be nuts.

When running inference workloads via something like llama.cpp, only 1 GPU is ever used at a time, so you would have 1 active GPU and 4 idle GPUs. That should make the power usage less insane in practice than you expect.

Re: Run DeepSeek R1 Dynamic 1.58-bit

#275

Post by ryao »

brap wrote: post #34

As someone who is out of the loop, what’s the verdict on R1? Was anyone able to reproduce the results yet? Is the claim that it only took $5M to train generally accepted? It’s a very bold claim which is really shaking up the markets, so I can’t help but wonder if it was even verified at this point.

They claimed that it only took $5 million of GPUs to train Deepseek v3, which was the base model. They did not claim that the total costs were $5 million. They omitted the costs of additional hardware, electricity, personnel, training dataset acquisition, etcetera. They likely spent tens of times more on this at a minimum.

That said, what they did with $5 million of GPUs is impressive. Reportedly, they resorted to using PTX assembly to make it possible:

https://www.tomshardware.com/tech-industry/artificial-intell...

Re: Run DeepSeek R1 Dynamic 1.58-bit

#276

Post by smcleod »

danielhanchen wrote: post #255
smcleod wrote: post #69

Earlier quoted context omitted.

min_p is great, do you apply a small amount of temperate as well?

The recommended temperature from DeepSeek is 0.6 so I leave it at that!

I think most of the model creators share their model usage examples so high at 0.6-0.7 simply because it's what a lot of the client apps use. IMO this is WAY too high unless you're doing creative writing.

Generally I set temp to 0-0.4 at absolute most.

min_p actually needs a little temperature to work effectively so with min_p I almost always use 0.2

Re: Run DeepSeek R1 Dynamic 1.58-bit

#277

Post by tucnak »

DogRunner wrote: post #267

>For optimal performance, we recommend the sum of VRAM + RAM to be at least 80GB+. Oh nice! So I can try it in my local "low power/low cost" server at home. My homesystem does run in a ryzen 5500 + 64gb RAM + 7x RTX 3060 12gb So 64gb RAM plus 84gb VRAM I dont want to brag around, but point to solutions for us tinkerers with a small budget and high energy costs. such system can be build for around 1600 euro. The power…

How are you arriving at those numbers?

ryzen 5500 + 7x3060 + cooling ~= 1.6 kW off the wall, at 360 GB/s memory bandwidth, and considering your lane budget, most of it will be wasted in single PCIe lanes. After-market unit price of 3060's is 200 eur, so 1600 is not good-faith cost estimate.

From the looks of it, your setup is neither low-power, nor low-cost. You'd be better served with a refurbished mac studio (2022) at 400GB/s bandwidth fully utilised over 96 GB memory. Yes, it will cost you 50% more (considering real cost of such system closer to 2000 eur) however it would run at a fraction of power use (10x less, more or less)

I get it that hobbyists like to build PC's, but claiming that sticking seven five year out of date low-bandwidth GPU's in a box is "low power/low cost" is a silly proposition.

You're advocating for e-waste

Re: Run DeepSeek R1 Dynamic 1.58-bit

#278

Post by nyrikki »

whimsicalism wrote: post #192
wklauss wrote: post #167

Earlier quoted context omitted.

Ollama is pretty clear about it, it's not like they are trying to deceive. You can also download the 671B model with Ollama, if you like.

no they are not, they intentionally remove every reference to this not being r1 from the cli and changed the names from the ones both Deepseek and Huggingface used.

They used short strings for the names, which is very different from deception.

https://ollama.com/search

> DeepSeek's first-generation of reasoning models with comparable performance to OpenAI-o1, including six dense models distilled from DeepSeek-R1 based on Llama and Qwen.

Well I guess if you are in the Enterprise Java naming model you would expect something like "­Visitor­Model­Utils­List­Getter­Adapter­Interceptor­Message­Manager­Driven­Observer­Pool"

If you look at their API docs you will see:

    model: name of the model to push in the form of /:
I don't think there is any reason to jump to the conclusion it is some type of conspiracy here, just naming things based on a API that probably didn't think about distillation when they created it.

Re: Run DeepSeek R1 Dynamic 1.58-bit

#279

Post by tucnak »

mcv wrote: post #257
whimsicalism wrote: post #222

Earlier quoted context omitted.

my argument isn’t fallacious - it is logical: we can learn/use evidence from something without presuming it is all knowing. you are putting words in others mouths that they did not say

I'm sorry, I thought you introduced the "all-knowing" out of nowhere, but this was indeed mentioned by willsmith72. I'd missed that. Still, his implied assertion that markets that markets can often behave irrationally, and can't be used as evidence of technical matters, seems pretty valid to me. But I suppose you could see it as a sign that something is at least temporarily "generally accepted" among investors. That…

Don't worry, NVDA will bounce back and you will get a chance to get out.

Efficient market hypothesis is for nerds!

Re: Run DeepSeek R1 Dynamic 1.58-bit

#280

Post by danielhanchen »

smcleod wrote: post #276
danielhanchen wrote: post #255

Earlier quoted context omitted.

The recommended temperature from DeepSeek is 0.6 so I leave it at that!

I think most of the model creators share their model usage examples so high at 0.6-0.7 simply because it's what a lot of the client apps use. IMO this is WAY too high unless you're doing creative writing. Generally I set temp to 0-0.4 at absolute most. min_p actually needs a little temperature to work effectively so with min_p I almost always use 0.2

Ye lower temp is also good :) Tbh its all trial and error - I found temp=1.5, min_p=0.1 to be very useful for pass@k type workloads - ie calling the LLM multiple times and aggregating.

temp=0 is also good for singular outputs. For classification tasks, it's better to actually inspect the logits.

But my goto setting is always setting min_p at least 0.01 or 0.05! It vastly suppresses incorrect rare random tokens from being created, and it helps massively!

Post reply on HN