Run DeepSeek R1 Dynamic 1.58-bit

Post by riku_iki » Tue, Jan 28, 2025, 9:17 PM UTC

sourcecodeplz wrote: post #242

jmb99 wrote: post #210

Earlier quoted context omitted.

It’s more expensive, but Zen4 Threadripper Pro is probably the way to go on that front. 8 memory channels, with DIMMs available up to DDR5-7200 for 8x32GB (256GB), or DDR5-6800 for 8x48GB (384GB). It’ll set you back ~$3k for the RAM and ~$6k for a CPU with 8 CCDs (the 7985WX, at least), and then ~$1k for motherboard and however much you want to spend on NVME. Basically ~$10k for a 384GB DDR5 system with ~435GB/s actu…

At these prices, I would just get 2xDigits for $6k and have 256gb.

Post by nyrikki » Tue, Jan 28, 2025, 9:18 PM UTC

dagmx wrote: post #174

rafaelmn wrote: post #156

Earlier quoted context omitted.

>Like, I get that shared memory architectures like a 192GB Mac Ultra are a big deal, but who’s dropping $6,000+ on that setup? AMD strix halo APU will have quad channel memory and will launch soon so expect these kinds of setups available for much less. Apple is charging an arm and a leg for memory upgrades, hopefully we get competition soon. From what I saw at CES OEMs are paying attention to this use case as well -…

Unfortunately, Apple’s RAM and Storage upgrade prices are very in line with other class comparable OEMs. I’m sure there’ll be some amount of undercutting but I don’t think it’ll be a huge difference on the RAM side itself.

Post by danielhanchen » Tue, Jan 28, 2025, 9:27 PM UTC

smcleod wrote: post #254

eurekin wrote: post #141

Earlier quoted context omitted.

I have similar set-up - can you help out with running it? Was it in ollama? EDIT: It seems that original authors provided a nice write-up: https://unsloth.ai/blog/deepseekr1-dynamic#:~:text=%F0%9F%96...

Yep that's pretty much what I did, their calculation for the layers was slightly off though, I found I could offload an extra 1-2 layers to the GPUs

Post by danielhanchen » Tue, Jan 28, 2025, 9:28 PM UTC

JKCalhoun wrote: post #144

Jasondells wrote: post #44

An 80% size reduction is no joke, and the fact that the 1.58-bit version runs on dual H100s at 140 tokens/s is kind of mind-blowing. That said, I’m still skeptical about how practical this really is for most people. Like, yeah, you can run it on 24GB VRAM or even with just 20GB RAM, but "slow" is an understatement—those speeds would make even the most patient person throw their hands up. And then there’s the whole re…

Layman here — but I am hopeful for 1.58 bit plus custom silicon to be the Holy Grail. I suppose I am setting high expectations on Apple to integrate said in their next "A" chip. Wishful thinking.

Post by danielhanchen » Tue, Jan 28, 2025, 9:32 PM UTC

iamnotagenius wrote: post #153

danielhanchen wrote: post #25

Earlier quoted context omitted.

Hey! :) Coincidentally the seeds I always use are 3407, 3408 and 3409 :) 3407 because of https://arxiv.org/abs/2109.08203 I also tried not setting the seeds, but the results are still the same - quantizing all layers seems to make the model forget and repeat everything - I put all examples here: https://docs.unsloth.ai/basics/deepseek-r1-dynamic-1.58-bit#...

would be great to have dynamic quants of V3-non-R1 version, as for some tasks it is good enough. Also would be very interesting to see degradation with dynamic quants on small/medium size MoEs, such as older Deepseek models, Mixtrals, IBM tiny Granite MoE. Would be fun if Granite 1b MoE will still be functioning at 1.58bit.