Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Run DeepSeek R1 Dynamic 1.58-bit

unsloth.ai

Post reply on HN
Print view View original

241–250 of 346 posts

Re: Run DeepSeek R1 Dynamic 1.58-bit

#241

Post by sourcecodeplz »

xbmcuser wrote: post #119
nodja wrote: post #21

Earlier quoted context omitted.

While 192GB of ram is appealing, it's also quite expensive at $6000. For that price I rather buy a system with 5 used 3090s, which while being "only" 120GB of VRAM, you benefit from much faster tokens/s and prompt processing speed (the macs are notoriously slow at consuming large contexts).

I think just getting nvidia Project Digits might be the best option. A lot of people when it was announced were underwhelmed. But I think now it could be just the thing for people making their own ai home servers. https://www.nvidia.com/en-us/project-digits/

Yep, I think the same. With 128GB fast memory one could run this.

Re: Run DeepSeek R1 Dynamic 1.58-bit

#242

Post by sourcecodeplz »

jmb99 wrote: post #210
throw-qqqqq wrote: post #45

Earlier quoted context omitted.

The bottleneck is mainly memory bandwidth. AMD EPYC hw is appealing for local inference because it has a higher memory bandwidth than desktop gear (because 8-12 memory channels vs 2 on almost everything else), but not as fast as the Apple architectures and nowhere near VRAM speeds. If you want to drastically exceed ~3-5 tokens/s on 70b-q4 models, you usually still need GPUs.

It’s more expensive, but Zen4 Threadripper Pro is probably the way to go on that front. 8 memory channels, with DIMMs available up to DDR5-7200 for 8x32GB (256GB), or DDR5-6800 for 8x48GB (384GB). It’ll set you back ~$3k for the RAM and ~$6k for a CPU with 8 CCDs (the 7985WX, at least), and then ~$1k for motherboard and however much you want to spend on NVME. Basically ~$10k for a 384GB DDR5 system with ~435GB/s actu…

At these prices, I would just get 2xDigits for $6k and have 256gb.

Re: Run DeepSeek R1 Dynamic 1.58-bit

#243

Post by emseetech »

xiphias2 wrote: post #236

Has it been tried on 128GB M4 MacBook Pro? I'm gonna try it, but I guess it will be too slow to be usable. I love the original DeepSeek model, but the distilled versions are too dumb usually. I'm excited to try my own queries on it.

I'm downloading it now and will report back.

(I've been using the 32B and while it could always be better, I'm not unhappy with it)

Re: Run DeepSeek R1 Dynamic 1.58-bit

#244

Post by yieldcrv »

yobid20 wrote: post #91
apples_oranges wrote: post #16

Random observation 1: I was running DeepSeek yesterday on my Linux with a RTX 4090 and I noticed that the models should fit into VRAM, which is 24GB. Or they are simply slow. So the Apple shared memory architecture has an advantage here. A 192GB Mx Ultra can load and process large models efficiently. Random observation 2: It's time to cancel the OpenAI subscription.

I canceled my OpenAI subscription last night, as did many many others. There were some threads in reddit with everyone chiming in they all just canceled too. imo OpenAI is done, and will go through massive cuts and probably acquired by the end of the year for a very tiny fraction of its current value.

I disagree, I don't really need "conversational chat responses", I need multimodal

ChatGPT is the king of the multimodal experience still. Anthropic is a distant second, only because it lets you upload images from the clipboard and responds to them, but it can't do anything else like generate images - sometimes it will do a flowchat which is kind of cool, GPT won't do that - but will it speak to you, have tones, listen to you? no.

And in the open source side, this area has been stagnant for like 18 months. There is no cohesive multimodal experience yet. Just a couple vision models with chat capabilities and pretty pathetic GUIs to support them. You have to still do everything yourself there.

There is a huge utility for me, and many others that dont know it yet, if we could just load a couple models at once that work together seamlessly in a single seamless GUI like how ChatGPT works.

Re: Run DeepSeek R1 Dynamic 1.58-bit

#245

Post by Davidzheng »

danielbln wrote: post #13
tarruda wrote: post #9

Earlier quoted context omitted.

I remember that Llama 3 was trained on data curated by Llama 2 and it resulted in a model with a significant performance boost (even though it was trained by a previous generation model of the same size). Maybe using a strong reasoning model such as R1 the next generation, even more performance can be extracted from smaller models.

That's already happening, and is in fact even part of the R1 training pipeline. An intermediate small reasoning model churns out training data for RL a larger model, rinse and repeat. Deepseek also showed model distillation with synthetic reasoning data to work quite well.

Is your first claim in the R1 paper? I didn't see it when I looked

Re: Run DeepSeek R1 Dynamic 1.58-bit

#246

Post by jchw »

criddell wrote: post #142
jchw wrote: post #88

Earlier quoted context omitted.

Only on Hacker News would I have someone arguing with me that laptop theft is not a concern. You know what, you win. It's your $6,000 laptop, not mine.

A $6000 laptop doesn’t look much different than a $1000 laptop. I don’t think it’s a bigger theft risk than any other laptop. Make sure the laptop is insured and that full disk encryption is enabled. If it’s a Mac, make sure you have it in Find My so you can wipe it remotely if that’s something you worry about.

Honestly, I didn't bother making a better case for why I wouldn't want a $6,000 laptop in large part because the nerve people have to argue that theft isn't a concern at all made me stubborn. Theft is one reason, but a laptop is also a hell of a lot easier to simply break or lose than a desktop that is permanently installed somewhere, and a desktop is more upgradable and repairable, with typically much more I/O.

Today's baseline laptops are really good as it is. 32-64 GiB of RAM is plenty, and at least on PC laptops you can do it fairly cheaply. Apple has been a consistent year or two ahead in mobile CPU performance but it fell out of my consideration ever since I realized the M1 and 7040 were both very sufficient for any local computation I cared about. (I'm not going to say I'd specifically go for less efficiency or performance, but it has become significantly lower priority over other things like repairability.)

Not really specifically hating on Apple, here. If I was going to get another Mac it'd be a Mac Mini or Mac Studio probably, ideally with a third-party SSD upgrade to both save on costs and get a slight bit of extra drive performance too. I've definitely considered it, even though I am very far from an Apple fan, just due to the superior value and efficiency they have in many categories.

Re: Run DeepSeek R1 Dynamic 1.58-bit

#247

Post by thot_experiment »

whimsicalism wrote: post #190
elorant wrote: post #53

Earlier quoted context omitted.

Not everyone needs the largest model. There are variations or R1 with fewer parameters that can easily run on consumer hardware. With 80% size reduction you could run 70B on 8-bit on an RTX 3090. Other than that, if you really need the big one you can get six 3090s and you're good to go. It's not cheap, but you're running a ChatGPT equivalent model from your basement. A year ago this was a wetdream for most enthusias…

There’s a huge difference both in capabilities and in meaning between “variations of r1” and “r1 distill”. ollama is intentionally misleading people on this but the distills are much much worse

They're really not? Both subjectively and in benchmarks there is no world in which the delta between the models deserves a "much much".

Re: Run DeepSeek R1 Dynamic 1.58-bit

#248

Post by Pxtl »

Is there any good quick summary of what's special about DeepSeek? I know it's OSS and incredibly efficient, but news laymen are saying it's trained purely on AI info instead of using a corpus of tagged data... which, I assume, means it's somehow extracting weights or metadata or something from other AIs. Is that it?

Re: Run DeepSeek R1 Dynamic 1.58-bit

#249

Post by dbtc »

ddingus wrote: post #229
bnchrch wrote: post #200

Earlier quoted context omitted.

Everyone has the need for on device LLM, if the response rate was fast!

I fired the model up at 8:39, ready to chat just a bit before 8:40, during which I input the following prompt: Please define "lark" for me, as in "I did it on a lark." Output began a bit before 8:41, and was complete before 8:43: ---Begin Model Output "On a lark" means without much planning or reason, impulsively or unexpectedly. Here's how it works: "I did it on a lark." This means the person acted without a specifi…

I opened Dictionary.app and typed in lark and got the definition with etymology in under 30 seconds.

> British English informal used to suggest that an activity is foolish or a waste of time: he's serious about this music lark.

Re: Run DeepSeek R1 Dynamic 1.58-bit

#250

Post by [deleted] »

yobid20 wrote: post #93
nodja wrote: post #21

Earlier quoted context omitted.

While 192GB of ram is appealing, it's also quite expensive at $6000. For that price I rather buy a system with 5 used 3090s, which while being "only" 120GB of VRAM, you benefit from much faster tokens/s and prompt processing speed (the macs are notoriously slow at consuming large contexts).

The power requirement for 5x5090s is 10x higher , so you'll spend far more than $6000 in electricity over time.

[deleted]
Post reply on HN