Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Run DeepSeek R1 Dynamic 1.58-bit

unsloth.ai

Post reply on HN
Print view View original

331–340 of 346 posts

Re: Run DeepSeek R1 Dynamic 1.58-bit

#331

Post by xiphias2 »

emseetech wrote: post #312
TheTaytay wrote: post #310

Earlier quoted context omitted.

How'd it go, and which client are you using? :)

Pretty rough. Using LM Studio, trying to load the model throws an error of "insufficient system resources." I disabled this error, set the context length to 1024 and was able to get 0.24 tokens per second. Comparatively, the 32B distill model gets about 20 tokens per second. And it became incredibly flaky, using up all available ram, and crashing the whole system a few times. While the M4 Max 128GB handles the 32B we…

Maybe VLLM is better at inferencing MoE (also you can set the number of experts to use).

In theory half of the model fits to RAM, so it should be GPU limited if memory management is smart.

Re: Run DeepSeek R1 Dynamic 1.58-bit

#332

Post by brookst »

hdjjhhvvhga wrote: post #309
brookst wrote: post #176

Earlier quoted context omitted.

Or if you want a large model but don’t need high performance, get a Mac with 128GB UMA.

How many tokens/s would you get in such a setup?

This Reddit thread says an m3 max 128GB gets 23 tokens/sec with deepseek r1 32B, and 4 tokens / sec with 70b: https://www.reddit.com/r/LocalLLaMA/comments/1i69dhz/deepsee...

Re: Run DeepSeek R1 Dynamic 1.58-bit

#333

Post by sureglymop »

phaedrix wrote: post #323
sureglymop wrote: post #315

Earlier quoted context omitted.

I have Kagi Ultimate and it is nice for this. But a cheaper suggestion would be to use OpenRouter and then use these models via Fireworks or TogetherAI. It also integrates into much more applications. AFAIK Kagi doesn't document a user facing API for the assistant feature.

Unfortunately those are both 10-15x the cost of deepseek direct. Deepinfra is pretty cheap though as a deepseek provider.

Sure. I meant moreso that this would be cheaper than Kagi while providing the same selection of models.

As for deepseek, I couldn't even sign up because my email domain is not on their whitelist. To just try it out for now I don't mind the increased cost.

Re: Run DeepSeek R1 Dynamic 1.58-bit

#334

Post by crazygringo »

vladms wrote: post #224
crazygringo wrote: post #184

Earlier quoted context omitted.

You're going to take your laptop with you into the toilet on the train...? I don't think I've ever seen a human being do that before on a train. Not to go to the toilet, nor to grab a coffee in another car. You can't be paranoid about everything. My friend in France had put his laptop back into his bag where it wasn't visible and assumed that was good enough, but someone must have seen him do it and just took the who…

Yes, if I go to the toilet I take my backpack/small bag with me, because usually I have valuable stuff in them and are easy to carry. This does not apply to a large bag (in which I don't put valuable stuff). The standard is mine and I follow it. The same way I find absurd not to do it, you find it unreasonable to do it. I find the expectation that things are not stolen (if unsupervised in public places) strange consi…

I mean, that's great for you, but it's not just what 99% of people do. You don't usually see people take their backpack into a train bathroom. I've taken a lot of trains and sat near the bathroom often enough (unfortunately). But like I said, it applies to the cafe car too.

If you find it absurd how 99% of people act on long-distance trains, I don't know what to tell you.

Re: Run DeepSeek R1 Dynamic 1.58-bit

#335

Post by ddingus »

jpc0 wrote: post #295
ddingus wrote: post #227

Earlier quoted context omitted.

I have MLCCHAT on my old Note 9 phone. It is actually still a great phone, but has 5GB RAM. Running an on device model is the first and only use case the RAM actually matters. And it has a headphone jack, OK? I just hate Bluetooth earbuds. And yeah, it isna problem, but I digress. When I run a 2.5B model, I get respectable output. Takes a minute or two to process the context, then output begins at somewhere on the or…

> First aid, how to make fires, materials and uses This scares me more than it should... Please do not trust an AI in actual life and death situations... Sure if it is literally your only option, but this implies you have a device on you that could make a phone call to an emergency number where a real human with real training and actually correct knowledge can assist you. Even as an avid hiker the amount of times I'v…

I asked the question. It did a respectable job.

For choking, as an example, it outlines all the basics, affirmed it was educational info only, and urged the help of a professional.

All in all, reasonable.

Your point stands, but this little model 2.5B has clear bounds it seems.

I plan to poke at it a few more times asking for higher risk info, curious about what it may do.

Re: Run DeepSeek R1 Dynamic 1.58-bit

#336

Post by greenavocado »

im3w1l wrote: post #252
greenavocado wrote: post #160

Earlier quoted context omitted.

Would fine-tuning by using a LoRA paper over the censorship to a large degree?

Why even bother decensoring it (except academic curiosity ig)? There are a million other ways you can learn about those subjects. The people making the model probably don't really give a shit about politics and just did the minimum to avoid being embarassed, but if people start jailbreaking it they will be forced to care.

Because the best way to learn is through interrogation.

I don't give a damn about ideology I just want everything ever thought or written searchable and interactive

Re: Run DeepSeek R1 Dynamic 1.58-bit

#337

Post by sebastiennight »

Kye wrote: post #85
Mashimo wrote: post #64

Earlier quoted context omitted.

> ran whatever version Ollama downloaded on a 3070ti (laptop version). It's reasonably fast. Probably was not r1, but one of the other models that got trained on r1, which apparently might still be quite good.

I'm not too hip to all the LLM terminology, so maybe someone can make sense of this and see if it's r1 or something based on r1: >>> /show info Model architecture qwen2 parameters 7.6B context length 131072 embedding length 3584 quantization Q4_K_M

So... it's not R1 itself.

It's a model called Qwen, trained by Alibaba, which the DeepSeek team has used to "distill" knowledge from their own (100x bigger) model.

Think of it as forcing a junior Qwen to listen in while the smarter, PhD-level model was asked thousands of tough problems. It will acquire some of that knowledge and learn a lot of the reasoning process.

It cannot become exactly as smart, for the same reason a dog can learn lots of tricks from a human but not become human-level itself: it doesn't have enough neurons/capacity. Here, Qwen is a 7B model so it can't cram within 7 billion parameters as much data as you can cram into 671 billion. It can literally only learn 1% as much, BUT the distillation process is cleverly built and allows to focus on the "right" 1%.

Then this now-smarter Qwen is quantized. This means that we take its parameters (16-bit floats, super precise numbers) and truncate them to make them use less memory space. This also makes it less precise. Think of it as taking a super high resolution movie picture and compressing it into a small GIF. You lose some information, but the gist of it is preserved.

As a result of both of these transformations, you get something that can run on your local machine — but is a bit dumber than the original — because it's about 400 times smaller than the real deal.

Re: Run DeepSeek R1 Dynamic 1.58-bit

#338

Post by blacklightpy »

whimsicalism wrote: post #192
wklauss wrote: post #167

Earlier quoted context omitted.

Ollama is pretty clear about it, it's not like they are trying to deceive. You can also download the 671B model with Ollama, if you like.

no they are not, they intentionally remove every reference to this not being r1 from the cli and changed the names from the ones both Deepseek and Huggingface used.

Yet, I did not see a single issue made on the GitHub repository, and I just made one myself (https://github.com/ollama/ollama/issues/8698).

Re: Run DeepSeek R1 Dynamic 1.58-bit

#339

Post by blacklightpy »

bluesounddirect wrote: post #134

Hi small comment, please remember in china many things are sponsored by or subsidized by the government. "We[china] can do it for less.." , "it's cheaper in china.." only means the government gave us a pile of cash and help to get here . I 100% expect some downvotes from the ccp.

It's just the way it should be.

Re: Run DeepSeek R1 Dynamic 1.58-bit

#340

Post by cynicalpeace »

tivert wrote: post #324
cynicalpeace wrote: post #194

Earlier quoted context omitted.

> And that's a really important strategic advantage China has versus America, which has such an insane fixation on pure(ish) free markets and free trade that it gives away its advantages in strategic industry after strategic industry. > Some people falsely infer from the experience with the Soviet Union that freer markets always win geopolitical competition, but that's false. The data we have is 500 years of free mar…

> The data we have is 500 years of free markets in the western world and the verdict is overwhelmingly: Yes, more freedom means more winning. No, more freedom means more winning to a point . Past that point it does not, and I'd argue that's where the US is. > Just invite some incompetent bureaucrat over your house to dictate how you should cook and you'll quickly agree. That's supposed to be convincing, somehow? Just…

"I'm from the government and I'm here to help" are words you like to hear? At least anybody else I can tell to leave me alone.
Post reply on HN