Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Run DeepSeek R1 Dynamic 1.58-bit

unsloth.ai

Post reply on HN
Print view View original

161–170 of 346 posts

Re: Run DeepSeek R1 Dynamic 1.58-bit

#161

Post by flir »

generalizations wrote: post #138
yobid20 wrote: post #91

Earlier quoted context omitted.

I canceled my OpenAI subscription last night, as did many many others. There were some threads in reddit with everyone chiming in they all just canceled too. imo OpenAI is done, and will go through massive cuts and probably acquired by the end of the year for a very tiny fraction of its current value.

In my experience with deepseek and o1, openai's big talk about (and investment into) hallucination avoidance might save their hides here. Deepseek may be smarter, and understand complex problems better, but it also seems to make mistakes more often. (It's as if it's comprehension is better, but it's worse at memorization/recall.) Need an LLM to one-shot some complex network scripting? as of last night, o1 is still wh…

My experience gels with yours. Given the same code sample, DeepSeek has better, more creative suggestions about how to improve it, but it can't implement them without breaking the code. o1, generally, can implement DeepSeek's suggestions successfully. I think chaining them together might have quite interesting results.

Re: Run DeepSeek R1 Dynamic 1.58-bit

#162

Post by iamnotagenius »

Gracana wrote: post #94
timschmidt wrote: post #75

Earlier quoted context omitted.

On Zen5 you also get AVX512 which llamafile takes advantage of for drastically improved speeds during prompt processing, at least. And the 12 channel Epycs actually seem to have more memory bandwidth available than the Apple M series. Especially considering it's all available to the CPU as opposed to just some portion of it.

Maybe EPYC can make better use of the available bandwidth, but for comparison I have a water cooled Xeon W5-3435X running at 4.7GHz all-core with 8 channels of DDR5-6400, and CPU inference is still dog slow. With a 70B Q8 model I get 1 tok/s, which is a lot less than I thought I would get with 410GB/s max RAM bandwidth. If I run on 5x A4000s I get 6.1 tok/s, which makes sense... 448GB/s / 70GB = 6.4 tok/s max.

very strange as I get on old i5-12400+DDR4 2 tok/sec with 14B/q8 model.

Re: Run DeepSeek R1 Dynamic 1.58-bit

#163

Post by regularfry »

lolinder wrote: post #152
sailingparrot wrote: post #131

Earlier quoted context omitted.

You want to bet? The panic around deepseek is getting completely disconnected from reality. Don’t get me wrong what DS did is great, but anyone thinking this reshape the fundamental trend of scaling laws and make compute irrelevant is dead wrong. I’m sure OpenAI doesn’t really enjoy the PR right now, but guess what OpenAI/Google/Meta/Anthropic can do if you give them a recipe for 11x more efficient training ? They ca…

> The panic around deepseek is getting completely disconnected from reality. This entire hype cycle has long been completely disconnected from reality. I've watched a lot of hype waves, and I've never seen one that oscillates so wildly. I think you're right that OpenAI isn't as hurt by DeepSeek as the mass panic would lead one to believe, but it's also true that DeepSeek exposes how blown out of proportion the initia…

In a sense it doesn't, in that if DeepSeek can do this, making OpenAI-type capabilities available for Llama-type infrastructure costs, then if you apply OpenAI scale infrastructure again to a much more efficient training/evaluation system, everything multiplies back up. I think that's where they'll have to head: using their infrastructure moat (such as it is) to apply these efficiency learnings to allow much more capable models at the top end. Yes, they can't sleep-walk into it, but I don't think that was ever the game.

Re: Run DeepSeek R1 Dynamic 1.58-bit

#164

Post by regularfry »

immibis wrote: post #65
apples_oranges wrote: post #16

Random observation 1: I was running DeepSeek yesterday on my Linux with a RTX 4090 and I noticed that the models should fit into VRAM, which is 24GB. Or they are simply slow. So the Apple shared memory architecture has an advantage here. A 192GB Mx Ultra can load and process large models efficiently. Random observation 2: It's time to cancel the OpenAI subscription.

The real insult here is graphics card vendors refusing to make ones with more than 24GB for several years now. They do this so you'll have to buy several cards for your AI workstation. Hopefully Apple eating their lunch fixes this.

The 5090 is 32GB out of the box. Not that that's anywhere near the top of what you can do on an Apple, but at least it's movement.

Re: Run DeepSeek R1 Dynamic 1.58-bit

#167

Post by wklauss »

woadwarrior01 wrote: post #99
Mashimo wrote: post #64

Earlier quoted context omitted.

> ran whatever version Ollama downloaded on a 3070ti (laptop version). It's reasonably fast. Probably was not r1, but one of the other models that got trained on r1, which apparently might still be quite good.

Ollama has been deliberately misrepresenting R1 distill models as "R1" for marketing purposes. A lot of "AI" influencers on social media are unabashedly doing the same. Ollama's default "R1" model is a 4-bit RTN quantized 7B model, which is nowhere close to the real R1 (a 671B parameter fp8 MoE). https://www.reddit.com/r/LocalLLaMA/comments/1i8ifxd/ollama_...

Ollama is pretty clear about it, it's not like they are trying to deceive. You can also download the 671B model with Ollama, if you like.

Re: Run DeepSeek R1 Dynamic 1.58-bit

#168

Post by mcv »

huijzer wrote: post #74
willsmith72 wrote: post #60

Earlier quoted context omitted.

Because the markets are rational, all-knowing, and have never been wrong?

That was not the question.

It is if you're using market movements as evidence of anything factual. If markets aren't rational, you can't use them that way.

Re: Run DeepSeek R1 Dynamic 1.58-bit

#169

Post by woadwarrior01 »

wklauss wrote: post #167
woadwarrior01 wrote: post #99

Earlier quoted context omitted.

Ollama has been deliberately misrepresenting R1 distill models as "R1" for marketing purposes. A lot of "AI" influencers on social media are unabashedly doing the same. Ollama's default "R1" model is a 4-bit RTN quantized 7B model, which is nowhere close to the real R1 (a 671B parameter fp8 MoE). https://www.reddit.com/r/LocalLLaMA/comments/1i8ifxd/ollama_...

Ollama is pretty clear about it, it's not like they are trying to deceive. You can also download the 671B model with Ollama, if you like.

Yeah, they're so clear in fact that they call the distilled models "R1" in the url and everywhere on the page[1], instead of using the "DeepSeek-R1-Distill-" prefix, as DeepSeek themselves do[2].

[1]: https://ollama.com/library/deepseek-r1

[2]: https://github.com/deepseek-ai/DeepSeek-R1#deepseek-r1-disti...

Re: Run DeepSeek R1 Dynamic 1.58-bit

#170

Post by tivert »

bluesounddirect wrote: post #134

Hi small comment, please remember in china many things are sponsored by or subsidized by the government. "We[china] can do it for less.." , "it's cheaper in china.." only means the government gave us a pile of cash and help to get here . I 100% expect some downvotes from the ccp.

> Hi small comment, please remember in china many things are sponsored by or subsidized by the government. "We[china] can do it for less.." , "it's cheaper in china.." only means the government gave us a pile of cash and help to get here .

And that's a really important strategic advantage China has versus America, which has such an insane fixation on pure(ish) free markets and free trade that it gives away its advantages in strategic industry after strategic industry.

Some people falsely infer from the experience with the Soviet Union that freer markets always win geopolitical competition, but that's false.

Post reply on HN