Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Run DeepSeek R1 Dynamic 1.58-bit

unsloth.ai

Post reply on HN
Print view View original

81–90 of 346 posts

Re: Run DeepSeek R1 Dynamic 1.58-bit

#81

Post by infecto »

brap wrote: post #34

As someone who is out of the loop, what’s the verdict on R1? Was anyone able to reproduce the results yet? Is the claim that it only took $5M to train generally accepted? It’s a very bold claim which is really shaking up the markets, so I can’t help but wonder if it was even verified at this point.

I think the jury is out. With folks trying to replicate the process we will see if the low budget is true or not. I am still on the fence, there was comments from Scale CEO that they have a huge number of H100s they used. On the market side I think regardless if this was true or not, this gave people the opportunity to sell what is perhaps overinflated valuations.

Re: Run DeepSeek R1 Dynamic 1.58-bit

#82

Post by infecto »

huijzer wrote: post #40
brap wrote: post #34

As someone who is out of the loop, what’s the verdict on R1? Was anyone able to reproduce the results yet? Is the claim that it only took $5M to train generally accepted? It’s a very bold claim which is really shaking up the markets, so I can’t help but wonder if it was even verified at this point.

> Is the claim that it only took $5M to train generally accepted? Based on Nvidia being down 18% yesterday I would say the claim is generally accepted.

While Deepseek was an instigator in the price movements I would not say its accepted.

Re: Run DeepSeek R1 Dynamic 1.58-bit

#83

Post by merman »

miohtama wrote: post #41
danielhanchen wrote: post #32

Earlier quoted context omitted.

:) It's my goto test :) I did amp it up by adding 10 conditions and made a scoring card - I found the original R1 to sometimes forget "import os" or miss some lines as well, so I thought it was at least a good check! I also like to ask the models to create a simple basic Minecraft type game where you can break pieces and store them in your inventory, but disallow building stuff

I feel any AI can fix those problems when they can finally act. The problem AIs cannot run or debug code, or even book a hotel for me. When that is solved and an AI can interact with the code like a human does, it can fix its problems like a human does.

Exactly! Why can’t LLMs run their own code?

Re: Run DeepSeek R1 Dynamic 1.58-bit

#84

Post by ricardobeat »

Jasondells wrote: post #44

An 80% size reduction is no joke, and the fact that the 1.58-bit version runs on dual H100s at 140 tokens/s is kind of mind-blowing. That said, I’m still skeptical about how practical this really is for most people. Like, yeah, you can run it on 24GB VRAM or even with just 20GB RAM, but "slow" is an understatement—those speeds would make even the most patient person throw their hands up. And then there’s the whole re…

The repetition issue happens on simple quantization, what they are releasing is an approach that fixes that.

Re: Run DeepSeek R1 Dynamic 1.58-bit

#85

Post by Kye »

Mashimo wrote: post #64
Kye wrote: post #63

Earlier quoted context omitted.

I ran whatever version Ollama downloaded on a 3070ti (laptop version). It's reasonably fast. Generative stuff can get weird if you do prompts like "in the style of" or "a new episode of" because it doesn't seem to have much pop culture in its training data. It knows the Stargate movie, for example, and seems to have the IMDB info for the series, but goes absolutely ham trying to summarize the series. This line in the…

> ran whatever version Ollama downloaded on a 3070ti (laptop version). It's reasonably fast. Probably was not r1, but one of the other models that got trained on r1, which apparently might still be quite good.

I'm not too hip to all the LLM terminology, so maybe someone can make sense of this and see if it's r1 or something based on r1:

>>> /show info

  Model

    architecture        qwen2

    parameters          7.6B

    context length      131072

    embedding length    3584

    quantization        Q4_K_M

Re: Run DeepSeek R1 Dynamic 1.58-bit

#86

Post by dist-epoch »

cubefox wrote: post #77

For anyone wondering why "1.58" bits: 2^1.58496... = 3. The weights have one of the three states {-1, 0, 1}.

They say something else:

> We managed to selectively quantize certain layers to higher bits (like 4bit), and leave most MoE layers (like those used in GPT-4) to 1.5bit

Re: Run DeepSeek R1 Dynamic 1.58-bit

#87

Post by TeMPOraL »

orf wrote: post #73
jchw wrote: post #71

Earlier quoted context omitted.

I am very happy for you, but laptops get stolen in public in most countries.

Laptops get stolen on a train? An enclosed, single-direction space that only occasionally allows you to exit between infrequent, long-distance stops? A thing that contains ticket inspectors and a literal guard? How many laptops have you personally seen be stolen on a train?

You mean a tight, enclosed, single-direction space, crowded with people who are tired, and/or trying to relax, and/or thinking about the destination, and/or otherwise not particularly focused after hours of travel; a thing that contains ticket inspectors that show up every now and then to check tickets, and from which passengers embark and disembark at dozens point along the length of the thing, simultaneously, with no supervision or security checks.

Depending on the train type and configuration, many actually seem like pickpocket paradise.

Re: Run DeepSeek R1 Dynamic 1.58-bit

#88

Post by jchw »

orf wrote: post #73
jchw wrote: post #71

Earlier quoted context omitted.

I am very happy for you, but laptops get stolen in public in most countries.

Laptops get stolen on a train? An enclosed, single-direction space that only occasionally allows you to exit between infrequent, long-distance stops? A thing that contains ticket inspectors and a literal guard? How many laptops have you personally seen be stolen on a train?

Only on Hacker News would I have someone arguing with me that laptop theft is not a concern. You know what, you win. It's your $6,000 laptop, not mine.

Re: Run DeepSeek R1 Dynamic 1.58-bit

#89

Post by TeMPOraL »

thenickdude wrote: post #59
karamanolev wrote: post #42

Earlier quoted context omitted.

For personal usage, does it matter though? In most places residential electricity is cheap compared to everything else. In a DC context I feel it matters a lot more compared to the capex.

1x 3090 (350W power limit) already makes it feel like I'm running a fan heater under my desk, 5x would be nuts.

Place and time your use right, and you'll save a bit on heating at winter and/or at nights.

Re: Run DeepSeek R1 Dynamic 1.58-bit

#90

Post by TeMPOraL »

immibis wrote: post #65
apples_oranges wrote: post #16

Random observation 1: I was running DeepSeek yesterday on my Linux with a RTX 4090 and I noticed that the models should fit into VRAM, which is 24GB. Or they are simply slow. So the Apple shared memory architecture has an advantage here. A 192GB Mx Ultra can load and process large models efficiently. Random observation 2: It's time to cancel the OpenAI subscription.

The real insult here is graphics card vendors refusing to make ones with more than 24GB for several years now. They do this so you'll have to buy several cards for your AI workstation. Hopefully Apple eating their lunch fixes this.

> They do this so you'll have to buy several cards for your AI workstation.

AFAIK you can't do that with newer consumer cards, which is why this became an annoyance. Even a RTX 4070 Ti with its 12 GB would be fine, if you could easily stack a bunch of them like you used to be able with older cards.

Post reply on HN