Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Run DeepSeek R1 Dynamic 1.58-bit

unsloth.ai

Post reply on HN
Print view View original

131–140 of 346 posts

Re: Run DeepSeek R1 Dynamic 1.58-bit

#131

Post by sailingparrot »

yobid20 wrote: post #91
apples_oranges wrote: post #16

Random observation 1: I was running DeepSeek yesterday on my Linux with a RTX 4090 and I noticed that the models should fit into VRAM, which is 24GB. Or they are simply slow. So the Apple shared memory architecture has an advantage here. A 192GB Mx Ultra can load and process large models efficiently. Random observation 2: It's time to cancel the OpenAI subscription.

I canceled my OpenAI subscription last night, as did many many others. There were some threads in reddit with everyone chiming in they all just canceled too. imo OpenAI is done, and will go through massive cuts and probably acquired by the end of the year for a very tiny fraction of its current value.

You want to bet? The panic around deepseek is getting completely disconnected from reality.

Don’t get me wrong what DS did is great, but anyone thinking this reshape the fundamental trend of scaling laws and make compute irrelevant is dead wrong. I’m sure OpenAI doesn’t really enjoy the PR right now, but guess what OpenAI/Google/Meta/Anthropic can do if you give them a recipe for 11x more efficient training ? They can scale it to their 100k GPUs clusters and still blow everything. This will be textbook Jevons paradox.

Compute is still king and OpenAI has worked on their training platform longer than anyone.

Of course as soon as the next best model is released, we can train on its output and catch up at a fraction of the cost, and thus the infinite bunny hopping will continue.

But OpenAI is very much alive.

Re: Run DeepSeek R1 Dynamic 1.58-bit

#132

Post by samvher »

coffeebeqn wrote: post #61
deskamess wrote: post #51

Earlier quoted context omitted.

> Nvidia being down 18% The only part of DeepSeek-R1 I do not like. I hope it's over, but I am not holding my breath.

Nvidia is now up only 1906% over 5 years. What a disgrace

It crashed all the way back to June 2024 levels, eons of progress wiped out

Re: Run DeepSeek R1 Dynamic 1.58-bit

#133

Post by kebaman »

yobid20 wrote: post #91
apples_oranges wrote: post #16

Random observation 1: I was running DeepSeek yesterday on my Linux with a RTX 4090 and I noticed that the models should fit into VRAM, which is 24GB. Or they are simply slow. So the Apple shared memory architecture has an advantage here. A 192GB Mx Ultra can load and process large models efficiently. Random observation 2: It's time to cancel the OpenAI subscription.

I canceled my OpenAI subscription last night, as did many many others. There were some threads in reddit with everyone chiming in they all just canceled too. imo OpenAI is done, and will go through massive cuts and probably acquired by the end of the year for a very tiny fraction of its current value.

Doesn't Microsoft own 49% of OpenAI? They'll end up with it all as a division of Microsoft.

Re: Run DeepSeek R1 Dynamic 1.58-bit

#134

Post by bluesounddirect »

Hi small comment, please remember in china many things are sponsored by or subsidized by the government. "We[china] can do it for less.." , "it's cheaper in china.." only means the government gave us a pile of cash and help to get here .

I 100% expect some downvotes from the ccp.

Re: Run DeepSeek R1 Dynamic 1.58-bit

#136

Post by therealpygon »

immibis wrote: post #65
apples_oranges wrote: post #16

Random observation 1: I was running DeepSeek yesterday on my Linux with a RTX 4090 and I noticed that the models should fit into VRAM, which is 24GB. Or they are simply slow. So the Apple shared memory architecture has an advantage here. A 192GB Mx Ultra can load and process large models efficiently. Random observation 2: It's time to cancel the OpenAI subscription.

The real insult here is graphics card vendors refusing to make ones with more than 24GB for several years now. They do this so you'll have to buy several cards for your AI workstation. Hopefully Apple eating their lunch fixes this.

I honestly don’t know why people aren’t more upset by this and still get on their knees for Nvidia. They made the decision specifically to cripple consumer card memory because they didn’t like data centers were using them instead of buying their overpriced enterprise cards that were less performant. They removed NVLink because people were getting better performance out of their two $400 cards than the $1,500 cards Nvidia was trying to peddle. They willfully screw consumers and people love them for it.

Re: Run DeepSeek R1 Dynamic 1.58-bit

#137

Post by MyFirstSass »

Is this akin to the quants already being done to various models when you download a GGUF at 4 bits for example, or is this variable layer compression something new that can also be make existing smaller models smaller so we can fit more into say 12 or 16 gb's of vram?

Re: Run DeepSeek R1 Dynamic 1.58-bit

#138

Post by generalizations »

yobid20 wrote: post #91
apples_oranges wrote: post #16

Random observation 1: I was running DeepSeek yesterday on my Linux with a RTX 4090 and I noticed that the models should fit into VRAM, which is 24GB. Or they are simply slow. So the Apple shared memory architecture has an advantage here. A 192GB Mx Ultra can load and process large models efficiently. Random observation 2: It's time to cancel the OpenAI subscription.

I canceled my OpenAI subscription last night, as did many many others. There were some threads in reddit with everyone chiming in they all just canceled too. imo OpenAI is done, and will go through massive cuts and probably acquired by the end of the year for a very tiny fraction of its current value.

In my experience with deepseek and o1, openai's big talk about (and investment into) hallucination avoidance might save their hides here. Deepseek may be smarter, and understand complex problems better, but it also seems to make mistakes more often. (It's as if it's comprehension is better, but it's worse at memorization/recall.)

Need an LLM to one-shot some complex network scripting? as of last night, o1 is still where its at.

Re: Run DeepSeek R1 Dynamic 1.58-bit

#139

Post by sitkack »

michaelt wrote: post #115
jairuhme wrote: post #98

Earlier quoted context omitted.

At my work, we self-host some models and have found that for anything remotely similar to RAG or use cases that are very specific, the quantized models have proven to be more than sufficient. This helps us keep them running on smaller infra and generally lower costs

Personally I've noticed major changes in performance between different quantisations of the same model. Mistral's large 123B model works well (but slowly) at 4-bit quantisation, but if I knock it down to 2.5-bit quantisation for speed, performance drops to the point where I'm better off with a 70B 4-bit model. This makes me reluctant to evaluate new models in heavily quantised forms, as you're measuring the quantisat…

How are you doing your evals?

Being able to do semantic diffs of the output of the two models should tell you what you need to do.

Re: Run DeepSeek R1 Dynamic 1.58-bit

#140

Post by Gracana »

diggan wrote: post #126
Gracana wrote: post #118

Earlier quoted context omitted.

It's "easy" if you have a place to build an open frame rig with riser cables and whatnot. I can't do that, so I'm going the single slot waterblock route, which unfortunately rules out 3090s due to the memory on the back side of the PCB. It's very frustrating.

I think parents point is that NVLink no longer ships with consumer cards. Before you could buy two cards + a cable between them, and software can treat them as one card. Today you need software support for splitting between the cards, unless you go for "professional" cards or whatever they call them.

Maybe that's what they meant, and it'd be cool if nvidia still offered that on consumer cards, but thankfully you don't need it for LLM inference. The traffic between cards is very small.
Post reply on HN