Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DeepSeek-v3.2: Pushing the frontier of open large language models [pdf]

huggingface.co

Post reply on HN
Print view View original

451–460 of 485 posts

Re: DeepSeek-v3.2: Pushing the frontier of open large language models [pdf]

#451

Post by kriops »

tomhow wrote: post #273
kriops wrote: post #179

Earlier quoted context omitted.

[flagged]

Several of your comments in this subthread have broken the guidelines. The guidelines ask us not to use HN for political/ideological battle and to "assume good faith". They ask us to "be kind", "eschew flamebait", and ask that "comments should get more thoughtful and substantive, not less as a topic gets more divisive." The topic itself, like any topic, is fine to discuss here, but care must be taken to discuss it in…

Can you, by any chance, delete my account? I have tried to do so before but it is not possible through the GUI. And I see you are associated with HN.

Other than that let's be very clear that there was no personal attack. You left out the part where I explain why I think the comment was made in bad faith. I.e. the part that makes it not a personal attack. And a part which I, upon request, elaborated on in the same comment tree.

As you said: Words matter.

Re: DeepSeek-v3.2: Pushing the frontier of open large language models [pdf]

#453

Post by __alexs »

elif wrote: post #428
chazeon wrote: post #292

Earlier quoted context omitted.

Well the seemingly cheap comes with significantly degraded performance, particular for agentic use. Have you tried replacing Claude Code with some locally deployed model, say, on 4090 or 5090? I have. It is not usable.

Strictly speaking, you have not deployed any model on a 5090 because a 5090 card has never been produced. And without specifying your quantization level it's hard to know what you mean by "not usable" Anyway if you really wanted to try cheap distilled/quantized models locally you would be using used v100 Teslas and not 4 year old single chip gaming GPUs.

Are you a time traveller from the past? https://www.nvidia.com/en-gb/geforce/graphics-cards/50-serie...

Re: DeepSeek-v3.2: Pushing the frontier of open large language models [pdf]

#454

Post by ActorNightly »

eru wrote: post #355
ActorNightly wrote: post #318

Earlier quoted context omitted.

Lol its kinda suprising that the level of understanding around LLMs is so little. You already have agents, that can do a lot of "thinking", which is just generating guided context, then using that context to do tasks. You already have Vector Databases that are used as context stores with information retrieval. Fundamentally, you can have the same exact performance on a lot of task whether all the information exists i…

You should start a company and try your strategy. I hope it works! (Though I am doubtful.) In any case, models are useful, even when they don't hit these efficiency targets you are projecting. Just like cars are useful, even when they are bigger than a pack of cards.

If someone wants to fund me, Ill gladly work on this. There is no money in this though, because selling cloud service is much more profitable.

Its also not a matter of it working or not. It already works. Take a small model that fits on a GPU with a large context window, like Gemma 27b or smaller ones, give it a whole bunch of context on the topic, and ask it questions and it will generate very accurate results based on the context.

So instead of encoding everything into the model itself, you can just take training data, store it in vector DBs, and train a model to retrieve that data based on query, and then the rest of it is just training context extraction.

Re: DeepSeek-v3.2: Pushing the frontier of open large language models [pdf]

#455

Post by embedding-shape »

miki123211 wrote: post #329
tedivm wrote: post #181

Earlier quoted context omitted.

If you're trying to build AI based applications you can and should compare the costs between vendor based solutions and hosting open models with your own hardware. On the hardware side you can run some benchmarks on the hardware (or use other people's benchmarks) and get an idea of the tokens/second you can get from the machine. Normalize this for your usage pattern (and do your best to implement batch processing whe…

> with your own hardware Or with somebody else's. If you don't have strict data residency requirements, and if you aren't doing this at an extremely large scale, doing it on somebody else's hardware makes much more economic sense. If you use MoE models (al modern >70B models are MoE), GPU utilization increases with batch size. If you don't have enough requests to keep GPUs properly fed 24/7, those GPUs will end up un…

> If you use MoE models (al modern >70B models are MoE), GPU utilization increases with batch size

Isn't that true for any LLM, MoE or not? In fact, doesn't that apply to most concepts within ML, as long as it's possible to do batching at all, you can scale it up and utilize more of the GPU, until you saturate some part of the process.

Re: DeepSeek-v3.2: Pushing the frontier of open large language models [pdf]

#456

Post by boringg »

KvanteKat wrote: post #394
moffkalast wrote: post #380

Earlier quoted context omitted.

Training cost-effectiveness doesn't matter for open models since someone else ate the cost. In this case, Chinese taxpayers.

Deepseek is a private corporation funded by a hedge fund (High-Flyer). I doubt much public money was spent by the Chinese state on this. Like with LLMs in the US, the people paying for it so far are mainly investors who are betting on a return in the long to medium term.

Do you actually believe what you just wrote or are you trolling? One version at least has a foot planted in reality. The other one well...

Re: DeepSeek-v3.2: Pushing the frontier of open large language models [pdf]

#457

Post by kmacdough »

tedivm wrote: post #181
jstummbillig wrote: post #41

Earlier quoted context omitted.

How could we judge if anyone is "winning" on cost-effectiveness, when we don't know what everyones profits/losses are?

If you're trying to build AI based applications you can and should compare the costs between vendor based solutions and hosting open models with your own hardware. On the hardware side you can run some benchmarks on the hardware (or use other people's benchmarks) and get an idea of the tokens/second you can get from the machine. Normalize this for your usage pattern (and do your best to implement batch processing whe…

Furthermore, paid models are heavily subsidized by bullish investors playing for monopoly. So that tips the scales further towards Deepseek.

Re: DeepSeek-v3.2: Pushing the frontier of open large language models [pdf]

#458

Post by vitaflo »

chistev wrote: post #401
zug_zug wrote: post #16

Well props to them for continuing to improve, winning on cost-effectiveness, and continuing to publicly share their improvements. Hard not to root for them as a force to prevent an AI corporate monopoly/duopoly.

How do they make their money

Deepseek is owned by a Chinese hedge fund. It was originally created for finance and then generalized later. In any case you pay for it like any other LLM.

Re: DeepSeek-v3.2: Pushing the frontier of open large language models [pdf]

#459

Post by JosephjackJR »

chazeon wrote: post #292
tedivm wrote: post #181

Earlier quoted context omitted.

If you're trying to build AI based applications you can and should compare the costs between vendor based solutions and hosting open models with your own hardware. On the hardware side you can run some benchmarks on the hardware (or use other people's benchmarks) and get an idea of the tokens/second you can get from the machine. Normalize this for your usage pattern (and do your best to implement batch processing whe…

Well the seemingly cheap comes with significantly degraded performance, particular for agentic use. Have you tried replacing Claude Code with some locally deployed model, say, on 4090 or 5090? I have. It is not usable.

they took the already ridiculous v3.1 terminus model, added this new deepseek sparse attention thing, and suddenly it’s doing 128k context at basically half the inference cost of the old version with no measurable drop in reasoning or multilingual quality. like, imo gold medal level math and code, 100+ languages, all while sipping tokens at 14 cents per million input. that’s stupid cheap. the rl recipe they used this time also seems way more stable. no more endless repetition loops or random language switching you sometimes got with the earlier open models. it just works. what really got me is how fast the community moved. vllm support landed the same day, huggingface space was up in hours, and people are already fine-tuning it for agent stuff and long document reasoning. i’ve been playing with it locally and the speed jump on long prompts is night and day. feels like the gap to the closed frontier models just shrank again. anyone else tried it yet?

Re: DeepSeek-v3.2: Pushing the frontier of open large language models [pdf]

#460

Post by embedding-shape »

storus wrote: post #413
embedding-shape wrote: post #397

Earlier quoted context omitted.

6 tok/sec might be acceptable for a dense model that doesn't do thinking, but for something like DeepSeek 3.2 that does do reasoning, 6 tok/sec isn't acceptable for anything else but async/batched stuff, sadly. Even for a response with just 100 tokens we're talking a minute for it to just write the response, for anything except the smallest of prompts you'll easily be hitting 1000 tokens (600 seconds!). Maybe my 6000…

Dunno, DeepSeek on MacStudio doesn't feel much slower than when using it directly on deepseek.com; 6t/s is still around 24 characters per second which is faster than many people could read. I also have 6000 Pro but you won't fit any large model there and to be able to run DeepSeek R1/3.1/3.2 671B at Q4 you'd need 5-6 of them depending on the communication overhead. MacStudio is the simplest solution to run it locally…

> 6t/s is still around 24 characters per second which is faster than many people could read.

But again, not if you're using thinking/reasoning, which if you want to use this specific model properly, you are. Then you have a huge delay before the actual response comes through.

> MacStudio is the simplest solution to run it locally.

Obviously, that's Apple's core value proposition after all :) One does not acquire a state-of-the-art GPU and then expect simple stuff, especially when it's a fairly uncommon and new one. You cannot really be afraid of diving into CUDA code and similar fun rabbit holes. Simply two very different audiences for the two alternatives, and the Apple way is the simpler one, no doubt about it.

Post reply on HN