Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DeepSeek-v3.2: Pushing the frontier of open large language models [pdf]

huggingface.co

Post reply on HN
Print view View original

371–380 of 485 posts

Re: DeepSeek-v3.2: Pushing the frontier of open large language models [pdf]

#371

Post by cubefox »

reilly3000 wrote: post #246
cubefox wrote: post #133

Earlier quoted context omitted.

> There are plenty of 3rd party and big cloud options to run these models by the hour or token. Which ones? I wanted to try a large base model for automated literature (fine-tuned models are a lot worse at it) but I couldn't find a provider which makes this easy.

If you’re already using GCP, Vertex AI is pretty good. You can run lots of models on it: https://docs.cloud.google.com/vertex-ai/generative-ai/docs/m... Lambda.ai used to offer per-token pricing but they have moved up market. You can still rent a B200 instance for sub $5/hr which is reasonable for experimenting with models. https://app.hyperbolic.ai/models Hyperbolic offers both GPU hosting and token pricing for popu…

> If you’re already using GCP, Vertex AI is pretty good. You can run lots of models on it:

> https://docs.cloud.google.com/vertex-ai/generative-ai/docs/m...

I don't see any large base models there. A base model is a pretrained foundation model without fine tuning. It just predicts text.

> Lambda.ai used to offer per-token pricing but they have moved up market. You can still rent a B200 instance for sub $5/hr which is reasonable for experimenting with models.

A B200 is probably not enough: it has just 192 GB RAM while DeepSeek-V3.2-Exp-Base, the base model for DeepSeek-V3.2, has 685 billion BF16 parameters. Though I assume they have larger options. The problem is that all the configuration work is then left to the user, which I'm not experienced in.

> https://app.hyperbolic.ai/models Hyperbolic offers both GPU hosting and token pricing for popular OSS models

Thanks. They do indeed have a single base model: Llama 3.1 405B BASE. This one is a bit older (July 2024) and probably not as good as the base model for the new DeepSeek release. But that might the the best one can do, as there don't seem to be any inference providers which have deployed a DeepSeek or even Kimi base model.

Re: DeepSeek-v3.2: Pushing the frontier of open large language models [pdf]

#372

Post by evrenesat »

culi wrote: post #262
hammeiam wrote: post #222

Earlier quoted context omitted.

Sparse Attention, it's the highlight of this model as per the paper

How did we come to the place that the most transparent and open models are now coming out of China—freely sharing their research and source code—while all the American ones are fully locked down

China needs to build the world's trust and respect, while the US is slowly but surely losing theirs.

Re: DeepSeek-v3.2: Pushing the frontier of open large language models [pdf]

#373

Post by GTP »

Muromec wrote: post #217
tedivm wrote: post #181

Earlier quoted context omitted.

If you're trying to build AI based applications you can and should compare the costs between vendor based solutions and hosting open models with your own hardware. On the hardware side you can run some benchmarks on the hardware (or use other people's benchmarks) and get an idea of the tokens/second you can get from the machine. Normalize this for your usage pattern (and do your best to implement batch processing whe…

>That's just one factor though. Another is what hardware you can actually run things on. DeepSeek and Qwen will function on cheap GPUs that other models will simply choke on. What's cheap nowdays? I'm out of the loop. Does anything ever run on integrated AMD that is Ryzen AI that comes in framework motherboards? Is under 1k americans cheap?

Not really in the loop either, but when Deepseek R1 was released, I sumbled upon this YouTube channel [1] that made local AI PC builds in the 1000-2000$ range. But he doesn't always use GPUs, maybe the cheaper builds were CPU plus a lot of RAM, I don't remember.

[1] https://youtube.com/@digitalspaceport?si=NrZL7MNu80vvAshx

Re: DeepSeek-v3.2: Pushing the frontier of open large language models [pdf]

#374

Post by JSR_FDED »

ActorNightly wrote: post #318
JSR_FDED wrote: post #187

Earlier quoted context omitted.

Nobody is winning until cars are the size of a pack of cards. Which is big enough to transport even the largest cargo.

Lol its kinda suprising that the level of understanding around LLMs is so little. You already have agents, that can do a lot of "thinking", which is just generating guided context, then using that context to do tasks. You already have Vector Databases that are used as context stores with information retrieval. Fundamentally, you can have the same exact performance on a lot of task whether all the information exists i…

https://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect

Re: DeepSeek-v3.2: Pushing the frontier of open large language models [pdf]

#375

Post by pembrook »

deadfoxygrandpa wrote: post #328
pembrook wrote: post #287

Earlier quoted context omitted.

Yes, he did, and it was fundamental to his entire economic philosophy: https://en.wikipedia.org/wiki/Tendency_of_the_rate_of_profit...

no, he didn't, and your link has nothing to do with your fallacy you were talking about

It absolutely does, and the fact that now 2 marxists (which I can see from your comment history) have a total inability to offer any actual rebuttal, does not surprise me.

Re: DeepSeek-v3.2: Pushing the frontier of open large language models [pdf]

#376

Post by timschmidt »

eru wrote: post #359
timschmidt wrote: post #314

Earlier quoted context omitted.

Someone always crawls out of the woodwork to repeat this supposed "fact" which hasn't been true for the entire half-century it's been repeated. Jim Keller (designer of most of the great CPUs of the last couple decades) gave a convincing presentation several years ago about just how not-true it is: https://www.youtube.com/watch?v=oIG9ztQw2Gc Everything he says in it still applies today. Intel struggled for a decade, a…

During the 1990s (and for some years before and after) we got 'Dennard scaling'. The frequency of processors tended to increase exponentially, too, and featured prominently in advertising and branding. I suspect many people conflated Dennard scaling with Moore's law and the demise of Dennard scaling is what contributes to the popular imagination that Moore's law is dead: frequencies of processors have essentially sta…

Yup. Since then we've seen scaling primarily in transistor count, though clock speed has increased slowly as well. Increased transistor count has led to increasingly complex and capable instruction decode, branch prediction, out of order execution, larger caches, and wider execution pipelines in attempt to increase single-threaded performance. We've also seen the rise of embarrassingly parallel architectures like GPUs which more effectively make use of additional transistors despite lower clock speeds. But Moore's been with us the whole time.

Chiplets and advanced packaging are the latest techniques improving scaling and yield keeping Moore alive. As well as continued innovation in transistor design, light sources, computational inverse lithography, and wafer scale designs like Cerebras.

Re: DeepSeek-v3.2: Pushing the frontier of open large language models [pdf]

#378

Post by badmonster »

jstummbillig wrote: post #41
zug_zug wrote: post #16

Well props to them for continuing to improve, winning on cost-effectiveness, and continuing to publicly share their improvements. Hard not to root for them as a force to prevent an AI corporate monopoly/duopoly.

How could we judge if anyone is "winning" on cost-effectiveness, when we don't know what everyones profits/losses are?

Good point. Could usage patterns + inference costs give us proxy metrics? What would be a fair baseline?

Re: DeepSeek-v3.2: Pushing the frontier of open large language models [pdf]

#379

Post by cubefox »

big_man_ting wrote: post #240
cubefox wrote: post #133

Earlier quoted context omitted.

> There are plenty of 3rd party and big cloud options to run these models by the hour or token. Which ones? I wanted to try a large base model for automated literature (fine-tuned models are a lot worse at it) but I couldn't find a provider which makes this easy.

have you checked OpenRouter if they offer any providers who serve the model you need?

I searched for "base" and the best available base model seems to be indeed Llama 3.1 405B Base at Hyperbolic.ai, as mentioned in the comment above.

Re: DeepSeek-v3.2: Pushing the frontier of open large language models [pdf]

#380

Post by moffkalast »

ericskiff wrote: post #53
jstummbillig wrote: post #41

Earlier quoted context omitted.

How could we judge if anyone is "winning" on cost-effectiveness, when we don't know what everyones profits/losses are?

I believe this was a statement on cost per token to us as consumers of the service

Training cost-effectiveness doesn't matter for open models since someone else ate the cost. In this case, Chinese taxpayers.
Post reply on HN