Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DeepSeek-v3.1

api-docs.deepseek.com

Post reply on HN
Print view View original

181–190 of 273 posts

Re: DeepSeek-v3.1

#181

Post by segmondy »

tw1984 wrote: post #72
danielhanchen wrote: post #39

For local runs, I made some GGUFs! You need around RAM + VRAM >= 250GB for good perf for dynamic 2bit (2bit MoE, 6-8bit rest) - can also do SSD offloading but it'll be slow. ./llama.cpp/llama-cli -hf unsloth/DeepSeek-V3.1-GGUF:UD-Q2_K_XL -ngl 99 --jinja -ot ".ffn_.*_exps.=CPU" More details on running + optimal params here: https://docs.unsloth.ai/basics/deepseek-v3.1

for such dynamic 2bit, is there any benchmark results showing how many performance I would give up compared to the original model? thanks.

if you are running a 2bit quant, you are not giving up performance but gaining 100% performance since the alternative is usually 0%. Smaller quants are for folks who won't be able to run anything at all, so you run the largest you can run relative to your hardware. I for instance often ran Q3_K_L, I don't think of how much performance I'm giving up, but rather how without Q3, I won't be able to run it at all. With that said, for R1, I did some tests against 2 public interfaces and my local Q3 crushed them. The problem with a lot of model providers is we can never be sure what they are serving up and could take shortcuts to maximize profit.

Re: DeepSeek-v3.1

#182

Post by xadhominemx »

okasaki wrote: post #108
rsanek wrote: post #86

Looks to be the ~same intelligence as gpt-oss-120B, but about 10x slower and 3x more expensive? https://artificialanalysis.ai/models/deepseek-v3-1-reasoning

My experience is that gpt-oss doesn't know much about obscure topics, so if you're using it for anything except puzzles or coding in popular languages, it won't do well as the bigger models. It's knowledge seems to be lacking even compared to gpt3. No idea how you'd benchmark this though.

> My experience is that gpt-oss doesn't know much about obscure topics

That is the point of these small models. Remove the bloat of obscure information (address that with RAG), leaving behind a core “reasoning” skeleton.

Re: DeepSeek-v3.1

#183

Post by diggan »

danielhanchen wrote: post #39

For local runs, I made some GGUFs! You need around RAM + VRAM >= 250GB for good perf for dynamic 2bit (2bit MoE, 6-8bit rest) - can also do SSD offloading but it'll be slow. ./llama.cpp/llama-cli -hf unsloth/DeepSeek-V3.1-GGUF:UD-Q2_K_XL -ngl 99 --jinja -ot ".ffn_.*_exps.=CPU" More details on running + optimal params here: https://docs.unsloth.ai/basics/deepseek-v3.1

> More details on running + optimal params here: https://docs.unsloth.ai/basics/deepseek-v3.1

Was that document almost exclusively written with LLMs? I looked at it last night (~8 hours ago) and it was riddled with mistakes, most egregious was that the "Run with Ollama" section had instructions for how to install Ollama, but then the shell commands were actually running llama.cpp, a mistake probably no human would make.

Do you have any plans on disclosing how much of these docs are written by humans vs not?

Regardless, thanks for the continued release of quants and weights :)

Re: DeepSeek-v3.1

#184

Post by diggan »

guerrilla wrote: post #179

So, is the output price there why most models are extremely verbose? Is it just a ploy to make extra cash? It's super annoying that I have to constantly tell it to be more and more concise.

> It's super annoying that I have to constantly tell it to be more and more concise.

While system promting is the easy way of limiting the output in a somewhat predictable manner, have you tried setting `max_tokens` when doing inference? For me that works very well for constraining the output, if you set it to 100 you get very short answers while if you set it to 10,000 you can very long responses.

Re: DeepSeek-v3.1

#185

Post by bavell »

pdimitar wrote: post #16
gunalx wrote: post #13

Earlier quoted context omitted.

You would still need drivers and all the stuff difficult with nvidia in linux with a egpu. (Its not nessecarily terrible just suboptimal) Rather just add the second GPU in the Workstation, or just run the llm in your AMD GPU.

Oh, we can run LLMs efficiently with AMD GPUs now? Pretty cool, I haven't been following, thank you.

IDK about "efficiently" but we've been able to run llms locally with AMD for 1.5-2 years now

Re: DeepSeek-v3.1

#186

Post by Plasmoid2000ad »

pdimitar wrote: post #16
gunalx wrote: post #13

Earlier quoted context omitted.

You would still need drivers and all the stuff difficult with nvidia in linux with a egpu. (Its not nessecarily terrible just suboptimal) Rather just add the second GPU in the Workstation, or just run the llm in your AMD GPU.

Oh, we can run LLMs efficiently with AMD GPUs now? Pretty cool, I haven't been following, thank you.

Yes - I'm running a LM Studio on windows on a 6800xt, and everything works more-or-less out of the box using always using Vulkan llama.cpp on the gpu I believe.

There's also ROCm. That's not working for me in LM Studio at the moment. I used that early last year to get some LLMs and stable diffusion running. As far as I can tell, it was faster before, but Vulkan implementations have caught up or something - so much the mucking about isn't often worth it. I believe ROCm is hit or miss for a lot of people, especially on windows.

Re: DeepSeek-v3.1

#187

Post by petesergeant »

indigodaddy wrote: post #168
petesergeant wrote: post #129

Earlier quoted context omitted.

With all these things, it depends on your own eval suite. gpt-oss-120b works as well as o4-mini over my evals, which means I can run it via OpenRouter on Cerebras where it's SO DAMN FAST and like 1/5th the price of o4-mini.

How would you compare gpt-oss-120b to (for coding): Qwen3-Coder-480B-A35B-Instruct GLM4.5 Air Kimi K2 DeepSeek V3 0324 / R1 0528 GPT-5 Mini Thanks for any feedback!

I’m afraid I don’t use any of those for coding

Re: DeepSeek-v3.1

#188

Post by okasaki »

xadhominemx wrote: post #182
okasaki wrote: post #108

Earlier quoted context omitted.

My experience is that gpt-oss doesn't know much about obscure topics, so if you're using it for anything except puzzles or coding in popular languages, it won't do well as the bigger models. It's knowledge seems to be lacking even compared to gpt3. No idea how you'd benchmark this though.

> My experience is that gpt-oss doesn't know much about obscure topics That is the point of these small models. Remove the bloat of obscure information (address that with RAG), leaving behind a core “reasoning” skeleton.

Yeah I guess. Just wanted to say the size difference might be accounted for by the model knowing more.

Seems more user-friendly to bake it in.

Re: DeepSeek-v3.1

#189

Post by solarkraft »

mFixman wrote: post #113
danielhanchen wrote: post #88

Earlier quoted context omitted.

As an update, I pushed https://github.com/unslothai/unsloth-zoo/commit/ae675a0a2d20... (1) Removed and disabled sudo (2) Installing via apt-get will ask user's input() for permission (3) Added an error if failed llama.cpp and provides instructions to manual compile llama.cpp

Maybe it's a personal preference, but I don't want external programs to ever touch my package manager, even with permission. Besides, this will fail loudly for systems that don't use `apt-get`. I would just ask the user to install the package, and _maybe_ show the command line to install it (but never run it).

I like it when software does work for me.

Quietly installing stuff at runtime is shady for sure, but why not if I consent?

Re: DeepSeek-v3.1

#190

Post by noodletheworld »

dmos62 wrote: post #170
irthomasthomas wrote: post #157

Earlier quoted context omitted.

It just means that once you send your test questions to a model API, that company now has your test. So 'private' benchmarks take it on faith that the companies won't look at those requests and tune their models or prompts to beat them.

Sounds a bit presumptious to me. Sure, they have your needle, but they also need a cost-efficient way to find it in their hay stack.

Security through obscurity is not security.

Your api key is linked to your credit card, which is linked to your identity.

…but hey, youre right.

Lets just trust them not to be cheating. Cool.

Post reply on HN