Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DeepSeek-v3.1

api-docs.deepseek.com

Post reply on HN
Print view View original

101–110 of 273 posts

Re: DeepSeek-v3.1

#101

Post by zargon »

danielhanchen wrote: post #39

For local runs, I made some GGUFs! You need around RAM + VRAM >= 250GB for good perf for dynamic 2bit (2bit MoE, 6-8bit rest) - can also do SSD offloading but it'll be slow. ./llama.cpp/llama-cli -hf unsloth/DeepSeek-V3.1-GGUF:UD-Q2_K_XL -ngl 99 --jinja -ot ".ffn_.*_exps.=CPU" More details on running + optimal params here: https://docs.unsloth.ai/basics/deepseek-v3.1

Thanks for your great work with quants. I would really appreciate UD GGUFs for V3.1-Base (and even more so, GLM-4.5-Base + Air-Base).

Re: DeepSeek-v3.1

#102

Post by woile »

danielhanchen wrote: post #99
devin wrote: post #94

Earlier quoted context omitted.

Don't optimize for these people.

Yep agreed - I primarily thought it was a reasonable "hack", but it's pretty bad security wise, so apologies again. The current solution hopefully is in between - ie sudo is gone, apt-get will run only after the user agrees by pressing enter, and if it fails, it'll tell the user to read docs on installing llama.cpp

Don't apologize, you are doing amazing work. I appreciate the effort you put.

Usually you don't make assumptions on the host OS, just try to find the things you need and if not, fail, ideally with good feedback. If you want to provide the "hack", you can still do it, but ideally behind a flag, `allow_installation` or something like that. This is, if you want your code to reach broader audiences.

Re: DeepSeek-v3.1

#103

Post by epolanski »

esafak wrote: post #5

It seems behind Qwen3 235B 2507 Reasoning (which I like) and gpt-oss-120B: https://artificialanalysis.ai/models/deepseek-v3-1-reasoning Pricing: https://openrouter.ai/deepseek/deepseek-chat-v3.1

I too like Qwen a lot, it's one of the best models for programming, I generally use it via the chat.

Re: DeepSeek-v3.1

#104

Post by exe34 »

pshirshov wrote: post #60
danielhanchen wrote: post #39

For local runs, I made some GGUFs! You need around RAM + VRAM >= 250GB for good perf for dynamic 2bit (2bit MoE, 6-8bit rest) - can also do SSD offloading but it'll be slow. ./llama.cpp/llama-cli -hf unsloth/DeepSeek-V3.1-GGUF:UD-Q2_K_XL -ngl 99 --jinja -ot ".ffn_.*_exps.=CPU" More details on running + optimal params here: https://docs.unsloth.ai/basics/deepseek-v3.1

By the way, I'm wondering why unsloth (a goddamn python library) tries to run apt-get with sudo (and fails on my nixos). Like how tf we are supposed to use that?

hey fellow crazy person! slight tangent: one thing that helps keep me grounded with "LLMs are doing much more than regurgitation" is watching them try to get things to work on nixos - and hitting every rake on the way to hell!

nixos is such a great way to expose code doing things it shouldn't be doing.

Re: DeepSeek-v3.1

#105

Post by danielhanchen »

zargon wrote: post #101
danielhanchen wrote: post #39

For local runs, I made some GGUFs! You need around RAM + VRAM >= 250GB for good perf for dynamic 2bit (2bit MoE, 6-8bit rest) - can also do SSD offloading but it'll be slow. ./llama.cpp/llama-cli -hf unsloth/DeepSeek-V3.1-GGUF:UD-Q2_K_XL -ngl 99 --jinja -ot ".ffn_.*_exps.=CPU" More details on running + optimal params here: https://docs.unsloth.ai/basics/deepseek-v3.1

Thanks for your great work with quants. I would really appreciate UD GGUFs for V3.1-Base (and even more so, GLM-4.5-Base + Air-Base).

Thanks! Oh base models? Interesting since I normally do only Instruct models - I can take a look though!

Re: DeepSeek-v3.1

#106

Post by exe34 »

danielhanchen wrote: post #80
lambda wrote: post #78

Earlier quoted context omitted.

[flagged]

I added it since many people who used Unsloth don't know how to compile llama.cpp, so the only way from Python's side is to either (1) Install it via apt-get within the Python shell (2) Error out then tell the user to install it first, then continue again I chose (1) since it was mainly for ease of use for the user - but I agree it's not a good idea sorry! :( I also added a section to manually compile llama.cpp here:…

have you considered cosmopolitan? e.g. like llamafile that works on everything up to and including toasters.

Re: DeepSeek-v3.1

#107

Post by danielhanchen »

woile wrote: post #102
danielhanchen wrote: post #99

Earlier quoted context omitted.

Yep agreed - I primarily thought it was a reasonable "hack", but it's pretty bad security wise, so apologies again. The current solution hopefully is in between - ie sudo is gone, apt-get will run only after the user agrees by pressing enter, and if it fails, it'll tell the user to read docs on installing llama.cpp

Don't apologize, you are doing amazing work. I appreciate the effort you put. Usually you don't make assumptions on the host OS, just try to find the things you need and if not, fail, ideally with good feedback. If you want to provide the "hack", you can still do it, but ideally behind a flag, `allow_installation` or something like that. This is, if you want your code to reach broader audiences.

Thank you! :)

Re: DeepSeek-v3.1

#108

Post by okasaki »

rsanek wrote: post #86

Looks to be the ~same intelligence as gpt-oss-120B, but about 10x slower and 3x more expensive? https://artificialanalysis.ai/models/deepseek-v3-1-reasoning

My experience is that gpt-oss doesn't know much about obscure topics, so if you're using it for anything except puzzles or coding in popular languages, it won't do well as the bigger models.

It's knowledge seems to be lacking even compared to gpt3.

No idea how you'd benchmark this though.

Re: DeepSeek-v3.1

#109

Post by danielhanchen »

exe34 wrote: post #106
danielhanchen wrote: post #80

Earlier quoted context omitted.

I added it since many people who used Unsloth don't know how to compile llama.cpp, so the only way from Python's side is to either (1) Install it via apt-get within the Python shell (2) Error out then tell the user to install it first, then continue again I chose (1) since it was mainly for ease of use for the user - but I agree it's not a good idea sorry! :( I also added a section to manually compile llama.cpp here:…

have you considered cosmopolitan? e.g. like llamafile that works on everything up to and including toasters.

Oh llamafile is very cool! I might add it as an option actually :) For generic exports (ie to vLLM, llamafile etc), normally finetunes end with model.save_pretrained_merged and that auto merges to 16bit safetensors which allows for further processing downstream - but I'll investigate llamafile more! (good timing since llamafile is cross platform!)

Re: DeepSeek-v3.1

#110

Post by Imustaskforhelp »

danielhanchen wrote: post #97
Imustaskforhelp wrote: post #93

Earlier quoted context omitted.

Hey man, I was seeing your comments and you do seem to respond to each and everyone nicely regarding this sudo shenanigan. I think that you have removed sudo so this is nice, my suggestion is pretty similar to that of pxc (basically determine different distros and use them as that) I wonder if we will ever get a working universal package manager in linux, to me flatpak genuinely makes the most sense even sometimes fo…

Hey :) I love suggestions and keep them coming! :) I agree on handling different distros - sadly I'm not familiar with others, so any help would be appreciated! For now I'm most familiar with apt-get, but would 100% want to expand out! Interesting will check flatpak out!

Just to let you know though that its really rare that flatpak is used for cli's. I think I mentioned it in my comment too or if not, my apologies but flatpak is used mostly in gui's.

I doubt its efficacy here, they might be more useful if you provide a whole jupyter / browser gui though but a lot o f us run it just in cli so I doubt flatpak.

I didn't mean to say that flatpak was the right tool for this job, I seriously don't know too much to comment and so I'd prefer if you could ask someone definitely experienced regarding it.

My reasoning for flatpak was chunking support (that I think is rare in appimage) and easier gpu integration (I think) compared to docker, though my reasoning might be flawed since flatpak isn't mostly used with cli.

Post reply on HN