Looks to be the ~same intelligence as gpt-oss-120B, but about 10x slower and 3x more expensive? https://artificialanalysis.ai/models/deepseek-v3-1-reasoning
Clearly, this is a dark harbinger for Chinese AI supremacy /s
DeepSeek-v3.1
141–150 of 273 posts
Re: DeepSeek-v3.1
#142Earlier quoted context omitted.
It shouldn't install any packages itself. Just print out a message about the missing packages and your guess of the command to install them, then exit. That way users can run the command themselves if it's appropriate or add the packages to their container build or whatever. People set up machines in a lot of different ways, and automatically installing things is going to mess that up.
Hmmm so I should get rid of the asking / permissions message?
Yes, since you won't actually need the permissions.
Re: DeepSeek-v3.1
#143For local runs, I made some GGUFs! You need around RAM + VRAM >= 250GB for good perf for dynamic 2bit (2bit MoE, 6-8bit rest) - can also do SSD offloading but it'll be slow. ./llama.cpp/llama-cli -hf unsloth/DeepSeek-V3.1-GGUF:UD-Q2_K_XL -ngl 99 --jinja -ot ".ffn_.*_exps.=CPU" More details on running + optimal params here: https://docs.unsloth.ai/basics/deepseek-v3.1
>250GB, how do you guys run this stuff?
Re: DeepSeek-v3.1
#144Earlier quoted context omitted.
It won't work well if you deal with non ubuntu+cuda combination. Better just fail with a reasonable message.
For now I'm re-directly people to our docs https://docs.unsloth.ai/basics/troubleshooting-and-faqs#how-... But I'm working on more cross platform docs as well!
My current solution is to pack llama.cpp as a custom nix formula (the one in nixpkgs has the conversion script broken) and run it myself. I wasn't able to run unsloth on ROCM nor for inference nor for conversion, sticking with peft for now but I'll attempt again to re-package it.
Re: DeepSeek-v3.1
#145Earlier quoted context omitted.
Aren't good benchmarks supposed to be secret?
How can a benchmark be secret if you post it to an API to test a model on it? "We totally promise that when we run your benchmark against our API we won't take the data from it and use to be better at your benchmark next time" :P If you want to do it properly you have to avoid any 3rd party hosted model when you test your benchmark, which means you can't have GPT5, claude, etc. on it; and none of the benchmarks want…
How do you propose that would work? A pipeline that goes through query-response pairs to deduce response quality and then uses the low-quality responses for further training? Wouldn't you need a model that's already smart enough to tell that previous model's responses weren't smart enough? Sounds like a chicken and egg problem.
Re: DeepSeek-v3.1
#146Earlier quoted context omitted.
Hmmm so I should get rid of the asking / permissions message?
Yes, since you won't actually need the permissions.
Hmmm I'm worried people will really not get on how to install / compile / use the terminal hmmm hence I thought permissions were like a compromise solution
Re: DeepSeek-v3.1
#147Some of it is in Kagi already. Impressive from both DeepSeek and Kagi.
Is Kagi a Chinese-backed company?
Re: DeepSeek-v3.1
#148Earlier quoted context omitted.
By the way, I'm wondering why unsloth (a goddamn python library) tries to run apt-get with sudo (and fails on my nixos). Like how tf we are supposed to use that?
Oh hey I'm assuming this is for conversion to GGUF after a finetune? If you need to quantize to GGUF Q4_K_M, we have to compile llama.cpp, hence apt-get and compiling llama.cpp within a Python shell. There is a way to convert to Q8_0, BF16, F16 without compiling llama.cpp, and it's enabled if you use `FastModel` and not on `FastLanguageModel` Essentially I try to do `sudo apt-get` if it fails then `apt-get` and if al…
I'll venture that whoever is going to fine-tune their own models probably already has llama.cpp installed somewhere, or can install if required.
Please, please, never silently attempt to mutate the state of my machine, that is not a good practice at all and will break things more often than it will help because you don't know how the machine is set up in the first place.
Re: DeepSeek-v3.1
#149For local runs, I made some GGUFs! You need around RAM + VRAM >= 250GB for good perf for dynamic 2bit (2bit MoE, 6-8bit rest) - can also do SSD offloading but it'll be slow. ./llama.cpp/llama-cli -hf unsloth/DeepSeek-V3.1-GGUF:UD-Q2_K_XL -ngl 99 --jinja -ot ".ffn_.*_exps.=CPU" More details on running + optimal params here: https://docs.unsloth.ai/basics/deepseek-v3.1
>250GB, how do you guys run this stuff?
I'm working on sub 165GB ones!
165GB will need a 24GB GPU + 141GB of RAM for reasonably fast inference or a Mac
Re: DeepSeek-v3.1
#150Earlier quoted context omitted.
Don't apologize, you are doing amazing work. I appreciate the effort you put. Usually you don't make assumptions on the host OS, just try to find the things you need and if not, fail, ideally with good feedback. If you want to provide the "hack", you can still do it, but ideally behind a flag, `allow_installation` or something like that. This is, if you want your code to reach broader audiences.
Yep there's no need to apologize, you've been very courteous and took all that feedback constructively. Good stuff :)
Thank you!