Sad to see the off peak discount go. I was able to crank tokens like crazy and not have it cost anything. That said the pricing is still very very good so I can't complain too much.
DeepSeek-v3.1
71–80 of 273 posts
Re: DeepSeek-v3.1
#72For local runs, I made some GGUFs! You need around RAM + VRAM >= 250GB for good perf for dynamic 2bit (2bit MoE, 6-8bit rest) - can also do SSD offloading but it'll be slow. ./llama.cpp/llama-cli -hf unsloth/DeepSeek-V3.1-GGUF:UD-Q2_K_XL -ngl 99 --jinja -ot ".ffn_.*_exps.=CPU" More details on running + optimal params here: https://docs.unsloth.ai/basics/deepseek-v3.1
for such dynamic 2bit, is there any benchmark results showing how many performance I would give up compared to the original model? thanks.
Re: DeepSeek-v3.1
#73Earlier quoted context omitted.
[flagged]
Incredible how "keeping their people down" means leaps in personal wealth and happiness for huge swathes of the population and internal criticism is that it is a "poverty reduction machine" that is too focused.
They're not mutually exclusive. Lots of terrible and mismanaged governments rely on genius short-term economic exploitation, like Syria, Iran, India, Korea, etc.
What would be incredible is China sticking the landing to a third-sector economy. Plenty of countries have industrialized over the past century, only a handful became true service economies.
Re: DeepSeek-v3.1
#74Earlier quoted context omitted.
Any example or prompt you use to make this statment?
I remember asking for quotes about the Spanish conquest of South America because I couldn't remember who said a specific thing. The GPT model started hallucinating quotes on the topic, while DeepSeek responded with, "I don't know a quote about that specific topic, but you might mean this other thing." or something like that then cited a real quote in the same topic, after acknowledging that it wasn't able to find the…
Was that true for GPT-5? They claim it is much better at not hallucinating
Re: DeepSeek-v3.1
#75Earlier quoted context omitted.
By the way, I'm wondering why unsloth (a goddamn python library) tries to run apt-get with sudo (and fails on my nixos). Like how tf we are supposed to use that?
Oh hey I'm assuming this is for conversion to GGUF after a finetune? If you need to quantize to GGUF Q4_K_M, we have to compile llama.cpp, hence apt-get and compiling llama.cpp within a Python shell. There is a way to convert to Q8_0, BF16, F16 without compiling llama.cpp, and it's enabled if you use `FastModel` and not on `FastLanguageModel` Essentially I try to do `sudo apt-get` if it fails then `apt-get` and if al…
Dude, this is NEVER ok. What in the world??? A third party LIBRARY running sudo commands? That’s just insane.
You just fail and print a nice error message telling the user exactly what they need to do, including the exact apt command or whatever that they need to run.
Re: DeepSeek-v3.1
#76Earlier quoted context omitted.
There is one: https://pricepertoken.com/
Claude's Opus pricing is nuts. I'd be surprised if anyone uses it without the top max subscription.
Sure I do, but not as part of any tools, just for one-off conversations where I know it's going to be the best out there. For tasks where reasoning helps little to none, it's often still number one.
Re: DeepSeek-v3.1
#77Earlier quoted context omitted.
Oh hey I'm assuming this is for conversion to GGUF after a finetune? If you need to quantize to GGUF Q4_K_M, we have to compile llama.cpp, hence apt-get and compiling llama.cpp within a Python shell. There is a way to convert to Q8_0, BF16, F16 without compiling llama.cpp, and it's enabled if you use `FastModel` and not on `FastLanguageModel` Essentially I try to do `sudo apt-get` if it fails then `apt-get` and if al…
Dude, this is NEVER ok. What in the world??? A third party LIBRARY running sudo commands? That’s just insane. You just fail and print a nice error message telling the user exactly what they need to do, including the exact apt command or whatever that they need to run.
How unusual is this for the ecosystem?
Re: DeepSeek-v3.1
#78Earlier quoted context omitted.
By the way, I'm wondering why unsloth (a goddamn python library) tries to run apt-get with sudo (and fails on my nixos). Like how tf we are supposed to use that?
Oh hey I'm assuming this is for conversion to GGUF after a finetune? If you need to quantize to GGUF Q4_K_M, we have to compile llama.cpp, hence apt-get and compiling llama.cpp within a Python shell. There is a way to convert to Q8_0, BF16, F16 without compiling llama.cpp, and it's enabled if you use `FastModel` and not on `FastLanguageModel` Essentially I try to do `sudo apt-get` if it fails then `apt-get` and if al…
[flagged]
Re: DeepSeek-v3.1
#79Earlier quoted context omitted.
Oh hey I'm assuming this is for conversion to GGUF after a finetune? If you need to quantize to GGUF Q4_K_M, we have to compile llama.cpp, hence apt-get and compiling llama.cpp within a Python shell. There is a way to convert to Q8_0, BF16, F16 without compiling llama.cpp, and it's enabled if you use `FastModel` and not on `FastLanguageModel` Essentially I try to do `sudo apt-get` if it fails then `apt-get` and if al…
[flagged]
> What this whole thing hallucinated by an LLM?
probably not, because LLMs are a little more competent than this
Re: DeepSeek-v3.1
#80Earlier quoted context omitted.
Oh hey I'm assuming this is for conversion to GGUF after a finetune? If you need to quantize to GGUF Q4_K_M, we have to compile llama.cpp, hence apt-get and compiling llama.cpp within a Python shell. There is a way to convert to Q8_0, BF16, F16 without compiling llama.cpp, and it's enabled if you use `FastModel` and not on `FastLanguageModel` Essentially I try to do `sudo apt-get` if it fails then `apt-get` and if al…
[flagged]
I added it since many people who used Unsloth don't know how to compile llama.cpp, so the only way from Python's side is to either (1) Install it via apt-get within the Python shell (2) Error out then tell the user to install it first, then continue again
I chose (1) since it was mainly for ease of use for the user - but I agree it's not a good idea sorry!
:( I also added a section to manually compile llama.cpp here: https://docs.unsloth.ai/basics/troubleshooting-and-faqs#how-...
But I agree I should remove apt-gets - will do this asap! Thanks for the suggestions :)