Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DeepSeek-v3.1

api-docs.deepseek.com

Post reply on HN
Print view View original

241–250 of 273 posts

Re: DeepSeek-v3.1

#241

Post by azinman2 »

danielhanchen wrote: post #39

For local runs, I made some GGUFs! You need around RAM + VRAM >= 250GB for good perf for dynamic 2bit (2bit MoE, 6-8bit rest) - can also do SSD offloading but it'll be slow. ./llama.cpp/llama-cli -hf unsloth/DeepSeek-V3.1-GGUF:UD-Q2_K_XL -ngl 99 --jinja -ot ".ffn_.*_exps.=CPU" More details on running + optimal params here: https://docs.unsloth.ai/basics/deepseek-v3.1

It’d also be great if you guys could do a fine tune to run on an 8x80G A/H100. These H200/B200 configs are harder to come by (and much more expensive).

Re: DeepSeek-v3.1

#242

Post by danielhanchen »

diggan wrote: post #183
danielhanchen wrote: post #39

For local runs, I made some GGUFs! You need around RAM + VRAM >= 250GB for good perf for dynamic 2bit (2bit MoE, 6-8bit rest) - can also do SSD offloading but it'll be slow. ./llama.cpp/llama-cli -hf unsloth/DeepSeek-V3.1-GGUF:UD-Q2_K_XL -ngl 99 --jinja -ot ".ffn_.*_exps.=CPU" More details on running + optimal params here: https://docs.unsloth.ai/basics/deepseek-v3.1

> More details on running + optimal params here: https://docs.unsloth.ai/basics/deepseek-v3.1 Was that document almost exclusively written with LLMs? I looked at it last night (~8 hours ago) and it was riddled with mistakes, most egregious was that the "Run with Ollama" section had instructions for how to install Ollama, but then the shell commands were actually running llama.cpp, a mistake probably no human would ma…

Oh hey sorry the docs are still in construction! Are you referring to merging GGUFs to Ollama - it should work fine! Ie:

``` ./llama.cpp/llama-gguf-split --merge \ DeepSeek-V3.1-GGUF/DeepSeek-V3.1-UD-Q2_K_XL/DeepSeek-V3.1-UD-Q2_K_XL-00001-of-00006.gguf \ merged_file.gguf ```

Ollama can only allow merged GGUFs (not splitted ones), so hence the command.

All docs are made by humans (primarily my brother and me), just sometimes there might be some typos (sorry in advance)

I'm also uploading Ollama compatible versions directly so ollama run can work (it'll take a few more hours)

Re: DeepSeek-v3.1

#243

Post by bgilroy26 »

Leynos wrote: post #238
bgilroy26 wrote: post #224

Earlier quoted context omitted.

I should have said, I am looking for posted chat logs where the prompts are shared as well. I really enjoy the process of making stories with AI and I am curious to see how others do the same thing.

Look at NovelCrafter. They have a great Discord community where people share their prompts and workflows. There is also the WritingWithAI subreddit.

Thank you!

Re: DeepSeek-v3.1

#244

Post by danielhanchen »

azinman2 wrote: post #241
danielhanchen wrote: post #39

For local runs, I made some GGUFs! You need around RAM + VRAM >= 250GB for good perf for dynamic 2bit (2bit MoE, 6-8bit rest) - can also do SSD offloading but it'll be slow. ./llama.cpp/llama-cli -hf unsloth/DeepSeek-V3.1-GGUF:UD-Q2_K_XL -ngl 99 --jinja -ot ".ffn_.*_exps.=CPU" More details on running + optimal params here: https://docs.unsloth.ai/basics/deepseek-v3.1

It’d also be great if you guys could do a fine tune to run on an 8x80G A/H100. These H200/B200 configs are harder to come by (and much more expensive).

Unsloth should work on any GPU setup all the way until the old Tesla T4s and the newer B200s :) We're working on a faster and better multi GPU version, but using accelerate / torchrun manually + Unsloth should work out of the box!

Re: DeepSeek-v3.1

#245

Post by danielhanchen »

solarkraft wrote: post #189
mFixman wrote: post #113

Earlier quoted context omitted.

Maybe it's a personal preference, but I don't want external programs to ever touch my package manager, even with permission. Besides, this will fail loudly for systems that don't use `apt-get`. I would just ask the user to install the package, and _maybe_ show the command line to install it (but never run it).

I like it when software does work for me. Quietly installing stuff at runtime is shady for sure, but why not if I consent?

Do you think it's ok for permissioning I guess? I might also add a 30 second timer and just bail out if there's no response from the user

Re: DeepSeek-v3.1

#246

Post by azinman2 »

danielhanchen wrote: post #244
azinman2 wrote: post #241

Earlier quoted context omitted.

It’d also be great if you guys could do a fine tune to run on an 8x80G A/H100. These H200/B200 configs are harder to come by (and much more expensive).

Unsloth should work on any GPU setup all the way until the old Tesla T4s and the newer B200s :) We're working on a faster and better multi GPU version, but using accelerate / torchrun manually + Unsloth should work out of the box!

I guess I was hoping for you guys to put up these weights. I think they’d be popular for these very large models.

You guys already do a lot for the local LLM community and I appreciate it.

Re: DeepSeek-v3.1

#247

Post by danielhanchen »

solarkraft wrote: post #192
danielhanchen wrote: post #146

Earlier quoted context omitted.

Hmmm I'm worried people will really not get on how to install / compile / use the terminal hmmm hence I thought permissions were like a compromise solution

I think that it is, quite a good one, even: - Determine the command that has to be run by the algorithm above. This does most of the work a user would have to figure out what has to be installed on their system. - Ask whether to run the command automatically. This allows the “software should never install dependencies by itself” crowd to say no and figure out further steps, while allowing people who just want it to w…

Oh ok! I would say 50% of people manually install llama.cpp and the other 50% want it to be automated

Re: DeepSeek-v3.1

#248

Post by danielhanchen »

Balinares wrote: post #220
danielhanchen wrote: post #175

Earlier quoted context omitted.

Oh yes so before we install llama.cpp we do an path environment check and if its not defined then it'll install. But yes agreed there won't be any more random package installs sorry!

Thanks for the reply! If I can find the time (that's a pretty big if), I'll try to send a PR to help with the packaging.

No worries :)

Re: DeepSeek-v3.1

#249

Post by danielhanchen »

rpdillon wrote: post #195
danielhanchen wrote: post #178

Earlier quoted context omitted.

Thanks - hopefully the compromise solution ie python input asking for user permissions works ok?

As a guy that would naturally be in the camp of "installing packages is never okay", I also live in the more practical world where people want things to work. I think the compromise you're suggesting is a pretty good one. I think the highest quality implementation here would be. Try to find prebuilt and download. See if you can compile from source if a compiler is installed. If no compiler: prompt to install via sudo…

I'm going to see if I can make prebuilt versions work :) But thanks!

Re: DeepSeek-v3.1

#250

Post by danielhanchen »

segmondy wrote: post #181
tw1984 wrote: post #72

Earlier quoted context omitted.

for such dynamic 2bit, is there any benchmark results showing how many performance I would give up compared to the original model? thanks.

if you are running a 2bit quant, you are not giving up performance but gaining 100% performance since the alternative is usually 0%. Smaller quants are for folks who won't be able to run anything at all, so you run the largest you can run relative to your hardware. I for instance often ran Q3_K_L, I don't think of how much performance I'm giving up, but rather how without Q3, I won't be able to run it at all. With th…

Oh Q3_K_L as in upcasted embed_tokens + lm_head to Q8_0? I normally do Q4 embed Q6 lm_head - would a Q8_0 be interesting?
Post reply on HN