261–270 of 273 posts
Earlier quoted context omitted.
You do not need qwen-code or qwen.ai to use them; openrouter + opencode suffice.
Right, I'm aware, was just wondering about that specific scenario.
Earlier quoted context omitted.
Reminder that OpenAI is an American company whose headstart is attributed to stealing copyrighted material from everyone else. Without the huge theft, they'd be nowhere.
Last I checked, as it concerns the training of their models, all legal challenges are pending. No theft has yet been proven, as they used publicly available data.
Earlier quoted context omitted.
Don't listen to this crowd, these are "technical folks". Most of your audience will fail to figure it out. You can provide an option that llama.cpp is missing and give them an option where you auto install it or they can install it themselves and do manual configuration. I personally won't tho.
Who do you think the audience is here if not technical. We are in a discussion about a model that requires over 250gb of ram to run. I don't know a non-technical person with more than 32gb.
Earlier quoted context omitted.
I'm not aware of any proven IP theft by American companies in the AI space. Many pending legal challenges. None yet proven.
Alright. So there’s proven IP theft by the Chinese companies with completed legal proceedings?
For local runs, I made some GGUFs! You need around RAM + VRAM >= 250GB for good perf for dynamic 2bit (2bit MoE, 6-8bit rest) - can also do SSD offloading but it'll be slow. ./llama.cpp/llama-cli -hf unsloth/DeepSeek-V3.1-GGUF:UD-Q2_K_XL -ngl 99 --jinja -ot ".ffn_.*_exps.=CPU" More details on running + optimal params here: https://docs.unsloth.ai/basics/deepseek-v3.1
> More details on running + optimal params here: https://docs.unsloth.ai/basics/deepseek-v3.1 Was that document almost exclusively written with LLMs? I looked at it last night (~8 hours ago) and it was riddled with mistakes, most egregious was that the "Run with Ollama" section had instructions for how to install Ollama, but then the shell commands were actually running llama.cpp, a mistake probably no human would ma…
But in the docs I see things like
cp llama.cpp/build/bin/llama-* llama.cpp
Earlier quoted context omitted.
This is a basic question but maybe you can help: what is a good resource to use to understand how to take advantage of logits?
https://dottxt-ai.github.io/outlines/latest/ For OpenAI, you can just pass in the json_schema to activate it, no library needed. For direct LLM interfacing you will need to host your own LLM or use a cloud provider that allows you too hook in, but someone else may need to correct me on this. If anyone is using anything other than Outlines, please let us know.
Earlier quoted context omitted.
> More details on running + optimal params here: https://docs.unsloth.ai/basics/deepseek-v3.1 Was that document almost exclusively written with LLMs? I looked at it last night (~8 hours ago) and it was riddled with mistakes, most egregious was that the "Run with Ollama" section had instructions for how to install Ollama, but then the shell commands were actually running llama.cpp, a mistake probably no human would ma…
> but then the shell commands were actually running llama.cpp, a mistake probably no human would make. But in the docs I see things like cp llama.cpp/build/bin/llama-* llama.cpp Wouldn't this explain that? (Didn't look too deep)
```
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggerganov/llama.cpp cmake llama.cpp -B llama.cpp/build \ -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-quantize llama-cli llama-gguf-split llama-mtmd-cli llama-server
cp llama.cpp/build/bin/llama-* llama.cpp
```
but then Ollama is above it:
```
./llama.cpp/llama-gguf-split --merge \ DeepSeek-V3.1-GGUF/DeepSeek-V3.1-UD-Q2_K_XL/DeepSeek-V3.1-UD-Q2_K_XL-00001-of-00006.gguf \ merged_file.gguf
```
I'll edit the area to say you first have to install llama.cpp
Earlier quoted context omitted.
Oh ok sorry maybe I misunderstood sorry! I actually found my partial work I did for precompiled binaries! https://huggingface.co/datasets/unsloth/precompiled_llama_cp... I was trying to see if I could pre-compile some llama.cpp binaries then save them as a zip file (I'm a noob sorry) - but I definitely need to investigate further on how to do python pip binaries
https://docs.astral.sh/uv/guides/package/#publishing-your-pa...
But I think similarly for uv we need a setup.py for packaging binaries (more complex)
Earlier quoted context omitted.
Unsloth should work on any GPU setup all the way until the old Tesla T4s and the newer B200s :) We're working on a faster and better multi GPU version, but using accelerate / torchrun manually + Unsloth should work out of the box!
I guess I was hoping for you guys to put up these weights. I think they’d be popular for these very large models. You guys already do a lot for the local LLM community and I appreciate it.