Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DeepSeek-v3.1

api-docs.deepseek.com

Post reply on HN
Print view View original

161–170 of 273 posts

Re: DeepSeek-v3.1

#161

Post by pshirshov »

danielhanchen wrote: post #151
pshirshov wrote: post #144

Earlier quoted context omitted.

My current solution is to pack llama.cpp as a custom nix formula (the one in nixpkgs has the conversion script broken) and run it myself. I wasn't able to run unsloth on ROCM nor for inference nor for conversion, sticking with peft for now but I'll attempt again to re-package it.

Oh interesting oh for ROCM there are some installation instructions here: https://rocm.docs.amd.com/projects/ai-developer-hub/en/lates... I'm working with the AMD folks to make the process easier, but it looks like first I have to move off from pyproject.toml to setup.py (allows building binaries)

Yes, it's trivial with the pre-built vllm docker, but I need a declarative way to configure my environment. The lack of prebuilt rocm wheels for vllm is the main hindrance for now but I was shocked to see the sudo apt-get in your code. Ideally, llama.cpp should publish their gguf python library and the conversion script to pypi with every release, so you can just add that stuff as a dependency. vllm should start publishing a rocm wheel, after that unsloth would need to start publishing two versions - a cuda one and a rocm one.

Re: DeepSeek-v3.1

#162

Post by lyu07282 »

elteto wrote: post #75
danielhanchen wrote: post #68

Earlier quoted context omitted.

Oh hey I'm assuming this is for conversion to GGUF after a finetune? If you need to quantize to GGUF Q4_K_M, we have to compile llama.cpp, hence apt-get and compiling llama.cpp within a Python shell. There is a way to convert to Q8_0, BF16, F16 without compiling llama.cpp, and it's enabled if you use `FastModel` and not on `FastLanguageModel` Essentially I try to do `sudo apt-get` if it fails then `apt-get` and if al…

Dude, this is NEVER ok. What in the world??? A third party LIBRARY running sudo commands? That’s just insane. You just fail and print a nice error message telling the user exactly what they need to do, including the exact apt command or whatever that they need to run.

on a meta level its kind of worrying for the ecosystem that there is nothing in PyPI that blocks & bans developers who try to run sudo on setup. I get they don't have the resources to do manual checks, but literally no checks against malicious packages?

Re: DeepSeek-v3.1

#163

Post by segmondy »

danielhanchen wrote: post #146
mkl wrote: post #142

Earlier quoted context omitted.

Yes, since you won't actually need the permissions.

Hmmm I'm worried people will really not get on how to install / compile / use the terminal hmmm hence I thought permissions were like a compromise solution

Don't listen to this crowd, these are "technical folks". Most of your audience will fail to figure it out. You can provide an option that llama.cpp is missing and give them an option where you auto install it or they can install it themselves and do manual configuration. I personally won't tho.

Re: DeepSeek-v3.1

#164

Post by ilaksh »

seunosewa wrote: post #24
ivape wrote: post #18

Earlier quoted context omitted.

What formats? I thought the very schema of json is what allows these LLMs to enforce structured outputs at the decoder level? I guess you can do it with any format, but why stray from json?

Sometimes it will randomly generate something like this in the body of the text: ``` executeshell command echo "" >> novels/AI_Voodoo_Romance/chapter-1-a-new-dawn.txt ``` or this: ``` executeshell {"command": "pwd && ls -la"} ``` Prompting it to use the right format doesn't seem to work. Claude, Gemini, GPT5, and GLM 4.5, don't do that. To accomodate DeepSeek, the tiny agent that I'm building will have to support all…

Maybe you have your temperature turned up too high.

Re: DeepSeek-v3.1

#165

Post by okasaki »

scotty79 wrote: post #158
easygenes wrote: post #112

Earlier quoted context omitted.

Something I was doing informally that seems very effective is asking for details about smaller cities and towns and lesser points of interest around the world. Bigger models tend to have a much better understanding and knowledge base for the more obscure places.

I would really love if they figured out how to train a model that doesn't have any such knowledge baked it, but knows where to look for it. Maybe even has a clever database for that. Knowing this kind of trivia like this consistently of the top of your head is a sign of deranged mind, artificial or not.

Would that work as well? If I ask a big model to write like Shakespeare it just knows intuitively how to do that. If it didn't and had to look up how to do that, I'm not sure it would do a good job.

Re: DeepSeek-v3.1

#168

Post by indigodaddy »

petesergeant wrote: post #129
easygenes wrote: post #114

Earlier quoted context omitted.

Other benchmark aggregates are less favorable to GPT-OSS-120B: https://arxiv.org/abs/2508.12461

With all these things, it depends on your own eval suite. gpt-oss-120b works as well as o4-mini over my evals, which means I can run it via OpenRouter on Cerebras where it's SO DAMN FAST and like 1/5th the price of o4-mini.

How would you compare gpt-oss-120b to (for coding):

Qwen3-Coder-480B-A35B-Instruct

GLM4.5 Air

Kimi K2

DeepSeek V3 0324 / R1 0528

GPT-5 Mini

Thanks for any feedback!

Re: DeepSeek-v3.1

#169

Post by 7thpower »

refulgentis wrote: post #61
dragonwriter wrote: post #47

Earlier quoted context omitted.

> In the modes in APIs, the sampling code essentially "rejects and reinference" any token sampled that wouldn't create valid JSON under a grammar created from the schema. I thought the APIs in use generally interface with backend systems supporting logit manipulation, so there is no need to reject and reinference anything; its guaranteed right the first time because any token that would be invalid has a 0% chance of…

I maintain a cross-platform llama.cpp client - you're right to point out that generally we expect nuking logits can take care of it. There is a substantial performance cost to nuking, the open source internals discussion may have glossed over that for clarity (see github.com/llama.cpp/... below). The cost is very high, default in API* is not artificially lower other logits, and only do that if the first inference att…

This is a basic question but maybe you can help: what is a good resource to use to understand how to take advantage of logits?

Re: DeepSeek-v3.1

#170

Post by dmos62 »

irthomasthomas wrote: post #157
dmos62 wrote: post #145

Earlier quoted context omitted.

How do you propose that would work? A pipeline that goes through query-response pairs to deduce response quality and then uses the low-quality responses for further training? Wouldn't you need a model that's already smart enough to tell that previous model's responses weren't smart enough? Sounds like a chicken and egg problem.

It just means that once you send your test questions to a model API, that company now has your test. So 'private' benchmarks take it on faith that the companies won't look at those requests and tune their models or prompts to beat them.

Sounds a bit presumptious to me. Sure, they have your needle, but they also need a cost-efficient way to find it in their hay stack.
Post reply on HN