Earlier quoted context omitted.
Single request depth on vllm with dspark, I'm getting ~200 tps, I'd say it's pretty snappy.
I get like 80.
What's your setup? Happy to try to point you in the direction that worked for me.
DeepSeek V4 Pro 0813
311–320 of 493 posts
Re: DeepSeek V4 Pro 0813
#312Earlier quoted context omitted.
50% cache hit is really low - in a standard agentic loop you should expect like 99%+ cache hit percentage (which should also lower that $12.50 to like a couple of $ for the same amount of tokens). If you're using a customised harness you should make sure you don't have something that's e.g. changing your system prompt on some requests or rewriting history - it can be tempting to do stuff like strip old thinking token…
In my experience, that's the OpenRouter tax. Even a session that does everything right to remain sticky ends up getting moved between providers on a few requests, which bills you the full context as input every time the switch happens. I assume it's done as load balancing/latency mitigation, but it's put me off of OpenRouter for my use cases (limited use, limited need for changing models).
This has not been my experience. Generally I do pin to 1 provider, or 1 provider with a couple fallbacks (especially with deepseek - most providers are 10x the cached token price compared to deepseek themselves), but even when I don't I still usually see 99%+ cache hit percentage. Specifically using pi with various ad-hoc customisations (that I was careful not to break prompt caching with).
Re: DeepSeek V4 Pro 0813
#313Why does this link to OpenRouter, which has no useful information on its own? Linking to the official API or the benchmarks would make more sense: - https://api-docs.deepseek.com/ - https://x.com/ChrisGPT/status/2087572834650407024/photo/1 (officially posted on WeChat, this is just one of many reposts)
Moreover, OpenRouter is NOT Open Source, fair source, source available, etc. It's a proprietary cloud service that got first place in the API aggregation distribution game. Link to DeepSeek!
TrustedRouter is hosted and full opensource!
Re: DeepSeek V4 Pro 0813
#314Earlier quoted context omitted.
Moreover, OpenRouter is NOT Open Source, fair source, source available, etc. It's a proprietary cloud service that got first place in the API aggregation distribution game. Link to DeepSeek!
Open doesn't always refer to the code. Just like their previous project, it refers to an open marketplace where anybody can sign up to sell access to models. But it'd still be nice to post to wait an extra minute to find some other page/new url from deepseek for it instead of posting that it exists somewhere.
it's not open, I know people rejected by them (then they went to my site to be listed Trustedrouter.com)
Re: DeepSeek V4 Pro 0813
#315You mean to tell me it's been 10 hours and there's no unsloth quant? I've been running 0731 and love it.
Re: DeepSeek V4 Pro 0813
#316Earlier quoted context omitted.
I built TrustedRouter so this can fail closed. min_privacy=zdr rejects the request when the model has no ZDR provider; confidential requires provider-side confidential compute. https://trustedrouter.com/blog/how-confidential-computing-pr...
I don't understand how this works? Is it another proxy on top? What stops the provider from reading/storing the prompts at the LLM execution level?
it's confidential compute, it's open source and you can verify yourself that it's not reading the prompts
Re: DeepSeek V4 Pro 0813
#317Earlier quoted context omitted.
I've always wondered if I was using containers wrong because none of them I've ever had to create were complicated. Maybe it's because I choose tools that make local development easy (Go + sqlite + various CLTs) or maybe it's because I never hard to interact with this on the professional side outside of making images for our projects (which still weren't complicated for the reasons above). LLMs make containers in a p…
This was the repo: https://github.com/amalshaji/portr And this was my gh issue: https://github.com/amalshaji/portr/issues/308 And below was my prompt: """ give me single docker-compose file that i can run on my server to run current project... you can read README.md , and then, this relevant page: https://docs-custom-reverse-proxy.portr-docs.pages.dev/docs/... ... this was the result of me raising github issue: https…
I've been using DS4F+Pi with great results, but I think one thing that helps is at the end of my prompt I'll tell it how to verify it, e.g. "Make sure the compose file works by running it locally (use self-signed certs if required)".
The argument could be made that "the model should be smart enough to figure it out" , and maybe DS4 isn't. But with just a bit of steering you can get the correct result for like 1/10th the cost, or even cheaper.
Re: DeepSeek V4 Pro 0813
#318Re: DeepSeek V4 Pro 0813
#319Have been letting it spin pretty hard (~$12.50 for 2B, 50% cache hits) on my traffic simulator/distributed physics engine all day, it's found some pretty significant gains without introducing any new problems. I'm happy
Can you explain how you used 12 billion tokens to do useful work?
Re: DeepSeek V4 Pro 0813
#320Have been letting it spin pretty hard (~$12.50 for 2B, 50% cache hits) on my traffic simulator/distributed physics engine all day, it's found some pretty significant gains without introducing any new problems. I'm happy
Can you explain how you used 12 billion tokens to do useful work?
(Not the original commenter.)
You can rack up quite a lot of tokens if you ask it to try out a lot of things, eg for performance investigations and trying out optimisation ideas.