271–280 of 493 posts
I've been using the last Deepseek Flash update for a week and I'm amazed. It was a capable model for easy tasks but now it looks like it can do some heavy development for peanuts. I can't wait to try this new one.
IME I can't trust it to write it's own plans from a spec, but if I give it a detailed execution plan written by Opus, it's fast and cheap (if chatty) in executing it.
Earlier quoted context omitted.
Why are people giving these n=1 comparisons like they mean anything? The worst offender is that pelican guy. These are non-deterministic systems and a single trial should not update your priors much at all. Of course it's significant that your response had a bug and took four times longer, but if you're only going to try once, this isn't real science, it's just vibes.
I don’t understand why people are calling these transformers models non-deterministic? Are you referring to the temperature parameter? I haven’t played with transformer internals in a while but my understanding is that if the temperature is fixed at a value where the top logit is always picked, then because they weights are fixed, the exact same input should produce the exact same output. Am I missing something?
Yes.
Your input is part of a batch, and you don't know where in the batch it is. By default batches are not invariant and VLLM only supports invariance at all on some Huwaei Ascend hardware.
See https://docs.vllm.ai/projects/ascend/en/latest/user_guide/fe...
Earlier quoted context omitted.
If that wasn't impressive enough, it's actually ~60x cheaper if you take into account the typical cache-read/input/output split in agentic coding, and the deep discount for cache reads offered by DeepSeek. Opencode has some public data on the typical split [1]: For DeepSeek V4 Pro the typical split is 750 in, 290 out, 82k cached. Cost per request for V4 Pro: $0.000875 per request. Equivalent Opus cost (w/o taking int…
Can we have a conversation about subscription plans for a minute? I don't mean to hype up the US AI firms, but if a ChatGPT $200/m subscription can get you $16,000 in effective API costs, doesn't effectively every model get destroyed by the subsidized Claude/ChatGPT models? Both in price and intelligence.
If I spent that every night it would be 3x my GPT subscription.
Earlier quoted context omitted.
psst.. they all do. Also, what kind of IP are you protecting, are you protecting some crazy discovery, nothing you're throwing at them is special, they aren't going to steal your CRUD pomodora app. If anything Deepseek is the only company I'd want to consent to training on my data, they're by far the most altruistic. Atleast they give back all their IP in the form of research and open source weights. It's not like th…
>are you protecting some crazy discovery Yes. If someone figured out my current project they would have a huge scoop.
Graphs without labels and/or scales on the axes are useless. I know less after viewing that page than before, but I got to see some pretty lines that I guess must mean something.
Earlier quoted context omitted.
you're thinking of one shot. if they are running an agentic loop then they don't need multiple passes. an agentic loop is multiple passes with tool calls and tools could fail and agent would correct from seeing the failure. a bad model will compound on error and fail, a good model will correct. 1 test is fine to gauge the quality of the model.
An agent doing a task even with multiple back to back calls like normal without an example is zero shot. An agent doing a task with 1 example is one shot. An agent doing a task with a few examples is few shot. I don't think you are correctly using these terms. The multiple back to back LLM calls are done on accumulating context, so if there is a sampling error it could throw the entire session out of whack, because L…
This is a different thing. Yes, giving multiple example is called "few-shot prompting".
But one-shot vs few-shot benchmarking is different. In this context "one-shot" means "pass at 1 effort" as opposed to "multi-shot". In the literature this is called "pass@k".
Anthropic has a good explanation here: https://www.anthropic.com/engineering/demystifying-evals-for... (search for "pass@k").
In this discussion we are discussing pass@1 (single shot) vs pass@(k>1) (multi shot).
> The multiple back to back LLM calls are done on accumulating context, so if there is a sampling error it could throw the entire session out of whack, because LLM's build on the previous context.
This isn't really true. In an agentic loop the LLM can correct itself via in-context learning.
What I care about is whether the model is capable of the tasks I give it at the lowest cost. Right now I'm using Kimi-K3/GLM-5.2/Minimax. Sonnet is great but I burn through the tokens too fast. Opus 5 set to max is amazing and more intelligent than all of us. .998 of the time I don't need that kind of intelligence. I just need the job done.
Have been letting it spin pretty hard (~$12.50 for 2B, 50% cache hits) on my traffic simulator/distributed physics engine all day, it's found some pretty significant gains without introducing any new problems. I'm happy
If you're using a customised harness you should make sure you don't have something that's e.g. changing your system prompt on some requests or rewriting history - it can be tempting to do stuff like strip old thinking tokens or compact tool call results to reduce context size but it's a trap - you want to never change history because of how cheap cache is, even more so with deepseek because their cache hit pricing is so low compared to most other models.
Nice bicycle chain, the little basket with a fish didn't show up in the right place: https://tools.simonwillison.net/markdown-svg-renderer#url=ht...