101–110 of 172 posts
LLMs is quickly became commodities. US AI labs like OpenAI is losing their moat. Hyperscalers and data center owners who can sell cheap token will win
I believe Codex harness is pretty sticky though I haven’t tried many others. Does anyone else provide a harness of that quality?
i've never had any issues with 256k context. see no reason to bump up to 1m if it comes at a premium.
This seems functionally similar to OpenAI having a step in pricing once you exceed a certain context length (also at 272k aka 2^18 aka 256k). Having a lot of active context increases the per-token cost (flops issued and bytes read per token out) so it makes sense to pass that cost on to users. I'm actually surprised it's implemented as a hard cutoff instead of a smooth gradient.
Not surprising it's a hard cutoff: they almost certainly have two infrastructure configurations for the two max sequence lengths Fewer nodes dedicated to prefill per instance, and fewer nodes in total since you don't need to support a higher KV cache. Disaggregated inference also means they can tune the balance of compute dedicated to prefill seperately from decode
One would think compute-constrained actors like Anthropic would have done so, unless prefill isn’t really a bottleneck compared to decode?
This seems functionally similar to OpenAI having a step in pricing once you exceed a certain context length (also at 272k aka 2^18 aka 256k). Having a lot of active context increases the per-token cost (flops issued and bytes read per token out) so it makes sense to pass that cost on to users. I'm actually surprised it's implemented as a hard cutoff instead of a smooth gradient.
Earlier quoted context omitted.
The only downside with third party providers is that you have to trust that the provider have setup and configured it correctly, and is not secretly quantizing it. See Kimi Vendor Verifier.
Plenty of providers on that list aren't fly-by-nights and have easily proven themselves in past models. Jeremy Howard does demos on Fireworks. A nice thing about open router is you can specify filters, like "US hosting without data retention"
Codex uses 256k masterfully, 1M is luxurious but still quite expensive and seems not necessary as a default.
This.
When working on hard problems (not "vibecode me a script to show an alert box", but e.g. "let's see what this three-level LUT-state-machine obfuscated binary does"), hitting the 1M (!) context window with Claude Code feels like you were talking to Claude Claudewski when his shift just abruptly ends, he packs his things, throws the office keys at Claude Claudeson in-between the front door frame while handovering like "Hi! Nice to see you, good luck." and now here we go again, you are working with someone who just experienced an acute amnesia. It tries everything it already tried, everything it was told in the initial prompt to not do, everything it was told in follow-up prompts not to do. "You were right, this approach does not work and we don't have 20 TB RAM on this machine for full symbolic execution, let me try..."
In Codex, it's so seamless that I sometimes just notice "wait, the context was 20 % remaining, it is 70 % now, wow, when did this happen", while it seamlessly works on the task. Basically never had an issue with context on Codex, be it coding features, cracking hard crack-me ciphers, or researching basically anything.
(For full disclosure, my last experience with Claude was a few weeks ago when I cancelled the subscription, maybe they fully reworked the traumatic "Summarizing" - "Oh, hi! Where are we? Who am I? What we are doing? This is taking too long, let me take a shortcut..." lobotomy they were doing in the meantime.)
256k is enough when the harness uses it properly and the model is not stupid. And also when the tokenizer is not tuned to invoice as many tokens as possible...
Codex uses 256k masterfully, 1M is luxurious but still quite expensive and seems not necessary as a default.
I'll have to give it a shot some time soon. I just can't imagine working on any of my serious projects doing a new large feature with that little room. By the time it has looked at half the code required to start planning the work, it'd be out of context. Must use sub agents better or something.
https://platform.claude.com/docs/en/about-claude/models/what...
Since Claude is the first time for me really, really out (TIL against my wished about https://status.claude.com/ ), I am now interested enough to see what else works. But ... when I click pricing, I see "Join a waitlist". Wtf? Are they really that good, so were totally surprised and overwhelmed by the requests, is this a marketing stunt, or do they just don't have the hardware being in china?
Since the model is open-weights, you can get from other providers, for example see https://openrouter.ai/moonshotai/kimi-k3#providers
Earlier quoted context omitted.
This exact same comment is the top comment on the reddit thread for this news https://www.reddit.com/r/kimi/s/BFa1TR9vNg
I searched one other comment of theirs in their history and see the same thing. On mobile so I can’t easily tell if this account is the bot account or if the ones on Reddit are. HN: https://news.ycombinator.com/item?id=47934437 Reddit: https://www.reddit.com/r/worldnews/comments/1sxzzop/comment/...