Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kimi K3-256k

kimi.com

Post reply on HN
Print view View original

141–150 of 172 posts

Re: Kimi K3-256k

#141

Post by KronisLV »

XenophileJKO wrote: post #69
smotched wrote: post #44

Earlier quoted context omitted.

Once you have the plan you don't need to keep the 2hrs of research in the context (which is most of it) you can drop that plan into a file and start fresh for implementation.

That is true, but I feel like sometimes if the conversation contains useful rational it can help to keep it. I think sometimes it is a judgement call, I will sometimes compress the context first. If I feel like the model and I explored a lot of options I won't want to keep context as it might be confusing. I think the more you use it the better judge you are of whether you should purge, compress, or just keep the con…

> That is true, but I feel like sometimes if the conversation contains useful rational it can help to keep it. I think sometimes it is a judgement call, I will sometimes compress the context first.

Yes, pretty much - if there’s a lot of noise and jumping around and wrong conclusions and corrections, compress and only leave the correct stuff (maybe make some plan file briefly mention what NOT to look at/do). But if it’s all fairly straightforward then can just proceed with the execution.

Most of the time the planning stage ends up short of 200k tokens anyways, it mostly takes hours just cause I’m slow and need to explore the various options - still cheaper than building the wholly wrong thing and having to redo everything.

Compressing the context can also drop important information so it might be better to only do that when you need to / use the plan mechanism/files / do it after completing some large stage of the plan and so on - so a judgement call.

Re: Kimi K3-256k

#142

Post by gpugreg »

k__ wrote: post #132
HawtAds wrote: post #5

This is just an API level change right? The model itself should be the same I think.

As I understand it, they would have to train a whole knew model to hard cap it's context to different lengths. That would be cheaper to train and had cheaper inf, but still a huge investment. So I'd guess it's API level.

From the Kimi K3 technical report:

    Kimi K3 supports a context window of up to 1 million tokens. We achieve this through extending
    the context window progressively as training proceeds, following a four-stage curriculum. The
    window grows from 8K to 64K tokens during pre-training, and from 256K to 1M tokens during the
    cooldown phase.
https://arxiv.org/pdf/2607.24653 page 12

Re: Kimi K3-256k

#144

Post by tripplyons »

pornel wrote: post #104
wren6991 wrote: post #39

This seems functionally similar to OpenAI having a step in pricing once you exceed a certain context length (also at 272k aka 2^18 aka 256k). Having a lot of active context increases the per-token cost (flops issued and bytes read per token out) so it makes sense to pass that cost on to users. I'm actually surprised it's implemented as a hard cutoff instead of a smooth gradient.

RAM needed for keeping KV cache around may be the more expensive factor.

You only need to cache for tokens that are actually being used. Using 100k tokens of a 256k token window takes the amount of memory as using 100k tokens of a 1m token window.

Re: Kimi K3-256k

#145

Post by johnbarron »

SwellJoe wrote: post #137

I make a point of never going beyond about 220k, unless absolutely necessary (and it's almost never necessary), anyway, even with models that degrade more slowly, so this is just a discount.

64k ought to be enough for anybody

Re: Kimi K3-256k

#146

Post by anuramat »

NoahZuniga wrote: post #126
MrBuddyCasino wrote: post #103

Earlier quoted context omitted.

AI infra buildup is so massive that the frontier labs should be able to offer more than one level of context length to incentivize token thriftiness. One would think compute-constrained actors like Anthropic would have done so, unless prefill isn’t really a bottleneck compared to decode?

No, the main issue is that it's hard to communicate pricing where token price increases as token count increases, and they figure they can approximate the parabola well enough with two lines.

why can't they just increase the price for cached input instead?

Re: Kimi K3-256k

#147

Post by dannyw »

MrBuddyCasino wrote: post #103
BoorishBears wrote: post #47

Earlier quoted context omitted.

Not surprising it's a hard cutoff: they almost certainly have two infrastructure configurations for the two max sequence lengths Fewer nodes dedicated to prefill per instance, and fewer nodes in total since you don't need to support a higher KV cache. Disaggregated inference also means they can tune the balance of compute dedicated to prefill seperately from decode

AI infra buildup is so massive that the frontier labs should be able to offer more than one level of context length to incentivize token thriftiness. One would think compute-constrained actors like Anthropic would have done so, unless prefill isn’t really a bottleneck compared to decode?

There's the infra cost of having multiple SKUs.

If you create 3 buckets of inference pods, say, 256k, 512k, and 1M, then you have to worry about filling/dynamically-scaling all of them.

And my guess is there's probably not a huge amount of customers that want somewhere in between: if you're willing to pay the long context surcharge; you're probably semi-price-insensitive anyway to just use 1M.

Re: Kimi K3-256k

#148

Post by adamtaylor_13 »

johnbarron wrote: post #145
SwellJoe wrote: post #137

I make a point of never going beyond about 220k, unless absolutely necessary (and it's almost never necessary), anyway, even with models that degrade more slowly, so this is just a discount.

64k ought to be enough for anybody

Why stop there? No true developer needs more than 32k!

--

Jesting aside, I cannot think of a single session in recent memory that used less than ~70k tokens, so I presume you are joking.

Re: Kimi K3-256k

#149

Post by dannyw »

k__ wrote: post #132
HawtAds wrote: post #5

This is just an API level change right? The model itself should be the same I think.

As I understand it, they would have to train a whole knew model to hard cap it's context to different lengths. That would be cheaper to train and had cheaper inf, but still a huge investment. So I'd guess it's API level.

You absolutely don't need to 'retrain' to reduce your context window. In vLLM it is an inference parameter. Smaller context window, smaller KV, less RAM needed to serve the same volume of requests.

With a lot of architectures, you technically don't need to retrain to extend the context window either; e.g. RoPE scaling; but performance is typically crap.

Re: Kimi K3-256k

#150

Post by vehemenz »

jsemrau wrote: post #95
timcobb wrote: post #88

Codex uses 256k masterfully, 1M is luxurious but still quite expensive and seems not necessary as a default.

Needing large context windows is an illusion.

So, all types of knowledge work and coding require the same amount of context?

This seems like a hot take not really informed by experience.

Post reply on HN