[dead]
Ask HN: What is the best way to provide continuous context to models?
41–48 of 48 posts
Re: Ask HN: What is the best way to provide continuous context to models?
#42Every time you send a request to a model you're already providing all of the context history along with it. To edit the context, just send a different context history. You can send whatever you want as history, it's entirely up to you and entirely arbitrary. We only think in conversational turns because that's what we've expected a conversation to 'look like'. But that's just a very deeply ingrained convention. Forge…
Bigger context makes responses slower. Context is limited. You do not want the cloud provider running a context compaction if you can control it a lot better. There are even tips on when to ask the question like "send first the content then ask the question" vs. "ask the question then send the content"
When history is cached conversations tend not to be slower, because the LLM can 'continue' from a previous state.
So if there was already A + A1 + B + B1 + C + C1 and you asking 'D' ... well, [A->C1] is saved as state. It costs 10ms to prepare. Then, they add 'D' as your question and that will be done 'all tokens at once' in bulk - which is fast.
Then - they they generate D1 (the response) they have to do it one token at a time, which is slow. Each token has to be processed separately.
Also - even if they had to redo- all of [A->C1] 'from scratch' - its not that slow, because the entire block of tokens can be processed in one pass.
'prefill' (aka A->C1) is fast, which by the way is why it's 10x cheaper.
So prefill is 10x faster than generation, and cache is 10x cheaper than prefill as a very general rule of thumb.
Re: Ask HN: What is the best way to provide continuous context to models?
#43Earlier quoted context omitted.
Bigger context makes responses slower. Context is limited. You do not want the cloud provider running a context compaction if you can control it a lot better. There are even tips on when to ask the question like "send first the content then ask the question" vs. "ask the question then send the content"
When history is cached conversations tend not to be slower, because the LLM can 'continue' from a previous state. So if there was already A + A1 + B + B1 + C + C1 and you asking 'D' ... well, [A->C1] is saved as state. It costs 10ms to prepare. Then, they add 'D' as your question and that will be done 'all tokens at once' in bulk - which is fast. Then - they they generate D1 (the response) they have to do it one toke…
Thats only the case with KV Cache and we do not know how and how long providers keep it.
Re: Ask HN: What is the best way to provide continuous context to models?
#44I think the emerging best way is to do "agentic search" over files. If you think about it, Claude Code is quite good at navigating large codebases and finding the required context for a problem. Further, instead of polluting the context of your main agent, you can run a subagent to do search and retrieve the important bits of information and report back to your main agent. This is what Claude Code does if you use the…
You can also create per-project agents with specific "expertise" in different parts of the code. Basically they're just few kilobytes of text that's given as extra context to "explore" agents when looking at specific parts of the code.
How does one do this?
I have trouble making Claude even follow CLAUDE.md. Like, it just ignores it, and when it needs to do a task (like testing or running a certain command native to the code base), it does a lot of "find" and "ls" an so on to understand what to run, and frequently runs commands with the wrong flags, badly escaped input etc.
I think Claude could work a lot better (and waste vastly fewer tokens) if it had small bite-size instructions for various tasks ("to run the integration test suite, run "cargo run integration_tests..."), but so far I've been unable to control it.
Re: Ask HN: What is the best way to provide continuous context to models?
#45I've been building https://www.usesatori.sh/ to give persistent context to agents Would be happy to onboard you personally.
i see you are only using vector search; many solutions for memory involve some combination of vector search and/or graphs (mem0, zep, cognee, etc). have you compared against these?
Definitely going to incorporate graph's in the near future. Connectivity to business apps is I think a core differentiator the other products dont have -- but planning to iterate to what is useful and valuable
Re: Ask HN: What is the best way to provide continuous context to models?
#46Earlier quoted context omitted.
When history is cached conversations tend not to be slower, because the LLM can 'continue' from a previous state. So if there was already A + A1 + B + B1 + C + C1 and you asking 'D' ... well, [A->C1] is saved as state. It costs 10ms to prepare. Then, they add 'D' as your question and that will be done 'all tokens at once' in bulk - which is fast. Then - they they generate D1 (the response) they have to do it one toke…
Thats only the case with KV Cache and we do not know how and how long providers keep it.
Prefill is 10x faster than generation without caching, and 100x faster with caching - as a very crude measure. So it's not a matter of 'only the case'. Those are different scenarios. Some hosts are better than others with respect to managing caching, but the better one's provide decent SLA on that.
Re: Ask HN: What is the best way to provide continuous context to models?
#47[flagged]