I burned all my tokens researching how to save tokens
1–10 of 237 posts
Re: I burned all my tokens researching how to save tokens
#2What a great confirmation of what I've tried to achieve by getting models to communicate with my local data in shorter communicates to save on token command bandwidth, more or less the same results! Thanks for the article!
Re: I burned all my tokens researching how to save tokens
#3What a great confirmation of what I've tried to achieve by getting models to communicate with my local data in shorter communicates to save on token command bandwidth, more or less the same results! Thanks for the article!
Thank you, glad you found the article useful!
Re: I burned all my tokens researching how to save tokens
#4Makes sense.
LLMs only have 'knowledge' that was encoded by scanning as much data as possible. (Same way a search engine only can find data that was indexed)
American LLMs are by and large closed. Chinese are open to a point. And research how these things work is still a big mystery.
So yeah, was comprehensive data about saving tokens scanned and indexed? Likely no. So engaging about token saving is going to generate a lot of verbose and useless slop.
Re: I burned all my tokens researching how to save tokens
#5Makes sense. LLMs only have 'knowledge' that was encoded by scanning as much data as possible. (Same way a search engine only can find data that was indexed) American LLMs are by and large closed. Chinese are open to a point. And research how these things work is still a big mystery. So yeah, was comprehensive data about saving tokens scanned and indexed? Likely no. So engaging about token saving is going to generate…
Yeah, that was pretty much my experience. The models weren’t lacking knowledge as much as discipline. Without a good workflow, they will most likely spend thousands of tokens exploring dead ends.
Re: I burned all my tokens researching how to save tokens
#6Makes sense. LLMs only have 'knowledge' that was encoded by scanning as much data as possible. (Same way a search engine only can find data that was indexed) American LLMs are by and large closed. Chinese are open to a point. And research how these things work is still a big mystery. So yeah, was comprehensive data about saving tokens scanned and indexed? Likely no. So engaging about token saving is going to generate…
Yeah, that was pretty much my experience. The models weren’t lacking knowledge as much as discipline. Without a good workflow, they will most likely spend thousands of tokens exploring dead ends.
I think 'exploring the dead ends' is *possibly* very fruitful. Various disciplines have a lot of appearing dead ends that someone in another field did solve. And they don't talk with each other.
What I've seen with lots of the breathless 'OMG SCIENCE ADVANCEMENT' articles including the one yesterday, is that the LLMs are quite extraordinary about linking a dozen different fields together, and delivering an answer combined from all of them. Basically, the solutions to a lot of current scientific problems are partially solved, but partial from a lot of fields that don't talk with each other. And also, they don't use the same nomenclature, so simple searches don't suffice.
An LLM seems to use Chomsky's Universal Grammar, and thus on training, does normalize all training data, including different words for the same thing.
A lot (perhaps all?) Of these BIG scientific advancements were solved by humans, partially. An LLM jigsawed them together and appeared it solved the thing.
Re: I burned all my tokens researching how to save tokens
#7Simply starting in the right part of the search space is the biggest predictor of success.
The best way to save tokens is to start out the deep research pass with cheap models and then funnel the findings through increasingly powerful models. I've got a pipeline right now that uses all 3 of the gpt-5.6 model variants to address each stage of the process. If you are using models like sol or terra to generate hypotheticals and perform initial exploration, you are leaving money on the table.
5 hypotheticals out of luna will massively outperform 1 hypothetical out of sol, but the cost is the same and so is the runtime if you do it in parallel. The hypothesis generation phase is also a great place to mix and match models from different vendors. The more diversity at this step the better.
The other thing I started looking into is batch pricing which represents 50% off for OAI tokens right now. With some tweaks to the UI/UX of an enterprise chatbot, I think it is possible we could have users get comfortable with the idea that questions to the robot might take a day to come back. The key is that this has to actually work. Users don't mind trading time if their questions come back with high quality results.
Re: I burned all my tokens researching how to save tokens
#8Been through this exact loop building an LLM product. The counterintuitive lesson for me: most "token saving" ideas are cache killers — anything that makes the prompt prefix dynamic (rotating retrieval, appending context per turn) can cost more than the tokens it saves, because you lose the cached-prefix discount. Freezing the retrieval selection early in a conversation and paying a slightly larger fixed prefix beat every clever adaptive scheme I measured.
Re: I burned all my tokens researching how to save tokens
#9You should consider testing our RemembrallMCP (note I'm the author). It substantially cuts token usage and you get better results in codebases.
https://github.com/roboticforce/remembrallmcp
Re: I burned all my tokens researching how to save tokens
#10Been through this exact loop building an LLM product. The counterintuitive lesson for me: most "token saving" ideas are cache killers — anything that makes the prompt prefix dynamic (rotating retrieval, appending context per turn) can cost more than the tokens it saves, because you lose the cached-prefix discount. Freezing the retrieval selection early in a conversation and paying a slightly larger fixed prefix beat…
Yes, but you could have an uncached dynamic tail, this would not kill the cache though. And you could still periodically kill the cache, still keeping the same prefix every N requests. But it's questionable what all this extra complexity delivers.
I found that now that the models are better, at least Codex compaction with GPT 5.5+ xhigh is almost unnoticible. So the answer is: just let the context fill up, don't prune much, the model is smart enough. When you're out of space, have the model summarize and continue from that point. A smart model will know the information pieces it needs to re-read after compaction; it won't make incorrect guessed when some details got lost.