Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I burned all my tokens researching how to save tokens

quesma.com

Post reply on HN
Print view View original

11–20 of 237 posts

Re: I burned all my tokens researching how to save tokens

#11

Post by xyzsparetimexyz »

bob1029 wrote: post #7

Simply starting in the right part of the search space is the biggest predictor of success. The best way to save tokens is to start out the deep research pass with cheap models and then funnel the findings through increasingly powerful models. I've got a pipeline right now that uses all 3 of the gpt-5.6 model variants to address each stage of the process. If you are using models like sol or terra to generate hypotheti…

Couldn't someone build an adaptive system, where the llm is frequently judging the difficulty of a task and switching to a more/less powerful model?

Re: I burned all my tokens researching how to save tokens

#12

Post by dofm »

It sometimes feels to me that cloud AI providers have convinced people that cloud AI is worth it because of all the ways people have been able to use cloud AI to write blog posts about using cloud AI to make cloud AI more efficient for something that they haven't shipped yet and aren't really ready to talk about.

Re: I burned all my tokens researching how to save tokens

#13

Post by jscottbee »

I stopped using any of the built-in skills and skills agents and tried to keep things in the model only, and that seems to work better for token usage for me. It took me some time to tune my skills .md, but it rocks on now. I use them all in both Claude-code and Codex. I will say, GPT is better on token usage than Claude is, but for some of my code bases, Codex is not as good, but seems to be getting better with 5.6-sol.

Re: I burned all my tokens researching how to save tokens

#14

Post by noisy_boy »

edg5000 wrote: post #10
Arkhetia wrote: post #8

Been through this exact loop building an LLM product. The counterintuitive lesson for me: most "token saving" ideas are cache killers — anything that makes the prompt prefix dynamic (rotating retrieval, appending context per turn) can cost more than the tokens it saves, because you lose the cached-prefix discount. Freezing the retrieval selection early in a conversation and paying a slightly larger fixed prefix beat…

Yes, but you could have an uncached dynamic tail, this would not kill the cache though. And you could still periodically kill the cache, still keeping the same prefix every N requests. But it's questionable what all this extra complexity delivers. I found that now that the models are better, at least Codex compaction with GPT 5.5+ xhigh is almost unnoticible. So the answer is: just let the context fill up, don't prun…

That's what I have been doing with Cursor with auto mode. Let the model manage the cache through compaction but staying focused on the topic. It has been quite efficient cost wise and quality wise too. It found issuesbat cheaper cost that Sonnet 4.6 medium effort had missed while costing more. I would like to repeat this test with higher tier Anthropic models like Opus effort medium to see the results.

Re: I burned all my tokens researching how to save tokens

#15

Post by jscottbee »

bob1029 wrote: post #7

Simply starting in the right part of the search space is the biggest predictor of success. The best way to save tokens is to start out the deep research pass with cheap models and then funnel the findings through increasingly powerful models. I've got a pipeline right now that uses all 3 of the gpt-5.6 model variants to address each stage of the process. If you are using models like sol or terra to generate hypotheti…

Yeah, I do this as well. Some of the last large tools I built have been started with Gemini (I get it with my Google package), and even one with Copilot.

Re: I burned all my tokens researching how to save tokens

#16

Post by oggreen »

dofm wrote: post #12

It sometimes feels to me that cloud AI providers have convinced people that cloud AI is worth it because of all the ways people have been able to use cloud AI to write blog posts about using cloud AI to make cloud AI more efficient for something that they haven't shipped yet and aren't really ready to talk about.

The reality however is that unless you're spending tens of thousands on hardware or setting up cloud instances you can't run anything near close to state of the art.

In almost all cases right now, a subscription with a cloud AI wins.

Re: I burned all my tokens researching how to save tokens

#17

Post by bkotrys »

xyzsparetimexyz wrote: post #11
bob1029 wrote: post #7

Simply starting in the right part of the search space is the biggest predictor of success. The best way to save tokens is to start out the deep research pass with cheap models and then funnel the findings through increasingly powerful models. I've got a pipeline right now that uses all 3 of the gpt-5.6 model variants to address each stage of the process. If you are using models like sol or terra to generate hypotheti…

Couldn't someone build an adaptive system, where the llm is frequently judging the difficulty of a task and switching to a more/less powerful model?

definitely something that will come sooner than later. In my view, adaption of model routing / model orchestration and mixing providers is just natural next step.

Re: I burned all my tokens researching how to save tokens

#18

Post by iamflimflam1 »

xyzsparetimexyz wrote: post #11
bob1029 wrote: post #7

Simply starting in the right part of the search space is the biggest predictor of success. The best way to save tokens is to start out the deep research pass with cheap models and then funnel the findings through increasingly powerful models. I've got a pipeline right now that uses all 3 of the gpt-5.6 model variants to address each stage of the process. If you are using models like sol or terra to generate hypotheti…

Couldn't someone build an adaptive system, where the llm is frequently judging the difficulty of a task and switching to a more/less powerful model?

This only works well if you are creating sub agents with clean contexts for each task. If you constantly are switching models part way through some work then the whole session needs to be replayed each switch. You lose all the benefits of the context cache.

Re: I burned all my tokens researching how to save tokens

#19

Post by bkotrys »

jscottbee wrote: post #13

I stopped using any of the built-in skills and skills agents and tried to keep things in the model only, and that seems to work better for token usage for me. It took me some time to tune my skills .md, but it rocks on now. I use them all in both Claude-code and Codex. I will say, GPT is better on token usage than Claude is, but for some of my code bases, Codex is not as good, but seems to be getting better with 5.6-…

Try to mix both subscribtions and take the best from both providers. In my opinion, we should not just one over another but understand the power of both and mix them to reach the better results.

Re: I burned all my tokens researching how to save tokens

#20

Post by resonious »

dofm wrote: post #12

It sometimes feels to me that cloud AI providers have convinced people that cloud AI is worth it because of all the ways people have been able to use cloud AI to write blog posts about using cloud AI to make cloud AI more efficient for something that they haven't shipped yet and aren't really ready to talk about.

Trust me there are plenty of us using cloud AI to actually ship stuff. We just aren't writing blog posts about it.
Post reply on HN