Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I burned all my tokens researching how to save tokens

quesma.com

Post reply on HN
Print view View original

121–130 of 237 posts

Re: I burned all my tokens researching how to save tokens

#121

Post by bkotrys »

Bender wrote: post #23

I am not a proper developer and only use AI for faster research of topics so please forgive my ignorance. Could one not save a lot of money on tokens by using the 80/20 or 90/10 rule in that 90% of AI usage is on local models and save that last 10% or less for the frontier models where the local model did not meet the needs? Did they cover this and I misunderstood?

That can work, especially for privacy or repetitive tasks. My experiment focused on subscriptions I already had, but local models are a natural addition to the routing layer.

Re: I burned all my tokens researching how to save tokens

#122

Post by bkotrys »

Arkhetia wrote: post #8

Been through this exact loop building an LLM product. The counterintuitive lesson for me: most "token saving" ideas are cache killers — anything that makes the prompt prefix dynamic (rotating retrieval, appending context per turn) can cost more than the tokens it saves, because you lose the cached-prefix discount. Freezing the retrieval selection early in a conversation and paying a slightly larger fixed prefix beat…

[dead]

Re: I burned all my tokens researching how to save tokens

#123

Post by bkotrys »

iririririr wrote: post #84

echo "$OUT" | claude-mem-save so, does agents are ultra aware their output is used in a bash line or did bash now have some magic quote super power? I'd bet this only really work 2% of the time in real world.

At the shell level, the agent does not need to know about Bash. The contents of "$OUT" are passed to echo as one argument, then sent to claude-mem-save over stdin. Bash does not re-evaluate them as shell code. printf '%s\n' "$OUT" would be more robust than echo. Whether claude-mem-save can use arbitrary output is a separate question.

Re: I burned all my tokens researching how to save tokens

#124

Post by nojito »

inigyou wrote: post #88
nojito wrote: post #83

Earlier quoted context omitted.

Happier engineering teams? Happier end users of the data? What does useful mean to you?

So far you've only explained that AI made your coworkers happier. That's fine, but weren't we talking about productivity? Doing more with less?

Correct. We were chronically understaffed and now we are able to actually breathe and can argue that we are finally right sized.

Re: I burned all my tokens researching how to save tokens

#126

Post by inigyou »

xyzsparetimexyz wrote: post #117
inigyou wrote: post #58

Earlier quoted context omitted.

Cache sharing is not possible. The numbers in the cache are completely specific to the model.

Currently. I'm sure that you could make a system where the cache values are a superset C of e.g. models A and B where C is probably bigger than max(A,B) but smaller than A+B

It's equal to A+B. There is literally no sharing possible.

Re: I burned all my tokens researching how to save tokens

#127

Post by inigyou »

nojito wrote: post #124
inigyou wrote: post #88

Earlier quoted context omitted.

So far you've only explained that AI made your coworkers happier. That's fine, but weren't we talking about productivity? Doing more with less?

Correct. We were chronically understaffed and now we are able to actually breathe and can argue that we are finally right sized.

So far you have provided no evidence of achieving more results with less effort. You've only said that you can write more lines of code (effectively) and that your coworkers are happier

Re: I burned all my tokens researching how to save tokens

#128

Post by inigyou »

kittoes wrote: post #109
inigyou wrote: post #106

Earlier quoted context omitted.

this is definitely slop. It's not slop because it supports 4-player split screen? That's your excuse? I think if I posted this same project to you and you'd never heard of it, you would come to the same conclusion.

Did you actually try to read what I wrote, or are you just stopping at the very first thing you can parse? I'm engaging in good faith with you here, it would be nice if you even remotely attempted to do the same. You can literally walk around in a world, pick up an in-world device, turn it on, and watch/hear the device play an actual ROM. Furthermore, you could edit that ROM while you're playing it. Recursion and det…

A render graph is also nothing new, and neither is screen capture.

Re: I burned all my tokens researching how to save tokens

#129

Post by 6510 »

dominotw wrote: post #30
resonious wrote: post #20

Earlier quoted context omitted.

Trust me there are plenty of us using cloud AI to actually ship stuff. We just aren't writing blog posts about it.

what did you ship?

Thinking about it a better question [for me] would be what might have but didn't ship because of LLMs.

Re: I burned all my tokens researching how to save tokens

#130

Post by stavros »

dofm wrote: post #87
stavros wrote: post #79

Earlier quoted context omitted.

You're welcome! LLMs definitely help with the migrations, there were definitely a few iterations because the surface area was massive, but overall it took the project from "will never be prioritised" to "I can work on it on and off when in boring meetings", which was a massive win. I haven't found that LLMs help with CAD at all, but YMMV. As for sharing here, the last time I shared something with "here's something I…

Oh I definitely don't expect them to help with CAD and nor would I want them to — I really enjoy knowing FreeCAD. It's more the microcontroller choice, broad components, all that. The main areas I find LLMs struggle with research, I struggle with research too. I am a pretty solid researcher and I find it validating, in fact, to find that a more brute-force-trained thing is not better than me. I see your point about t…

By the way, I've written up my workflow if that interests you: https://www.stavros.io/posts/how-i-write-software-with-llms/
Post reply on HN