Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I burned all my tokens researching how to save tokens

quesma.com

Post reply on HN
Print view View original

41–50 of 237 posts

Re: I burned all my tokens researching how to save tokens

#41

Post by dofm »

stavros wrote: post #33
dominotw wrote: post #30

Earlier quoted context omitted.

what did you ship?

Whenever I've answered this question, the reply was always "this is shit", so I don't bother now. Bad faith questions just shouldn't be answered.

If you characterise questions as bad faith so you don’t have to answer, sure.

Since this is my thread, and since I am an exasperated freelancer who has made most of his post-dot-com career from writing exactly the kind of quite small, simple, unambitious, often internal things for smaller customers that nobody would confuse with anything cool, who is trying to understand how AI is going to make his life any better when it’s so opaque, fast moving, poorly documented and full of unverifiable claims battling unverifiable claims, let me assure you that the following question is not being asked in bad faith:

What did you ship?

Re: I burned all my tokens researching how to save tokens

#42

Post by itishappy »

stavros wrote: post #33
dominotw wrote: post #30

Earlier quoted context omitted.

what did you ship?

Whenever I've answered this question, the reply was always "this is shit", so I don't bother now. Bad faith questions just shouldn't be answered.

Why assume bad faith? I've witnessed a barrage of "proof of concepts" at my dayjob by folks who have zero intention of ever hooking up their dashboard to our actual messy data. Shipping is still a major barrier. Is your experience meaningfully different?

Re: I burned all my tokens researching how to save tokens

#43

Post by BobbyTables2 »

bkotrys wrote: post #5
nekusar wrote: post #4

Makes sense. LLMs only have 'knowledge' that was encoded by scanning as much data as possible. (Same way a search engine only can find data that was indexed) American LLMs are by and large closed. Chinese are open to a point. And research how these things work is still a big mystery. So yeah, was comprehensive data about saving tokens scanned and indexed? Likely no. So engaging about token saving is going to generate…

Yeah, that was pretty much my experience. The models weren’t lacking knowledge as much as discipline. Without a good workflow, they will most likely spend thousands of tokens exploring dead ends.

Sounds like an energetic developer that just graduated from college…

(Without the learning and growth potential)

Re: I burned all my tokens researching how to save tokens

#44

Post by fragmede »

oggreen wrote: post #16
dofm wrote: post #12

It sometimes feels to me that cloud AI providers have convinced people that cloud AI is worth it because of all the ways people have been able to use cloud AI to write blog posts about using cloud AI to make cloud AI more efficient for something that they haven't shipped yet and aren't really ready to talk about.

The reality however is that unless you're spending tens of thousands on hardware or setting up cloud instances you can't run anything near close to state of the art. In almost all cases right now, a subscription with a cloud AI wins.

> In almost all cases right now,

What sort of benchmark or test would convince you that "almost all" is more like "your weird set of cases" and nowhere near "almost all"?

Re: I burned all my tokens researching how to save tokens

#45

Post by xyzsparetimexyz »

iamflimflam1 wrote: post #18
xyzsparetimexyz wrote: post #11

Earlier quoted context omitted.

Couldn't someone build an adaptive system, where the llm is frequently judging the difficulty of a task and switching to a more/less powerful model?

This only works well if you are creating sub agents with clean contexts for each task. If you constantly are switching models part way through some work then the whole session needs to be replayed each switch. You lose all the benefits of the context cache.

Well yeah, it'd require the models to be loaded on the same system and the cache to be shared between them somehow

Re: I burned all my tokens researching how to save tokens

#47

Post by nojito »

dofm wrote: post #12

It sometimes feels to me that cloud AI providers have convinced people that cloud AI is worth it because of all the ways people have been able to use cloud AI to write blog posts about using cloud AI to make cloud AI more efficient for something that they haven't shipped yet and aren't really ready to talk about.

Cloud AI is 100% worth it. We've been able to do so much more with so much less.

Re: I burned all my tokens researching how to save tokens

#48

Post by dandellion »

kittoes wrote: post #36
dominotw wrote: post #30

Earlier quoted context omitted.

what did you ship?

https://github.com/ByteTerrace/Puck/blob/main/src/Puck.World... "Shipped" is a bit strong in my case, but I think this counts as something that isn't mere slop.

One sentence into that readme I can tell you that yes, it most definitely counts as slop.
Post reply on HN