Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I burned all my tokens researching how to save tokens

quesma.com

Post reply on HN
Print view View original

31–40 of 237 posts

Re: I burned all my tokens researching how to save tokens

#31

Post by jack_pp »

Bender wrote: post #23

I am not a proper developer and only use AI for faster research of topics so please forgive my ignorance. Could one not save a lot of money on tokens by using the 80/20 or 90/10 rule in that 90% of AI usage is on local models and save that last 10% or less for the frontier models where the local model did not meet the needs? Did they cover this and I misunderstood?

Unlees you invest thousands your local ai won't even come close to the cheap cloud llms. Local is only worth it if you care about privacy or have a legitimate usage for the hardware otherwise. Money wise it isn't worth it

Re: I burned all my tokens researching how to save tokens

#32

Post by jscottbee »

nekusar wrote: post #4

Makes sense. LLMs only have 'knowledge' that was encoded by scanning as much data as possible. (Same way a search engine only can find data that was indexed) American LLMs are by and large closed. Chinese are open to a point. And research how these things work is still a big mystery. So yeah, was comprehensive data about saving tokens scanned and indexed? Likely no. So engaging about token saving is going to generate…

What most AI model users don't understand is (most here will know) that they are probabilistic (statistically based) and not deterministic (static/fixed). So answers to the same prompts can vary (wildly sometimes)

Re: I burned all my tokens researching how to save tokens

#33

Post by stavros »

dominotw wrote: post #30
resonious wrote: post #20

Earlier quoted context omitted.

Trust me there are plenty of us using cloud AI to actually ship stuff. We just aren't writing blog posts about it.

what did you ship?

Whenever I've answered this question, the reply was always "this is shit", so I don't bother now. Bad faith questions just shouldn't be answered.

Re: I burned all my tokens researching how to save tokens

#35

Post by Forgeties79 »

stavros wrote: post #33
dominotw wrote: post #30

Earlier quoted context omitted.

what did you ship?

Whenever I've answered this question, the reply was always "this is shit", so I don't bother now. Bad faith questions just shouldn't be answered.

This just feels like a bad faith comment designed to spike the conversation entirely.

Re: I burned all my tokens researching how to save tokens

#36

Post by kittoes »

dominotw wrote: post #30
resonious wrote: post #20

Earlier quoted context omitted.

Trust me there are plenty of us using cloud AI to actually ship stuff. We just aren't writing blog posts about it.

what did you ship?

https://github.com/ByteTerrace/Puck/blob/main/src/Puck.World...

"Shipped" is a bit strong in my case, but I think this counts as something that isn't mere slop.

Re: I burned all my tokens researching how to save tokens

#37

Post by speedster217 »

Forgeties79 wrote: post #35
stavros wrote: post #33

Earlier quoted context omitted.

Whenever I've answered this question, the reply was always "this is shit", so I don't bother now. Bad faith questions just shouldn't be answered.

This just feels like a bad faith comment designed to spike the conversation entirely.

Agreed. AI has been "just trust me bro" for so long. People need to show receipts instead of spreading vague hype

Re: I burned all my tokens researching how to save tokens

#38

Post by vitally3643 »

I get a lot of satisfaction using a significant chunk of my work-provided Claude plan researching and developing local LLM solutions for work.

Claude has been helping me tune models to run better on our big compute rack at work, researching which leading edge open models will fit in hardware and are good for our workloads, and it's been helping me write test fixtures to evaluate how things perform.

At several points, Claude has expressed surprise at the quality of results from my local LLMs. While I know that doesn't mean anything, it still feels like giving Anthropic the finger, which is always great.

Re: I burned all my tokens researching how to save tokens

#39

Post by dominotw »

stavros wrote: post #33
dominotw wrote: post #30

Earlier quoted context omitted.

what did you ship?

Whenever I've answered this question, the reply was always "this is shit", so I don't bother now. Bad faith questions just shouldn't be answered.

shipped = paying real users ( or a non profit with users)

doesnt matter what subjective opinions are.

shipped is not pushing something on github.

Re: I burned all my tokens researching how to save tokens

#40

Post by pianopatrick »

Bender wrote: post #23

I am not a proper developer and only use AI for faster research of topics so please forgive my ignorance. Could one not save a lot of money on tokens by using the 80/20 or 90/10 rule in that 90% of AI usage is on local models and save that last 10% or less for the frontier models where the local model did not meet the needs? Did they cover this and I misunderstood?

To run a local AI that is half decent at research at usable speeds requires hardware that costs thousands of dollars.

Spending thousands of dollars on hardware to save dollars per month on tokens does not make financial sense.

If you run the numbers you'll probably find that using cheaper cloud models makes more financial sense than running those same models locally.

Post reply on HN