Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Lowfat – pluggable CLI filter that saved 91.8% of my LLM tokens

github.com

Post reply on HN
Print view View original

41–50 of 97 posts

Re: Show HN: Lowfat – pluggable CLI filter that saved 91.8% of my LLM tokens

#41

Post by mywittyname »

jemmyw wrote: post #13

I've tried rtx and lean-ctx and these tools seem to end up confusing the agent more than helping. Any saving is irrelevant if the agent decides to work around the tool and makes even more calls than it would otherwise. I don't know about cost saving, but if it's keeping the context size down I've had a lot better results using subagents to keep a higher order conversation clean for longer.

Also, most of the time when I'm having an agent look through logs or output, it's grepping for the bits of data relevant to its actions.

Re: Show HN: Lowfat – pluggable CLI filter that saved 91.8% of my LLM tokens

#42

Post by no-name-here »

onlyrealcuzzo wrote: post #34
alex7o wrote: post #5

I would like to have deeper comparison with alternatives like rtk, which are already fast and written in rust, also the previous comments mentioned something that has been a know problem with rtk that it sometimes strips the thing that the llm needs (or expects, causing more work to need to happan not less)

None of these tools measure how effective they are... It's a massive red flag to me when you could get decent data to see if your thing actually works, and they don't even attempt to... Have the LLM use your tool, run it on several of the coding benchmarks. If you're stingy, run it on the ones that don't cost much. Otherwise, I'm going to assume it doesn't actually work. If it did - Claude, Antigravity, Codex, Pi, or…

> I'm going to assume it doesn't actually work. If it did - Claude, Antigravity, Codex, Pi, or some major player would bundle tools like this into the CLI / harness.

VS Code launched it as a feature in their bundled AI functionality last month: https://code.visualstudio.com/updates/v1_121

Re: Show HN: Lowfat – pluggable CLI filter that saved 91.8% of my LLM tokens

#43

Post by rahulyc »

Great idea. I'm thinking if it could make sense to send the output to a cheap / local model to filter out only the bits that "matter" and pass that through - for the cost some extra time, but maybe it's worth it for saving tokens in the larger model.

Re: Show HN: Lowfat – pluggable CLI filter that saved 91.8% of my LLM tokens

#44

Post by irthomasthomas »

onlyrealcuzzo wrote: post #34
alex7o wrote: post #5

I would like to have deeper comparison with alternatives like rtk, which are already fast and written in rust, also the previous comments mentioned something that has been a know problem with rtk that it sometimes strips the thing that the llm needs (or expects, causing more work to need to happan not less)

None of these tools measure how effective they are... It's a massive red flag to me when you could get decent data to see if your thing actually works, and they don't even attempt to... Have the LLM use your tool, run it on several of the coding benchmarks. If you're stingy, run it on the ones that don't cost much. Otherwise, I'm going to assume it doesn't actually work. If it did - Claude, Antigravity, Codex, Pi, or…

My partial solution to this was to store the full response in a file and prompt the agent to read that if the condensed version had stuff missing.

Re: Show HN: Lowfat – pluggable CLI filter that saved 91.8% of my LLM tokens

#45

Post by onlyrealcuzzo »

no-name-here wrote: post #42
onlyrealcuzzo wrote: post #34

Earlier quoted context omitted.

None of these tools measure how effective they are... It's a massive red flag to me when you could get decent data to see if your thing actually works, and they don't even attempt to... Have the LLM use your tool, run it on several of the coding benchmarks. If you're stingy, run it on the ones that don't cost much. Otherwise, I'm going to assume it doesn't actually work. If it did - Claude, Antigravity, Codex, Pi, or…

> I'm going to assume it doesn't actually work. If it did - Claude, Antigravity, Codex, Pi, or some major player would bundle tools like this into the CLI / harness. VS Code launched it as a feature in their bundled AI functionality last month: https://code.visualstudio.com/updates/v1_121

Bundling implies interest...

Defaults imply working...

Re: Show HN: Lowfat – pluggable CLI filter that saved 91.8% of my LLM tokens

#46

Post by tim-projects »

avocadoking wrote: post #39

Do you have any insight if LLMs sometimes get confused by your filters?

He says he adds an output message, but I've tried this myself and I find that quite a lot of the time the agent prefers its own internal monologue over the output of a command.

Re: Show HN: Lowfat – pluggable CLI filter that saved 91.8% of my LLM tokens

#47

Post by taude »

onlyrealcuzzo wrote: post #34
alex7o wrote: post #5

I would like to have deeper comparison with alternatives like rtk, which are already fast and written in rust, also the previous comments mentioned something that has been a know problem with rtk that it sometimes strips the thing that the llm needs (or expects, causing more work to need to happan not less)

None of these tools measure how effective they are... It's a massive red flag to me when you could get decent data to see if your thing actually works, and they don't even attempt to... Have the LLM use your tool, run it on several of the coding benchmarks. If you're stingy, run it on the ones that don't cost much. Otherwise, I'm going to assume it doesn't actually work. If it did - Claude, Antigravity, Codex, Pi, or…

I don't think frontier model providers are going to be incentivized to invest in this much, yet. Once inference gets more competitive, sure. I haven't looked lately, but won't be surprised if tools like OpenCode do do what you're suggesting, though. Third-party coding harnesses ARE aligned to deliver this type of feature and optimization.

Re: Show HN: Lowfat – pluggable CLI filter that saved 91.8% of my LLM tokens

#49

Post by clutter55561 »

Tools that remove the fat seem like a good idea, but I’m highly suspicious of their effect on the LLM’s reasoning.

LLMs were trained in the typical full-fat output found everywhere on the internet, and all of sudden they get a slightly different response that may look like nothing they have seen before.

Does that really save tokens in the long run?

Re: Show HN: Lowfat – pluggable CLI filter that saved 91.8% of my LLM tokens

#50

Post by varispeed »

onlyrealcuzzo wrote: post #34
alex7o wrote: post #5

I would like to have deeper comparison with alternatives like rtk, which are already fast and written in rust, also the previous comments mentioned something that has been a know problem with rtk that it sometimes strips the thing that the llm needs (or expects, causing more work to need to happan not less)

None of these tools measure how effective they are... It's a massive red flag to me when you could get decent data to see if your thing actually works, and they don't even attempt to... Have the LLM use your tool, run it on several of the coding benchmarks. If you're stingy, run it on the ones that don't cost much. Otherwise, I'm going to assume it doesn't actually work. If it did - Claude, Antigravity, Codex, Pi, or…

You can't measure effectiveness, because you never know what kind of model will process your prompt. One request you might get full e.g. Opus and another they'll downgrade it to Sonnet or something more basic. I have this with "Opus 4.8" all the time.
Post reply on HN