Show HN: Tokensift, an open-sourced token-efficiency linter for LLM prompts
1–6 of 6 posts
Re: Show HN: Tokensift, an open-sourced token-efficiency linter for LLM prompts
#2[dead]
Re: Show HN: Tokensift, an open-sourced token-efficiency linter for LLM prompts
#3Does it work also on multi modal LLMs?
Re: Show HN: Tokensift, an open-sourced token-efficiency linter for LLM prompts
#4Does it work also on multi modal LLMs?
It does work partially. TokenSift can analyze the textual parts of prompts sent to multimodal models, but doesn’t calculate or optimize image/audio/video token usage. Non-text payload calculation is something I’d like to add.
Re: Show HN: Tokensift, an open-sourced token-efficiency linter for LLM prompts
#5Hello, author here.
Most prompts include personalizing with a user's data. These prompts grow, and there is quite a bit of extra content that can be removed. UUIDs, pretty-printed JSON, ISO timestamps, repeated instruction blocks, etc., and all of it tokenizes to cost a lot.
tokensift is a linter for that. It tokenizes the prompt with a real encoder the specific model uses: exact BPE counts for OpenAI and a calibrated estimate for Claude with the measured error shown. Then, it runs a set of rules against it and reports where the tokens are going, along with the dollar cost per finding.
Works as a library or a CLI, and runs in CI.
Current state: 20 rules, OpenAI and Claude supported, no Gemini yet.
Re: Show HN: Tokensift, an open-sourced token-efficiency linter for LLM prompts
#6[flagged]