Interesting concept, but is it going to age well with context sizes of models are changing all the time (growing, mostly)?
max context sizes are probably going to go up, but smaller contexts will always be cheaper/more-efficient than larger ones
Show HN: Badge that shows how well your codebase fits in an LLM's context window
11–20 of 47 posts
Re: Show HN: Badge that shows how well your codebase fits in an LLM's context window
#12[flagged]
Re: Show HN: Badge that shows how well your codebase fits in an LLM's context window
#13[flagged]
Re: Show HN: Badge that shows how well your codebase fits in an LLM's context window
#14What’s the going rate for tokens in terms of dollars? How much are companies spending on “tokens”? Also kind of ironic that small codebases are now in vogue, just when google monolithic repos were so popular.
> What’s the going rate for tokens in terms of dollars? It depends on the provider/model, usually pricing is calculated as $/million tokens with input/output tokens having different per token pricing (output tends to be more expensive than input). Some models also charge more per token if the context size is above a threshold. Cached operations may also reduce the price per token. OpenRouter has a good overview over…
Api keys with hard limits I assume?
Are there companies out there that add token counts to ticket “costs”, i.e. are story points being replaced/augmented by token counts?
Or even worse, an exchange rate of story points to tokens used…
Re: Show HN: Badge that shows how well your codebase fits in an LLM's context window
#15It's a fun, in the "style of the time" thing to track, but within a year or two, context window limitations won't be a thing.
Doubt me?
Think back 2 years. Now compare today. Change is at massive speed, and this issue is top line to be resolved in some fashion.
Re: Show HN: Badge that shows how well your codebase fits in an LLM's context window
#16I'm not sure that smaller bases are always better.
Re: Show HN: Badge that shows how well your codebase fits in an LLM's context window
#17It's a fun, in the "style of the time" thing to track, but within a year or two, context window limitations won't be a thing. Doubt me? Think back 2 years. Now compare today. Change is at massive speed, and this issue is top line to be resolved in some fashion.
I’m not so sure an increasingly large context window will be seen as a critical enabler (as it was viewed 6 months ago), after watching how amazingly effective subagents and tool calls are at tackling parts of the problem and surfacing the just the relevant bits for the task at hand. And if increasing the context window isn’t the current bottleneck, effort will be put elsewhere.
Re: Show HN: Badge that shows how well your codebase fits in an LLM's context window
#18It's a fun, in the "style of the time" thing to track, but within a year or two, context window limitations won't be a thing. Doubt me? Think back 2 years. Now compare today. Change is at massive speed, and this issue is top line to be resolved in some fashion.
Gemini 1.5 Announced the 1 million token context window in 2024. I admire this view of being forward looking towards new technologies, specially when we see the history of how bad people can be at predictions just by looking at history HN posts/comments.
If we look at back 2 years, companies weren't investing into training their LLMs so heavily on code. Any code they got their hands on was what was in the LLMs training corpus, it's well known that the most recent improvements in LLM productivity occurred after they spent millions on different labs to produce more coding datasets for them.
So while LLMs have gotten a lot better at not needing the entire codebase in context at once, because their weights are already so well tuned to development environments they can better infer and index things as needed. However, I fail to see how the context window limitation would no longer be an issue since it's a fundamental part of the real world. Would we get better and more efficient ways of splitting and indexing context windows? Surely. Will that reduce our fear of soiling our contexts with bad prompt response cycles? Probably not...
Re: Show HN: Badge that shows how well your codebase fits in an LLM's context window
#19Interesting, but not adding something to my CI for a badge, too paranoid.
Re: Show HN: Badge that shows how well your codebase fits in an LLM's context window
#20Useful and useless (or good and “less good”) aren’t easily mapped to big and small. From a purely UX perspective, showing a red badge seems you’re conflating “less good” with size. Who is the target for this? Lots of useful codebases are large. I do agree, however, that there’s value in splitting up domains into something a human can easily learn and keep in their head after, say, a few days of being deeply entrenche…
> Who is the target for this? Agents. Going to be more tools and software targeted for consumption by agents
Yeah, but a large monorepo can consist of many small subprojects. And arguably this is becoming a best practice.
Just spawn the agent in one of the subprojects