31–40 of 47 posts
For example in my current case, there are lots of files with CSS, SVG icons in separate files, old database migration scripts, etc. Those don't go in the LLM context 99% of the time.
Maybe a more useful metric would be "what percentage of files that have been edited in the last {n} days fit in the context"?
What’s the going rate for tokens in terms of dollars? How much are companies spending on “tokens”? Also kind of ironic that small codebases are now in vogue, just when google monolithic repos were so popular.
> What’s the going rate for tokens in terms of dollars? It depends on the provider/model, usually pricing is calculated as $/million tokens with input/output tokens having different per token pricing (output tends to be more expensive than input). Some models also charge more per token if the context size is above a threshold. Cached operations may also reduce the price per token. OpenRouter has a good overview over…
The downside with subscriptions is that your work with the LLM will grind to a halt for a number of hours if you hit the token limit. I was doing what I consider very trivial work adding Javadoc comments to a few dozen files using Claude Sonnet on the $20 plan and within 30 minutes had been told to sit out for a couple hours. The reason was that Claude was apparently repeatedly sending the files up and down to fill in the comments. In hindsight, sure, that's obvious, but you would think that Claude would be smart enough to do some sort of summarization to make things more efficient. Looking into it, it was on the order of several million tokens in a very short amount of time.
It really made me wonder how in the hell people are using Claude to do "real" work, but I've heard of people having multiple $200/month subscriptions, so I guess that could work. Definitely seems like a glimpse into the future of what these services will truly cost once people are hooked on them.
I think this gestures at a more general point - we're still focusing on how to integrate LLMs into existing dev tooling paradigms. We squeeze LLMs into IDEs for human dev ergonomics but we should start thinking about LLM dev ergonomics - what idioms and design patterns make software development easiest for AIs?
On a related note, this type of reasoning is what made me flip my opinion on microservices. I've generally been skeptical of a many-microservice architecture for the last decade but LLMs change that - a small microservice is more likely to fit in a context window. I think this gestures at a more general point - we're still focusing on how to integrate LLMs into existing dev tooling paradigms. We squeeze LLMs into IDE…
On a related note, this type of reasoning is what made me flip my opinion on microservices. I've generally been skeptical of a many-microservice architecture for the last decade but LLMs change that - a small microservice is more likely to fit in a context window. I think this gestures at a more general point - we're still focusing on how to integrate LLMs into existing dev tooling paradigms. We squeeze LLMs into IDE…
Earlier quoted context omitted.
> What’s the going rate for tokens in terms of dollars? It depends on the provider/model, usually pricing is calculated as $/million tokens with input/output tokens having different per token pricing (output tends to be more expensive than input). Some models also charge more per token if the context size is above a threshold. Cached operations may also reduce the price per token. OpenRouter has a good overview over…
> Ime, most companies rather buy a subscription than give their developers API keys (as it makes spending predictable). The downside with subscriptions is that your work with the LLM will grind to a halt for a number of hours if you hit the token limit. I was doing what I consider very trivial work adding Javadoc comments to a few dozen files using Claude Sonnet on the $20 plan and within 30 minutes had been told to…
So Claude can understand the codebase, it needs to document it. Makes sense and is also great for humans because now there is uptodate docu on the codebase.
I don’t know how much it cost but the codebase, I’m told, is around 2 to 3 million lines of code.
Maybe it’s useful to dig out the concept of modularization with a distinction between interface and implementation again, and construct agents that are able to make effective use of it. In the case that interfaces remain unchanged, agents only need to look at the implementation of a single module at a time plus the interfaces it consumes and implements. And when changing interfaces, agents only need to look at the in…
It is somewhat ironic that coding agents are notorious for generating much more code than necesary!
Maybe it’s useful to dig out the concept of modularization with a distinction between interface and implementation again, and construct agents that are able to make effective use of it. In the case that interfaces remain unchanged, agents only need to look at the implementation of a single module at a time plus the interfaces it consumes and implements. And when changing interfaces, agents only need to look at the in…
functional programming get recked, OOP is back, baby!
With even more suitable for LLM types and "contracts at the edges".