161–170 of 172 posts
I make a point of never going beyond about 220k, unless absolutely necessary (and it's almost never necessary), anyway, even with models that degrade more slowly, so this is just a discount.
Earlier quoted context omitted.
I had Claude build me a Python-inspired .NET language that treats .NET as a first class citizen, and breaks backwards compatibility where some Python nuances don't really apply to .NET for. I was able to get it to build a sample ASP .NET Web application that ran on Culebral code. Haven't gone back to it, have been using Claude Code on a private project I'm still architecting. https://github.com/Giancarlos/Culebral
That is pretty interesting. It compiles to the CLR?
Earlier quoted context omitted.
Models work best when they have short instructions and no noise. "Conversation [may] contain" also means "conversation has a lot of noise". It degrades performance and increases cost. I use handoff skill to ask model to write a prompt for itself.
The applicability of your advice is very model dependent. Some like Claude have very good long context performance, whereas others they fall off much quicker past some threshold.
I've found Opus 5 far better as a subagent with very limited context window use, which could suggest that it might not have good long context performance unlike its predecessors. (I was one of many tearing my hair out trying to work with Opus 5 for the past week.)
Earlier quoted context omitted.
Needing large context windows is an illusion.
So, all types of knowledge work and coding require the same amount of context? This seems like a hot take not really informed by experience.
Please read my essay on context window saturation. I am also the author of deepcq (intent-aware memory querying & context mgmt for LLM agents)
Earlier quoted context omitted.
That is pretty interesting. It compiles to the CLR?
Yeah, that was the goal. I've wanted something like Python for .NET as someone who does both, but backwards compatibility to Pythons ecosystem is something that will always make it harder than just having something Python-like as much as possible, but runs on .NET the other thing I wound up going for Rusts approach to type hints, method signatures should always have type hints, and when instantiating objects or types…
Earlier quoted context omitted.
So, all types of knowledge work and coding require the same amount of context? This seems like a hot take not really informed by experience.
This is a discussion on context window and not context. Please read my essay on context window saturation. I am also the author of deepcq (intent-aware memory querying & context mgmt for LLM agents)
Earlier quoted context omitted.
AI infra buildup is so massive that the frontier labs should be able to offer more than one level of context length to incentivize token thriftiness. One would think compute-constrained actors like Anthropic would have done so, unless prefill isn’t really a bottleneck compared to decode?
There's the infra cost of having multiple SKUs. If you create 3 buckets of inference pods, say, 256k, 512k, and 1M, then you have to worry about filling/dynamically-scaling all of them. And my guess is there's probably not a huge amount of customers that want somewhere in between: if you're willing to pay the long context surcharge; you're probably semi-price-insensitive anyway to just use 1M.
That seems like it is not an awfully hard problem, because the AI buildout scale is so massive. You simply don't provision too many of the smaller buckets, and when they are fully loaded you route the traffic to the larger buckets. Now you don't have a utilisation problem.
Now of course you can simply organize it that way internally, and present a single price to the customer. Perhaps thats the smarter thing to do.
Codex uses 256k masterfully, 1M is luxurious but still quite expensive and seems not necessary as a default.
I'll have to give it a shot some time soon. I just can't imagine working on any of my serious projects doing a new large feature with that little room. By the time it has looked at half the code required to start planning the work, it'd be out of context. Must use sub agents better or something.
Earlier quoted context omitted.
64k ought to be enough for anybody
Why stop there? No true developer needs more than 32k! -- Jesting aside, I cannot think of a single session in recent memory that used less than ~70k tokens, so I presume you are joking.
I've played with explorer agents giving exploration summaries to help the implementer agents use more of their context for implementation, but it doesn't work as well. There's always something lost in the handoff.
Hopefully this helps reduce some of the pressure on their infrastructure. Their models have all become super dumb recently and their support are not addressing it. I have a hunch they’ve been serving a significant percentage of requests with quantised models.
Ah come on, HN is not the place for these types of unfounded conspiracy theories that keep popping up on Reddit.