I care more for my refined spec than for the code. I refine the spec over multiple chats. Once it's fully refined and ready to be executed, the phased task itself is small enough that it will easily be done in 100K tokens.
Don't trust large context windows
191–200 of 211 posts
Re: Don't trust large context windows
#192Just because you have a large garage doesn't mean you can park your car easily.
Re: Don't trust large context windows
#193Earlier quoted context omitted.
For what it’s worth, as a third party I read your and qsera’s comments as saying the same thing.
Maybe I misread the comment then. I read it as a models performance being random and observed differences in the opinions are the results of the overinterpretation of the random outcomes. I think however that some people seem to be always lucky which indicates that it is not random but rather some fixed differences between people and their environments.
A models adherence to some configuration is a matter of probability. There might be some underlying pattern, but as far as I understand this is not documented and it may be even impossible to do so. So people are just trying stuff and sharing what appear to work. There is no causal link anywhere in this recommendations, and is just based on spurious correlations.
Re: Don't trust large context windows
#194Earlier quoted context omitted.
The tokens are still being burnt, they're just doing so in a parallel dimension from the users main context window.
It's true that the initial tool response still has the same amount of tokens but it doesn't keep dragged along in the longer-lived top context.
Don't you resend after every turn, so splitting it avoids the n^2 token usage (granted it's cached so there's some optimal amount here)
Re: Don't trust large context windows
#195Earlier quoted context omitted.
I see this said often and find it insane given how many times I find opus models making basic recall mistakes at Personally I consider < 60k to be the smart zone for opus. This is worse for opus 4.7 and 4.8 cause of the more granular tokenizer
>making basic recall mistakes at I usually see this when the context gets "tainted" as I call it. The model gets stuck on a bad path and there's no way to bring it back without clearing the context and starting again. Frequently it'll be something as small as 1 sentence of a prompt many messages ago. When cases like that happen, I reset the context and try to be explicit about assumptions and requirements to keep it…
They appear to lose recall almost immediately if you aren’t highly disciplined, yourself, and permit deviations towards trained behavior.
Overreacting seems to put them back on track, but they’ll “forget” again pretty quickly.
It really depends more on the thing you’re expecting it to “remember” and distance from the last wrist slap.
Re: Don't trust large context windows
#196Earlier quoted context omitted.
It's true that the initial tool response still has the same amount of tokens but it doesn't keep dragged along in the longer-lived top context.
Don't you resend after every turn, so splitting it avoids the n^2 token usage (granted it's cached so there's some optimal amount here)
Yes, exactly. You resend it on every turn (assuming no cache hits). This is why using the shorter-lived subagent to take in that context and only return the useful result back to the longer-lived context safes tokens.
Re: Don't trust large context windows
#197Earlier quoted context omitted.
The arbitrary and non-deterministic nature of LLM workflows gives me full on ick. As an old embedded/systems guy I have always prioritized determinism and repeatability in my workflows. But damn, agents are amazing and I'm enjoying being a "thought process designer". I'm not going back. Even if AI development stops today my career will never be the same.
I felt the same way about the non-determinism but realized it can be really beneficial to have a machine that can fairly reliably turn non-determinism into determinism. I’m working on a tiny agent harness at home to learn and the process of taking human speech and turning it into agent tool calls that output something generally deterministic depending on how the tool is defined is so interesting. One of the big takea…
it has a random seed cant you just set it
Re: Don't trust large context windows
#198I guess I am mostly enjoying learning the fundamentals of AI stuff, even though I disagree with the direction it is going. But I am struggling to put into words how alarming I find the comments on threads like this — all sorts of good-natured anecdotes about how XYZ works for them that are more like the suggestions in pet care or cookery threads on Facebook. (Or worse still, like any Facebook 3D printing group: anyon…
Early days. We can build abstractions that provide deterministic behavior atop seemingly nondeterministic systems. Insodoing you forgo better models, since a better model comes out every few months these tools are quickly forgotten.
Re: Don't trust large context windows
#199Earlier quoted context omitted.
I felt the same way about the non-determinism but realized it can be really beneficial to have a machine that can fairly reliably turn non-determinism into determinism. I’m working on a tiny agent harness at home to learn and the process of taking human speech and turning it into agent tool calls that output something generally deterministic depending on how the tool is defined is so interesting. One of the big takea…
it has a random seed cant you just set it
Wouldn't change much.
Would just make the answer to the same exact prompt X repeatedly the same.
It wouldn't change the fact that prompt X', functionaly indistinguisable from X, aside from small phrasing changes, can give a totally different looking answer.
Re: Don't trust large context windows
#200I guess I am mostly enjoying learning the fundamentals of AI stuff, even though I disagree with the direction it is going. But I am struggling to put into words how alarming I find the comments on threads like this — all sorts of good-natured anecdotes about how XYZ works for them that are more like the suggestions in pet care or cookery threads on Facebook. (Or worse still, like any Facebook 3D printing group: anyon…
> Any shared sense of rigour is just completely torpedoed by the LLM world Consider that this shared sense of rigour you have in mind is illusory, and LLMs and their context struggles are simply revealing this. I see precious little rigour in any of the 'tech' world I've lived in for decades. The tools proliferate, paradigms emerge and die and reemerge, and whatever stick you consider using to measure any of it has c…
>Consider that this shared sense of rigour you have in mind is illusory
Just because it was nowhere near perfect, and a lot of "religion" and hand waving was involved, doesn't mean it was illusory.
It's enough that it existed quite more than it does in the LLM era for what the parent said to make sense.