[flagged]
Don't trust large context windows
121–130 of 211 posts
Re: Don't trust large context windows
#122Earlier quoted context omitted.
This is interesting to me because reducing context & token usage is in the user's best interest but not in the financial interest of AI vendors. I am not an expert but it sounds like your "one simple trick" would fix context issues and allow much tighter control over token usage. Thanks for being willing to share this tip in an HN comment, changing how those in the know use AI agents going forward -- it's hard to kee…
The tokens are still being burnt, they're just doing so in a parallel dimension from the users main context window.
It's true that the initial tool response still has the same amount of tokens but it doesn't keep dragged along in the longer-lived top context.
Re: Don't trust large context windows
#123I guess I am mostly enjoying learning the fundamentals of AI stuff, even though I disagree with the direction it is going.
But I am struggling to put into words how alarming I find the comments on threads like this — all sorts of good-natured anecdotes about how XYZ works for them that are more like the suggestions in pet care or cookery threads on Facebook.
(Or worse still, like any Facebook 3D printing group: anyone who prints but wants to understand what is actually going on will know what I mean, I think)
Any shared sense of rigour is just completely torpedoed by the LLM world, particularly the cloud LLM world it seems, and we are reduced to cargo culting. Nobody is any more right or wrong than anyone else.
Have you tried cleaning your context with dawn dish soap, letting it dry and then adding a layer of glue stick?
--
ETA: I don't want to sound so mean about people who try to help, here or in facebook groups. I guess I just find these threads so different to threads on more or less any other topic, where someone's suggestion can be debated or refined by other commenters and then someone will explain a thing about how bash history selections work that will change your entire life. With these threads they devolve to "isn't it weird that threatening it works?"
Re: Don't trust large context windows
#124The problem with "context rot" is that its existence and severity is purely anecdotal. As far as I know, nobody has actually measured context rot systematically. The only thing we know is that memory degrades somewhat in long contexts, via things like needle in haystack tests. But that's not the same issue. Context rot is usually taken to mean that the model gets dumber even if it doesn't need to remember specific things in its context window.
This would be really easy to measure. Just take some standard benchmarks, but fill up the context beforehand. Is the benchmark performance degraded? If so, by how much?
Re: Don't trust large context windows
#125I guess I am mostly enjoying learning the fundamentals of AI stuff, even though I disagree with the direction it is going. But I am struggling to put into words how alarming I find the comments on threads like this — all sorts of good-natured anecdotes about how XYZ works for them that are more like the suggestions in pet care or cookery threads on Facebook. (Or worse still, like any Facebook 3D printing group: anyon…
> But I am struggling to put into words how alarming I find the comments on threads like this — all sorts of good-natured anecdotes about how XYZ works for them that are more like the suggestions in pet care or cookery threads on Facebook.
It will always be this way going forward. Everyone thinks differently about problems. In the past we had experts and only they could do the work at a high level. But now we have many people that are cranking out expert level solutions without much knowledge. Worrying about the minutia is a dying trend.
Edit: I see I touched a nerve. But that is how it is now. You can't fight reality.
Re: Don't trust large context windows
#126I've been able to avoid context size issues by applying one simple constraint to my agent loop. What I do is prevent all tool calling in the user's top-level conversation thread. Anything that needs to tool call must happen in a recursive invoke of the agent, which returns whatever results to caller. I can keep the same high level conversation going for an entire day over a million LOC+ codebase without ever hitting…
This makes intuitive sense. Can I ask what harness you're using that allows you to configure the constraint and how?
[deleted]
Re: Don't trust large context windows
#127Considerations about what goes on in agents internally will probably not be part of software development for long. Personally, I already see LLMs and agents as blackboxes. I give each feature request to multiple LLMs and then compare the results. I don't manually use "sessions" at all. I just look at the outcome. When I dislike it, I "git reset --hard", change my prompts and restart the feature request. To have an on…
This is an absolutely crazy wasteful thing to do considering the actual cost of all that inference and nothing to be proud of.
The cost is nothing compared to the outcome and time savings. What I see is that people with no money want to jump into this pool but they aren't having a good time. That is generally the case when you are poor.
Re: Don't trust large context windows
#128I guess I am mostly enjoying learning the fundamentals of AI stuff, even though I disagree with the direction it is going. But I am struggling to put into words how alarming I find the comments on threads like this — all sorts of good-natured anecdotes about how XYZ works for them that are more like the suggestions in pet care or cookery threads on Facebook. (Or worse still, like any Facebook 3D printing group: anyon…
This has always been a thing with IT advice, though - the more complex a system and the outcome, the harder it is to clearly define "better" or "worse". Add in the fact that LLMs are intensely and emphatically non-deterministic and LLM guidance basically becomes gardening advice.
Heck, even the 'benchmarks' are mostly somebody's attempt to crystallize their vibes with varying amounts of success.
Re: Don't trust large context windows
#129I've been able to avoid context size issues by applying one simple constraint to my agent loop. What I do is prevent all tool calling in the user's top-level conversation thread. Anything that needs to tool call must happen in a recursive invoke of the agent, which returns whatever results to caller. I can keep the same high level conversation going for an entire day over a million LOC+ codebase without ever hitting…
This is interesting to me because reducing context & token usage is in the user's best interest but not in the financial interest of AI vendors. I am not an expert but it sounds like your "one simple trick" would fix context issues and allow much tighter control over token usage. Thanks for being willing to share this tip in an HN comment, changing how those in the know use AI agents going forward -- it's hard to kee…
> This is interesting to me because reducing context & token usage is in the user's best interest but not in the financial interest of AI vendors.
AI vendors still need to compete with each other both in terms of token cost and competency. An agent that is costly and less effective by wasting tokens is less competitive.
Re: Don't trust large context windows
#130I guess I am mostly enjoying learning the fundamentals of AI stuff, even though I disagree with the direction it is going. But I am struggling to put into words how alarming I find the comments on threads like this — all sorts of good-natured anecdotes about how XYZ works for them that are more like the suggestions in pet care or cookery threads on Facebook. (Or worse still, like any Facebook 3D printing group: anyon…
> But I am struggling to put into words how alarming I find the comments on threads like this — all sorts of good-natured anecdotes about how XYZ works for them that are more like the suggestions in pet care or cookery threads on Facebook. It will always be this way going forward. Everyone thinks differently about problems. In the past we had experts and only they could do the work at a high level. But now we have ma…
At some level, we've always delegated worrying about the minutiae to someone who builds the tool that is one or two levels below.
I usually don't have to worry about compiler optimisations because compiler experts do that; sometimes they appear in a thread about code and say "compiler guy here — if you write your code like this the compiler can optimise it".
And that person will be provably right (or wrong), in that context. And it'll be the same each time you run the test!
I just… ehh. You make a good point and I worry you are not wrong. It's all so different.
I like my 3D printing analogy much more than I wish I did.