Yeah I feel like the longer it is, the more likely they are to just ignore it. With LLMs shorter is better. And nothing is guaranteed lol.
Handbook.md shows that long policy documents do not reliably govern agents
201–210 of 237 posts
Re: Handbook.md shows that long policy documents do not reliably govern agents
#202[deleted]
Re: Handbook.md shows that long policy documents do not reliably govern agents
#203Earlier quoted context omitted.
Why 25% and not 12% or 40%? Is this an arbitrary vibestimate or you had some tests done that pointed you to 25%? I am genuinely interested in how others deal with context issues. Also I would think the usable context window is variable depending on the task, for example summarizing documents vs analyzing large, scattered and complex instructions.
250k because thats what the smaller models supported and theres better training data in that part of the window there have been some papers suggesting that the useful context is even smaller, and stays fixed as you change the context window size. as a more general case though, i think the possibilities for what youd need to include in training to have many different paths of text be well represented enough over the l…
>in that part of the window
What matters is the relative distance. Using the latter 250k should be equivalent to using the first 250k.
Re: Handbook.md shows that long policy documents do not reliably govern agents
#204Earlier quoted context omitted.
The challenge with comparing these things to humans, is that humans learn. A newbie might not respect your organization’s set of policies on day one, but what about 3 months in? Or 3 years? Meanwhile there’s still no reasonable mechanism for automatically fine tuning LLMs or adjusting their harnesses to make them better at completing your organization’s objectives more successfully. They’re still overwhelmingly gover…
Models learn. It just costs $10B and 1 year to do what a human does every night.
This is not about learning only.
It is about context window too so no matter how much money you put to train your model it can still have degraded performance in following instructions but it will be better to execute the instructions they can follow.
It is also about the harness and how that can help drive the model and pick and choose what to include or not in the context.
These LLMs do not understand the project. For them any next word is good as long as it was picked by the token predictor. It does not have any way to understand but only to do. probability distribution over their vocabulary and if that vocabulary is tainted and lost parts of the original context what is a good candidate there will not match the intention of the initial project.
Re: Handbook.md shows that long policy documents do not reliably govern agents
#205[flagged]
Re: Handbook.md shows that long policy documents do not reliably govern agents
#206[flagged]
Re: Handbook.md shows that long policy documents do not reliably govern agents
#207Earlier quoted context omitted.
Most people jump straight to agents when what they actually need is a graph. Example: a mining company receives free-text reports from field geologists. You could have: Geologist report -> LLM call extracts minerals we are looking for (you inject a db query result on the user prompt), locations, assay mentions and risks into structured fields -> LLM call classifies evidence into positive indicators, negative indicato…
I would call that a pipeline, which may be a type of graph, but might have communicated your point more concretely to begin with
I called it a graph because really its a DAG (Directed Acyclic Graph) , in my example I kept it simple so there was only one input to every node, but in more advanced use cases you might have a synthesis node that takes in 2 or more inputs making it more akin to an execution graph in Tensorflow for example
Re: Handbook.md shows that long policy documents do not reliably govern agents
#208Earlier quoted context omitted.
This is a bit of a strawman. The parent comment wasn't suggesting that the inherent defects magically disappear nor was it suggesting that local inference today is sufficient for all tasks. At the heart of it, self-hosting liberates your use cases from all the horribly opaque configuration, shadow prompting, etc. And local models are only getting better and more diverse every month.
> This is a bit of a strawman. The parent comment wasn't suggesting that the inherent defects magically disappear The parent comment literally said that the common LLM defects would go away. Direct quote: > all of the common LLM defects will go away
Clearly it was meant that they will go away only if you use a custom harness, without the defects that now exist in all vendor-controlled harnesses, because of the non-aligned goals of the vendor and of the user.
While it was not spelled explicitly, it should be obvious that nothing can be improved if you run locally an LLM in the same way as the vendor. But you should not do that. It is easy to use better methods for sampling procedures and for prompt and context handling.
Re: Handbook.md shows that long policy documents do not reliably govern agents
#209Earlier quoted context omitted.
14 seconds per token? Not tokens per second. Seconds per token? That’s nowhere near the cusp!
At that rate, it would take me a mere 15 days to generate the number of tokens I typically use in a day. "Yes, I'm using AI to speed up development. I'll submit that PR in two weeks time!"
That speed is significantly lower that what can be attained with streaming from SSDs.
Not only in big desktops, but even in most recent mini-PCs, it is possible to read concurrently from one PCIe 5.0 SSDs and one PCIe 4.0 SSD, at a total sustained reading throughput of around 20 GB/s.
With an optimized inference implementation, it should be possible to overlap completely the computations with streaming weights from SSDs.
This should improve the inference speed to around at least 1 token per second, on a cheap computer, under $2000 even at the current super-inflated prices.
There are enough tasks where this would be useful. Obviously one should use for most tasks a fast small LLM and use the big one only when this actually saves time.
Re: Handbook.md shows that long policy documents do not reliably govern agents
#210Earlier quoted context omitted.
Few pointers: Dont try and handle the entire project in context. Use a well structured filesystem layout for your code with a few lines in an AGENTS.md describing the layout and core architectural requirements (no more than that, as per the article!). Then work on small-medium tasks at a time with a fresh context. At the end of your task, ask the agent if there are any key points about the project layout or architect…
Good points, but it still seems to me that the technology is far from mature.
Of course - its only a few years old :) But I think the biggest problem is that no-one directly tells you how to use the technology effectively. Mostly because no-one really knows, hence the reason for all the competing harnesses, agents, etc - the models are all just intelligence on top of that tooling.