I am surprised that not more people talk about this, I once had an ssh key deleted, so unexpected it took me a while to debug.
We live and learn.
Still a huge fan though.
LLMs corrupt your documents when you delegate
171–180 of 235 posts
Re: LLMs corrupt your documents when you delegate
#172Earlier quoted context omitted.
This seems closely related to the problem of model collapse [1][2][3], where LLMs lose the tails of the distribution, and so when you recursively train on the output of an LLM, or otherwise feed the output back into the input in subsequent stages, you lose the precision and diversity that human authors bring to the work. Eventually everything regresses to the mean and anything that would've made the content unique, u…
Yep humans and civilization are subject to the same model-collapse phenomenon as they interact more with LLMs, but engineering knowledge has always been held by a small minority with certain personality characteristics. Maybe the minority will get smaller but I'm not sure it will completely disappear. There's always people like yourself building archives.
See A Canticle for Leibowitz
Re: LLMs corrupt your documents when you delegate
#173[flagged]
Re: LLMs corrupt your documents when you delegate
#174I'm suspicious of their results with regards to tool usage. It's unsurprising that round-tripping long content through an LLM results in corruption. Frequent LLM users already know not to do that. They claim that tool use didn't help, which surprised me... but they also said: > To test this, we implemented a basic agentic harness (Yao et al., 2022) with file reading, writing, and code execution tools (Appendix M). We…
It's worth noting that Claude Code itself doesn't use the `insert` tool. (It also uses custom edit tool not the suite's predefined str_replace) Also as a person developing agentic code tools since before Claude Code, I'm skeptical if str_replace provides accuracy improvement over just full rewrite. Back in the day when SOTA models would do lazy coding like `// ... rest of the code ...`, full rewrite wasn't easy. Sear…
I've tested this extensively in a workflow (not agentic) context, and you're right, the underlying models are both good at full rewrite of code files, and at doing search/replace.
They've been decent at full rewrite for 2 years. I don't think they were good at search/replace until a year ago, but I'm not so sure.
It's true that the models 2 years ago would sometimes make errors in whole rewrite - e.g removing comments was fairly common. But I've never seen one randomly remove one character or anything like that. These days they're really good.
Main reason agentic harnesses use search/replace is speed and cost, surely! Whole file output is expensive for small changes.
Re: LLMs corrupt your documents when you delegate
#175Earlier quoted context omitted.
This will change too man. Maybe I am in a bubble but with how fast things are changing, it won’t be too long before the bubble becomes reality. Either way we should be doing experiments on the actual capabilities of AI not about the stupidest possible way to use AI because it helps validate your own negative bias against AI. Additionally as software engineers using agentic AI… which HN basically is… this experiment i…
> This will change too man. Maybe I am in a bubble but with how fast things are changing, it won’t be too long before the bubble becomes reality. You can’t get mad at an experiment for not happening in the future. > Either way we should be doing experiments on the actual capabilities of AI They simulated common end user behavior >because it helps validate your own negative bias against AI. We’ve gone from “this study…
> You can’t get mad at an experiment for not happening in the future.
I’m more getting mad at this sentence not making any sense. I’m disappointed at this experiment for not testing the actual capabilities of an LLM. Comprende?
> They simulated common end user behavior
Not the way you use it. And not the way it will be used.
You love it because you want it to stay this way so you can forever believe AI will never be better than you.
Bro the reality is unfolding as you speak. It’s like humanity just discovered guns but hasn’t discovered the bullets and your saying guns are useless because most of humanity hasn’t figured out bullets yet.
> We’ve gone from “this study is flawed because language models don’t do that” to “this study is flawed because while language models do do that, I don’t think that they will in the future” to “data that could support a bias other than my own is bad”
This is a flat out lie. Models DO do that. The only fucking argument you have is that non technical and average laymen people edit documents the wrong way while all people who use agentic AI as adepts use it the correct way. Like are you fucking kidding me?
The only change I acknowledge is your grandma copies and pastes essays into ChatGPT while YOU don’t. You go pretend you live in that reality where the bullets will never appear.
Re: LLMs corrupt your documents when you delegate
#176You can get around the problem by doing a git diff of the unstaged file and a previous commit.
This works well for code regressions but also works for document writing. I've automated it at this point.
A case where using the CLI agent is much better than using the web chat.
Re: LLMs corrupt your documents when you delegate
#177Earlier quoted context omitted.
That was true maybe 7 months ago. This is no longer the case. Harnesses use all kinds of tooling to edit things now.
People paste entire documents into gemini and chat gpt’s text boxes on the web and assume it will all turn out great edit: apparently got beaten to this
I don’t understand you. We have an AI model. The AI model is obviously capable.
But you want to use pretend that it’s not useful because non technical people haven’t figured out how to properly use it yet?
Do you think that’s a valid argument? This article is making a claim of 25 percent degredation. Do you think that claim is true because a lot of people don’t use it right?
Humans have 99 percent degredation when editing one punctuation point of an entire book when regurgitating that entire book just to change one punctuation point. Does this statement sound reasonable to you? Because that is the statement you and your genius interloper into this thread are standing behind. Just replace human with LLM and it’s the same kind of genius logic.
Re: LLMs corrupt your documents when you delegate
#178Earlier quoted context omitted.
A model solving original math problems may look like human reasoning, but internally the model is choosing the next token based on what it has learned about probability around various patterns and structures. The model knows about correlations between problems, proof techniques and answer structures, and when it "reasons" it's selecting a high probability trajectory through that learned knowledge. A calculator is dif…
[flagged]
You should leave this site. Comments like this are not good for this site. You should go somewhere else.
Re: LLMs corrupt your documents when you delegate
#179Earlier quoted context omitted.
I agree with most of what you wrote except for this: >Frequent LLM users already know not to do that. And I think that’s the biggest problem. Amidst the current push to utilize LLMs across orgs and groups there are a large (if even say majority) of people that are using them every day but who have never approached anything as technical as a “harness” before let alone an entire setup. For them the behavior mentioned h…
Exactly. When I use a scissor, I don't want the scissor to not work just because I'm not a "frequent scissor user," and then get told by someone who makes their breakfast with scissors that I'm doing it wrong. Most people will not be "frequent" anything users.
Most people also understand that, because they're not "frequent" users of a thing, they absolutely suck at using it, and set their expectations accordingly. In particular, they realize that doing anything non-trivial with the thing requires them to spend some learning and practice time, or asking/hiring a "frequent" user to do it for them.
So the reasonable response to being told you're holding your scissors wrong is to realize that yes, you most likely are holding your scissors wrong[0], and ask the other person for advice (or just to do the thing), or look up a YouTube video and learn, or sign up to a class, or such.
Expecting mastery in 30 seconds is not a reasonable attitude, but it's unfortunately the lie that software industry tried to sell to people for the past 15 years or so.
--
[0] - There's much more to it than one would think.
Re: LLMs corrupt your documents when you delegate
#180[flagged]