May your contexts always be short
Before I read some of the study, I thought that was relevant too, but each "step [was] conducted as an independent, single-turn session."
LLMs corrupt your documents when you delegate
121–130 of 235 posts
Re: LLMs corrupt your documents when you delegate
#122Earlier quoted context omitted.
I've definitely experienced this while coding with LLMs. Often, after a flurry of feature work in which I thought I was being reasonably careful but moving very fast, I take a closer look at some small piece of code and go "holy shit". Then I have to spend a few hours going over everything and carefully reworking parts where things didn't quite go how I'd like, where I was unclear, or where the LLM's brainworms kicke…
This seems closely related to the problem of model collapse [1][2][3], where LLMs lose the tails of the distribution, and so when you recursively train on the output of an LLM, or otherwise feed the output back into the input in subsequent stages, you lose the precision and diversity that human authors bring to the work. Eventually everything regresses to the mean and anything that would've made the content unique, u…
Yep humans and civilization are subject to the same model-collapse phenomenon as they interact more with LLMs, but engineering knowledge has always been held by a small minority with certain personality characteristics. Maybe the minority will get smaller but I'm not sure it will completely disappear. There's always people like yourself building archives.
Re: LLMs corrupt your documents when you delegate
#123This experiment needs to be put in perspective. Let me explain. IF you did this SAME experiment with a human and had a human read an ENTIRE document and then reproduce said document with edits. The DOCUMENT would DEGRADE even more. The way this experiment is conducted is not inline with how current agentic AI is used OR how even humans edit documents. Here's how agentic AI currently typically do edits: 1. They read t…
>IF you did this SAME experiment with a human and had a human read an ENTIRE document and then reproduce said document with edits. The DOCUMENT would DEGRADE even more.
I like the idea that imagining somebody doing something in a way that nobody does it because it makes no sense for a person to do it like that is helpful here. It is like
IF you made a human eat an ENTIRE IHOP™ Chicken Fajita Omelette in one bite they would CHOKE and the OMELETTE would go UNDIGESTED. It would get everywhere and the OMELETTE would be RUINED.
Re: LLMs corrupt your documents when you delegate
#124Earlier quoted context omitted.
People love to interpret the results in the most negative way possible because it's a threat to their occupation and identity. I refer to HN specifically. The fact of the matter is, if you want to edit a document by reading the document and then regurgitating the entire document with said edits... a human will DO worse then a 25% degradation. It's possible for a human to achieve 0% degradation but the human will have…
> People love to interpret the results in the most negative way possible because it's a threat to their occupation and identity. OR it could be because their concerns are genuine but are ignored in favour of a good sounding story.
But no one in this thread addressed the inaccuracy of the experiment. The experiment did not test the actuality of HOW LLMs are used in reality.
So that is definitively a biased interpretation. This is independent of how accurate my POV or your POV is on whether LLMs degrade documents. I am simply saying the experiment conducted is COMPLETELY DIFFERENT from how LLMs AND humans edit papers.
Re: LLMs corrupt your documents when you delegate
#125Earlier quoted context omitted.
People love to interpret the results in the most negative way possible because it's a threat to their occupation and identity. I refer to HN specifically. The fact of the matter is, if you want to edit a document by reading the document and then regurgitating the entire document with said edits... a human will DO worse then a 25% degradation. It's possible for a human to achieve 0% degradation but the human will have…
> a human will DO worse then a 25% degradation As I was reading this article, a similar thought occurred to me: "I wonder if that's better or worse than a human?" Unfortunately, there was no human baseline in this study. That said, there are studies that compare LLM to human performance. Usually, humans perform much better (like 5-7x better) at long-running tasks. In other words, a human would probably do better than…
You can do a baseline study right now. Read this entire thread and make an edit of changing every E to an I.
Show your edit by regurgitating this entire thread by hand on a paper. Don't use any additional tools like Find and replace.
Boom there's your baseline. I can simulate the result in my head.
Guys I'm basically saying the experiment is innaccurate to the practical reality of how LLMs are actually used.
Re: LLMs corrupt your documents when you delegate
#126This experiment needs to be put in perspective. Let me explain. IF you did this SAME experiment with a human and had a human read an ENTIRE document and then reproduce said document with edits. The DOCUMENT would DEGRADE even more. The way this experiment is conducted is not inline with how current agentic AI is used OR how even humans edit documents. Here's how agentic AI currently typically do edits: 1. They read t…
>IF you did this SAME experiment with a human and had a human read an ENTIRE document and then reproduce said document with edits. The DOCUMENT would DEGRADE even more. I like the idea that imagining somebody doing something in a way that nobody does it because it makes no sense for a person to do it like that is helpful here. It is like IF you made a human eat an ENTIRE IHOP™ Chicken Fajita Omelette in one bite they…
That's the point bro. I am saying this Experiment makes no sense.
Humans don't do that. And Claude doesn't edit documents like that. Because it makes no sense. The point is saying that the Experiment itself is not helpful here.
Re: LLMs corrupt your documents when you delegate
#127This experiment needs to be put in perspective. Let me explain. IF you did this SAME experiment with a human and had a human read an ENTIRE document and then reproduce said document with edits. The DOCUMENT would DEGRADE even more. The way this experiment is conducted is not inline with how current agentic AI is used OR how even humans edit documents. Here's how agentic AI currently typically do edits: 1. They read t…
>IF you did this SAME experiment with a human and had a human read an ENTIRE document and then reproduce said document with edits. The DOCUMENT would DEGRADE even more. Except that isn't how humans edit documents, and it isn't how LLMs work either. When a human edits a document, they don't typically "reproduce said document with edits", which I assume you mean read the document and reproduce it from memory. They have…
>Except that isn't how humans edit documents,
Bro. That's my point.
>and it isn't how LLMs work either.
This is also my point. To be more technical about it, the harness around the LLM pushes it to do surgical edits rather then regurgitation, so my point is this experiment is garbage and testing an impractical and rarely used use case.
>When a human edits a document, they don't typically "reproduce said document with edits", which I assume you mean read the document and reproduce it from memory.
No shit sherlock. The point of that sentence was to illustrate the absurdity of doing that which in turn illustrates the absurdity of this scientific paper. You're kind of lost.
Re: LLMs corrupt your documents when you delegate
#128I'm suspicious of their results with regards to tool usage. It's unsurprising that round-tripping long content through an LLM results in corruption. Frequent LLM users already know not to do that. They claim that tool use didn't help, which surprised me... but they also said: > To test this, we implemented a basic agentic harness (Yao et al., 2022) with file reading, writing, and code execution tools (Appendix M). We…
I agree with most of what you wrote except for this: >Frequent LLM users already know not to do that. And I think that’s the biggest problem. Amidst the current push to utilize LLMs across orgs and groups there are a large (if even say majority) of people that are using them every day but who have never approached anything as technical as a “harness” before let alone an entire setup. For them the behavior mentioned h…
Exactly - I am a lawyer and we are told to use dedicated AI products as much and however we want. There will be errors made
Re: LLMs corrupt your documents when you delegate
#129this is literally just “leave a child at the work computer with a real doc open playing office”. otoh it is good to design benchmarks tonground these things. on the flip side if you’re literally just using a bare bones harness on top of a stochastic parrot, of course stochastic errors accumulate. theres a lot of ways for improving text faithfulness through harness tool designs, and my incremental experiments seem pro…
It’s not a stochastic parrot.
It’s a stochastic goblin.
Re: LLMs corrupt your documents when you delegate
#130Earlier quoted context omitted.
>IF you did this SAME experiment with a human and had a human read an ENTIRE document and then reproduce said document with edits. The DOCUMENT would DEGRADE even more. I like the idea that imagining somebody doing something in a way that nobody does it because it makes no sense for a person to do it like that is helpful here. It is like IF you made a human eat an ENTIRE IHOP™ Chicken Fajita Omelette in one bite they…
That's the point bro. I am saying this Experiment makes no sense. Humans don't do that. And Claude doesn't edit documents like that. Because it makes no sense. The point is saying that the Experiment itself is not helpful here.
It is, in fact, pretty common for people to throw a document at a language model along with a “make it more gooder” prompt.