Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs corrupt your documents when you delegate

arxiv.org

Post reply on HN
Print view View original

11–20 of 235 posts

Re: LLMs corrupt your documents when you delegate

#11

Post by sebastiennight »

cyanydeez wrote: post #4

I played around with a local LLM to try and build a wiki like DAG. It made a lot of stupid errors from vague generic things like interpreting based on file names to not following redirects and placing the redirect response in them. I've also had them convert to markdown something like an excel formatted document. It worked pretty well as long as I was examining the output. But the longer it ran in context, the more l…

> I've also had them convert to markdown something like an excel formatted document.

This look like a task where the LLM would be best used in writing a deterministic script or program that then does the conversion.

Trusting a LLM to make the change without tools is like telling the smartest person you know to just recite the converted document out loud from memory. At some point they'll get distracted, wrong, or unwittingly inject their own biases and ideas into it whenever the source data is counter-intuitive to them.

Re: LLMs corrupt your documents when you delegate

#12

Post by adampunk »

polskibus wrote: post #5
causal wrote: post #3

Yeah I've been saying this for a while: AI-washing any text will degrade it, compounding with each pass. "Semantic ablation" is my favorite term for it: https://www.theregister.com/software/2026/02/16/semantic-abl...

By „with each pass” do you mean within the same session, or with new session (context window) each time?

Each edit, even with unrelated edits. I had a README referring to something as "the cathedral of s*t" (some HN commentators don't care for the swearing, which is systemically bad news but w/e) and the robot would lift that phrase out in drive-bys, repeatedly.

Occasionally it would report the action, sometimes it would not bother to report it. It never reached into the README on an unrelated doc edit, but if it was touching the README, that line was getting excised.

Re: LLMs corrupt your documents when you delegate

#13

Post by adampunk »

pyrolistical wrote: post #9
adampunk wrote: post #8

LLMs will make mistakes on every turn. The mistakes will have little to no apparent connection to "difficulty" or what may or may not be prevalent in the training data. They will be mistakes at all levels of operation, from planning to code writing to reporting. Whether those mistakes matter and whether you catch them is mostly up to you. I have yet to find a model that does not make mistakes each turn. I suspect tha…

As a human I make typos all the time

I do too! I also make higher level design errors and get too enthusiastic about projects before code is written.

We are, in a sense, fallible machines who have designed a planet-wide computational fabric around that fact.

Re: LLMs corrupt your documents when you delegate

#14

Post by timacles »

Least shocking thing I've read about LLMs recently.

They are essentially like that one JPEG meme, where each pass of saving as JPEG slightly degrades the quality until by the end its unrecognizable.

Except with LLMs, the starting point is intent. Each pass of the LLMs degrades the intent, like in the case of a precise scientific paper, just a little bit of nuance, a little bit of precision is lost with a re-wording here and there.

LLMs are mean reversion machines, the more 'outside of their training' the context/work load they are currently dealing with, the more they will tend to gradually pull that into some homogenous abstract equilibrium

Re: LLMs corrupt your documents when you delegate

#15

Post by dangus »

pyrolistical wrote: post #9
adampunk wrote: post #8

LLMs will make mistakes on every turn. The mistakes will have little to no apparent connection to "difficulty" or what may or may not be prevalent in the training data. They will be mistakes at all levels of operation, from planning to code writing to reporting. Whether those mistakes matter and whether you catch them is mostly up to you. I have yet to find a model that does not make mistakes each turn. I suspect tha…

As a human I make typos all the time

A human can sit down and say “I’m going to make sure this is correct on the first pass and make sure I make an exact copy.”

They have cognitive awareness of which tasks are highly critical and need more checking and re-checking without being prompted to think that way.

For a human, time doesn’t stop when the first pass of the prompt and response is over. An LLM effectively wipes its memory of what it just did unless something is keeping track of a highly resource constrained context.

An LLM is like an author of a book that immediately closes its eyes and wipes its memory after writing a chapter. Sure, it can pull some of that back in the next query via context, and it can regain context very quickly, but it effectively has no memory of the exact thing it just did.

When a human is doing these tasks there is a lot of room for mistakes but there’s also a wildly higher capacity for flowing through time.

Re: LLMs corrupt your documents when you delegate

#16

Post by trollbridge »

sebastiennight wrote: post #11
cyanydeez wrote: post #4

I played around with a local LLM to try and build a wiki like DAG. It made a lot of stupid errors from vague generic things like interpreting based on file names to not following redirects and placing the redirect response in them. I've also had them convert to markdown something like an excel formatted document. It worked pretty well as long as I was examining the output. But the longer it ran in context, the more l…

> I've also had them convert to markdown something like an excel formatted document. This look like a task where the LLM would be best used in writing a deterministic script or program that then does the conversion. Trusting a LLM to make the change without tools is like telling the smartest person you know to just recite the converted document out loud from memory. At some point they'll get distracted, wrong, or unw…

I see people cut and paste from Excep into a chat, as an image, and ask it to sum up numbers.

Re: LLMs corrupt your documents when you delegate

#17

Post by peyton »

adampunk wrote: post #13
pyrolistical wrote: post #9

Earlier quoted context omitted.

As a human I make typos all the time

I do too! I also make higher level design errors and get too enthusiastic about projects before code is written. We are, in a sense, fallible machines who have designed a planet-wide computational fabric around that fact.

[flagged]

Re: LLMs corrupt your documents when you delegate

#18

Post by adampunk »

dangus wrote: post #15
pyrolistical wrote: post #9

Earlier quoted context omitted.

As a human I make typos all the time

A human can sit down and say “I’m going to make sure this is correct on the first pass and make sure I make an exact copy.” They have cognitive awareness of which tasks are highly critical and need more checking and re-checking without being prompted to think that way. For a human, time doesn’t stop when the first pass of the prompt and response is over. An LLM effectively wipes its memory of what it just did unless…

[flagged]

Re: LLMs corrupt your documents when you delegate

#19

Post by cyanydeez »

sebastiennight wrote: post #11
cyanydeez wrote: post #4

I played around with a local LLM to try and build a wiki like DAG. It made a lot of stupid errors from vague generic things like interpreting based on file names to not following redirects and placing the redirect response in them. I've also had them convert to markdown something like an excel formatted document. It worked pretty well as long as I was examining the output. But the longer it ran in context, the more l…

> I've also had them convert to markdown something like an excel formatted document. This look like a task where the LLM would be best used in writing a deterministic script or program that then does the conversion. Trusting a LLM to make the change without tools is like telling the smartest person you know to just recite the converted document out loud from memory. At some point they'll get distracted, wrong, or unw…

it was, but the formatting was garbage so it ran again to fix thw format.

Re: LLMs corrupt your documents when you delegate

#20

Post by simonw »

I'm suspicious of their results with regards to tool usage.

It's unsurprising that round-tripping long content through an LLM results in corruption. Frequent LLM users already know not to do that.

They claim that tool use didn't help, which surprised me... but they also said:

> To test this, we implemented a basic agentic harness (Yao et al., 2022) with file reading, writing, and code execution tools (Appendix M). We note this is not an optimized state-of-the-art agent system; future work could explore more sophisticated harnesses.

And yeah, their basic harness consists of read_file() and write_file() - that's just round-tripping with an extra step!

The modern coding agent harnesses put a LOT of work into the design of their tools for editing files. My favorite current example of that is the Claude edit suite described here: https://platform.claude.com/docs/en/agents-and-tools/tool-us...

The str_replace and insert commands are essential for avoiding round-trip risky edits of the whole file.

They do at least provide a run_python() tool, so it's possible the better models figured out how to run string replacement using that. I'd like to see their system prompt and if it encouraged Python-based manipulation over reading and then writing the file.

Update: found that harness code here https://github.com/microsoft/delegate52/blob/main/model_agen...

The relevant prompt fragment is:

  You can approach the task in whatever
  way you find most effective:
  programmatically or directly
  by writing files
As with so many papers like this, the results of the paper reflect more on the design of the harness that the paper's authors used than on the models themselves.

I'm confident an experienced AI engineer / prompt engineer / pick your preferred title could get better results on this test by iterating on the harness itself.

Post reply on HN