Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs corrupt your documents when you delegate

arxiv.org

Post reply on HN
Print view View original

31–40 of 235 posts

Re: LLMs corrupt your documents when you delegate

#31

Post by carterschonwald »

this is literally just “leave a child at the work computer with a real doc open playing office”. otoh it is good to design benchmarks tonground these things.

on the flip side if you’re literally just using a bare bones harness on top of a stochastic parrot, of course stochastic errors accumulate.

theres a lot of ways for improving text faithfulness through harness tool designs, and my incremental experiments seem promising.

but unless work is gated on shit like “the script used must type checked ghc haskell or lean4”, unsupervised stuff is gonna decay

Re: LLMs corrupt your documents when you delegate

#32

Post by lelanthran »

threethirtytwo wrote: post #28
timacles wrote: post #14

Least shocking thing I've read about LLMs recently. They are essentially like that one JPEG meme, where each pass of saving as JPEG slightly degrades the quality until by the end its unrecognizable. Except with LLMs, the starting point is intent. Each pass of the LLMs degrades the intent, like in the case of a precise scientific paper, just a little bit of nuance, a little bit of precision is lost with a re-wording h…

A human doing the same tasks as what the LLM did in the paper that the human will degrade the document further then the LLM. If the LLM is 25%, a human would degrade it probably 80% if they used the same technique as the LLM did in this paper. I'm talking about a single pass. The fact of the matter is, humans don't edit things the way it was done in the paper and neither do coding agents like claude. Think about it:…

Are we comparing humans to LLMs or human written software to LLMs?

The whole point of creating software to do things used to be getting things done more accurately and consistently.

Re: LLMs corrupt your documents when you delegate

#33

Post by ben_w »

Twirrim wrote: post #27
timacles wrote: post #14

Least shocking thing I've read about LLMs recently. They are essentially like that one JPEG meme, where each pass of saving as JPEG slightly degrades the quality until by the end its unrecognizable. Except with LLMs, the starting point is intent. Each pass of the LLMs degrades the intent, like in the case of a precise scientific paper, just a little bit of nuance, a little bit of precision is lost with a re-wording h…

A coworker talks about LLMs as "bullshit" layers. Not exactly dismissing them or being derogatory about them, but emphasising that each time you feed something through an LLM, what comes out the other side may not be what you expect/want. Like that guy at the pub sharing what he'd seen online somewhere, after a few pints. Might be accurate, but carries notable risk it's not. So e.g., don't use an LLM to call an API t…

Similar experience. I wouldn't say it even needs to be like some random person in the local pub: this behaviour is what you'd get from any game of telephone, book authors will say how you need to be blunt and direct about points in the book because readers will miss subtlety, anyone who has been quoted in a newspaper will have a story about the paper getting it wrong, etc.

However, there's a reason pre-computing bureaucracy came with paper trails and meeting minutes getting written up, why court cases are increasingly cautious about the reliability of eye witnesses.

It is ironic, the more AI becomes like us and less it acts like a traditional computer program, the worse it is at many things we want to use it for, but because collectively we're oblivious to our cognitive limitations we race into completely avoidable failures like this.

Re: LLMs corrupt your documents when you delegate

#34

Post by twobitshifter »

I thought this was going to be about a problem we saw recently. Someone used an LLM to update the comment block at the start of each source file, and the LLM programmed its own tool that ended up changing ALL of the line endings when it output again with the corrected comment block. Instead of an LLM we could have used find and replace, but people are thinking LLM is the only tool.

Re: LLMs corrupt your documents when you delegate

#35

Post by [deleted] »

threethirtytwo wrote: post #26
simonw wrote: post #20

I'm suspicious of their results with regards to tool usage. It's unsurprising that round-tripping long content through an LLM results in corruption. Frequent LLM users already know not to do that. They claim that tool use didn't help, which surprised me... but they also said: > To test this, we implemented a basic agentic harness (Yao et al., 2022) with file reading, writing, and code execution tools (Appendix M). We…

People love to interpret the results in the most negative way possible because it's a threat to their occupation and identity. I refer to HN specifically. The fact of the matter is, if you want to edit a document by reading the document and then regurgitating the entire document with said edits... a human will DO worse then a 25% degradation. It's possible for a human to achieve 0% degradation but the human will have…

[deleted]

Re: LLMs corrupt your documents when you delegate

#36

Post by RevEng »

threethirtytwo wrote: post #28
timacles wrote: post #14

Least shocking thing I've read about LLMs recently. They are essentially like that one JPEG meme, where each pass of saving as JPEG slightly degrades the quality until by the end its unrecognizable. Except with LLMs, the starting point is intent. Each pass of the LLMs degrades the intent, like in the case of a precise scientific paper, just a little bit of nuance, a little bit of precision is lost with a re-wording h…

A human doing the same tasks as what the LLM did in the paper that the human will degrade the document further then the LLM. If the LLM is 25%, a human would degrade it probably 80% if they used the same technique as the LLM did in this paper. I'm talking about a single pass. The fact of the matter is, humans don't edit things the way it was done in the paper and neither do coding agents like claude. Think about it:…

Except that coding agents will do this at times. That's half the problem. A human will forget details and exaggerate others, but LLMs fail in spectacular ways that humans rarely would, like trying to copy a document from memory rather than one word at a time, side by side, or rewriting the whole thing just to make some simple changes. Coding agents will delete tests or return True to get them to pass - something you would never expect of even a junior professional.

And I know this because I see it all the time. I use composer-2 and sonnet 4.6 on a regular basis. It's not much better for my colleagues who use Opus or GPT or any of the other frontier models. Most of the time it's fine, but other times it does things simply unforgivable for a human. I have to watch the agent closely so that it doesn't decide to nuke my database; I don't have to do that with any of my juniors, even those with little experience and poor discipline.

Re: LLMs corrupt your documents when you delegate

#37

Post by Kim_Bruning »

timacles wrote: post #14

Least shocking thing I've read about LLMs recently. They are essentially like that one JPEG meme, where each pass of saving as JPEG slightly degrades the quality until by the end its unrecognizable. Except with LLMs, the starting point is intent. Each pass of the LLMs degrades the intent, like in the case of a precise scientific paper, just a little bit of nuance, a little bit of precision is lost with a re-wording h…

There's a kid's game that illustrates this too: https://en.wikipedia.org/wiki/Telephone_game

Re: LLMs corrupt your documents when you delegate

#38

Post by somewhatgoated »

trollbridge wrote: post #16
sebastiennight wrote: post #11

Earlier quoted context omitted.

> I've also had them convert to markdown something like an excel formatted document. This look like a task where the LLM would be best used in writing a deterministic script or program that then does the conversion. Trusting a LLM to make the change without tools is like telling the smartest person you know to just recite the converted document out loud from memory. At some point they'll get distracted, wrong, or unw…

I see people cut and paste from Excep into a chat, as an image, and ask it to sum up numbers.

I’ve seen people drink their own recycled piss and inject coffee into their ass - what’s your point?

Re: LLMs corrupt your documents when you delegate

#39

Post by devmor »

ekidd wrote: post #30
timacles wrote: post #14

Least shocking thing I've read about LLMs recently. They are essentially like that one JPEG meme, where each pass of saving as JPEG slightly degrades the quality until by the end its unrecognizable. Except with LLMs, the starting point is intent. Each pass of the LLMs degrades the intent, like in the case of a precise scientific paper, just a little bit of nuance, a little bit of precision is lost with a re-wording h…

Where this result is actually interesting and relevant is when a coding agent splits a large source file into multiple smaller files. Opus + Claude Code will try to recite long sections of source code from memory into each of the new files, instead of using some sort of copy/paste operation like a human would. Moving a file is a bit easier. LLMs may sometimes try to recite the file from memory. But if you tell them t…

If you’re using LLMs for agentic work it is absolutely essential that you have a robust set of tools for them to use and the correct instructions to prompt their use.

The LLM will come up with stupid ways to do things, common sense doesn’t exist for AI.

Re: LLMs corrupt your documents when you delegate

#40

Post by ACCount37 »

lelanthran wrote: post #32
threethirtytwo wrote: post #28

Earlier quoted context omitted.

A human doing the same tasks as what the LLM did in the paper that the human will degrade the document further then the LLM. If the LLM is 25%, a human would degrade it probably 80% if they used the same technique as the LLM did in this paper. I'm talking about a single pass. The fact of the matter is, humans don't edit things the way it was done in the paper and neither do coding agents like claude. Think about it:…

Are we comparing humans to LLMs or human written software to LLMs? The whole point of creating software to do things used to be getting things done more accurately and consistently.

No. The whole point of creating software is getting things done.

"More accurately and consistently" was merely downstream from what capabilities were natural for machine logic and hard algorithms.

Now, we're just spoiled for choice. We have hard algorithm software where we want to do things that benefit for accurate, consistent, highly deterministic behavior - and we have soft algorithm AI for when we want to do things that simply aren't amenable to hard logic.

Machine translation used to be a horrid mess when we were trying to do it with symbolic systems. Because symbolic systems are "consistent, highly deterministic" but not at all "accurate" on translation tasks. Being able to leverage LLMs for that is a generational leap.

Post reply on HN