Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs corrupt your documents when you delegate

arxiv.org

Post reply on HN
Print view View original

101–110 of 235 posts

Re: LLMs corrupt your documents when you delegate

#101

Post by nostrademons »

fiddlerwoaroof wrote: post #75
isityettime wrote: post #70

Earlier quoted context omitted.

I've definitely experienced this while coding with LLMs. Often, after a flurry of feature work in which I thought I was being reasonably careful but moving very fast, I take a closer look at some small piece of code and go "holy shit". Then I have to spend a few hours going over everything and carefully reworking parts where things didn't quite go how I'd like, where I was unclear, or where the LLM's brainworms kicke…

My experience mostly matches this: I think of a piece of development work having three phases: 1. Prototype 2. Initial production implementation 3. Hardening My experience with LLMs is that they solve “writer’s block” problems in the prototyping phase at the expense of making phases 2+3 slower because the system is less in your head. They also have a mixed effect on ongoing maintenance: small tasks are easier but you…

Usually you'll iterate several times on #1, which is where LLMs are really helpful. They let you get working code from stage #1 quite quickly, so you can check the output and behavior, and then oftentimes you'll find that you framed the problem incorrectly in the first place. Then you can fix your problem definition, have the LLM rewrite the code, try it again, and so on, until you get the results you want.

#1 -> #2 is a gap, but it also helps if you ask the LLM to explain its thinking and generate a human-readable design-doc of the approach it took and code organization it used. Then you read the design doc to gain the context, and pick up with #2.

Re: LLMs corrupt your documents when you delegate

#102

Post by the8472 »

nostrademons wrote: post #96
isityettime wrote: post #70

Earlier quoted context omitted.

I've definitely experienced this while coding with LLMs. Often, after a flurry of feature work in which I thought I was being reasonably careful but moving very fast, I take a closer look at some small piece of code and go "holy shit". Then I have to spend a few hours going over everything and carefully reworking parts where things didn't quite go how I'd like, where I was unclear, or where the LLM's brainworms kicke…

This seems closely related to the problem of model collapse [1][2][3], where LLMs lose the tails of the distribution, and so when you recursively train on the output of an LLM, or otherwise feed the output back into the input in subsequent stages, you lose the precision and diversity that human authors bring to the work. Eventually everything regresses to the mean and anything that would've made the content unique, u…

There are plenty of AIs that are immune to this because they're trained on something that won't be flooded with slop. E.g. robotics, self-driving cars (both trained on real camera/sensor inputs) or programming/proof-assistant stuff (trained on things that are verifiable).

Re: LLMs corrupt your documents when you delegate

#103

Post by hedora »

threethirtytwo wrote: post #25

This experiment needs to be put in perspective. Let me explain. IF you did this SAME experiment with a human and had a human read an ENTIRE document and then reproduce said document with edits. The DOCUMENT would DEGRADE even more. The way this experiment is conducted is not inline with how current agentic AI is used OR how even humans edit documents. Here's how agentic AI currently typically do edits: 1. They read t…

Benjamin Franklin famously taught himself to write well by doing what you describe: Read a piece of a book, then rewrite it, then compare.

At first his copies were badly degraded. Eventually, he was considered one of the best writers of his time.

I feel like there's probably some way "the copy is better" could be quantified (at least to the point where it fools most of the people most of the time). If so, then expect LLMs to learn the same trick within a generation or two.

Re: LLMs corrupt your documents when you delegate

#104

Post by shahbaby »

threethirtytwo wrote: post #26
simonw wrote: post #20

I'm suspicious of their results with regards to tool usage. It's unsurprising that round-tripping long content through an LLM results in corruption. Frequent LLM users already know not to do that. They claim that tool use didn't help, which surprised me... but they also said: > To test this, we implemented a basic agentic harness (Yao et al., 2022) with file reading, writing, and code execution tools (Appendix M). We…

People love to interpret the results in the most negative way possible because it's a threat to their occupation and identity. I refer to HN specifically. The fact of the matter is, if you want to edit a document by reading the document and then regurgitating the entire document with said edits... a human will DO worse then a 25% degradation. It's possible for a human to achieve 0% degradation but the human will have…

> People love to interpret the results in the most negative way possible because it's a threat to their occupation and identity.

OR it could be because their concerns are genuine but are ignored in favour of a good sounding story.

Re: LLMs corrupt your documents when you delegate

#105

Post by TedDoesntTalk »

timacles wrote: post #14

Least shocking thing I've read about LLMs recently. They are essentially like that one JPEG meme, where each pass of saving as JPEG slightly degrades the quality until by the end its unrecognizable. Except with LLMs, the starting point is intent. Each pass of the LLMs degrades the intent, like in the case of a precise scientific paper, just a little bit of nuance, a little bit of precision is lost with a re-wording h…

> the more they will tend to gradually pull that into some homogenous abstract equilibrium

I experienced this with resume editing. The LLM removes everything that differentiates my resume from a pile of junior engineers with “average” experience. Anything that was special or unique or different was eventually replaced with generic stuff

Of course I didn’t use what it produced, but it was maddening because the LLM kept insisting this was better than what I had.

I found LLMs to be much more useful in suggesting edits to very small chunks of my resume (a sentence or three) rather than the overall vision of the document.

Re: LLMs corrupt your documents when you delegate

#106

Post by 8note »

CamperBob2 wrote: post #89
oldsecondhand wrote: post #86

Earlier quoted context omitted.

> If you can come up with a way to do math without reasoning, that would be, in a sense, even more interesting than AI. Logic is just syntactic manipulation of formulas. By the early 90s logical reasoning was pretty much solved with classical AI (the last building block being constraint logic programming).

So you'll be able to show me the early-90s era program that can solve original IMO-level problems when supplied with the plaintext questions. Right?

if i presented math problems to the best english mathematicians in chinese, does that mean they arent able to reason? the plain text is an arbitrary constraint

Re: LLMs corrupt your documents when you delegate

#107

Post by Salgat »

devmor wrote: post #39
ekidd wrote: post #30

Earlier quoted context omitted.

Where this result is actually interesting and relevant is when a coding agent splits a large source file into multiple smaller files. Opus + Claude Code will try to recite long sections of source code from memory into each of the new files, instead of using some sort of copy/paste operation like a human would. Moving a file is a bit easier. LLMs may sometimes try to recite the file from memory. But if you tell them t…

If you’re using LLMs for agentic work it is absolutely essential that you have a robust set of tools for them to use and the correct instructions to prompt their use. The LLM will come up with stupid ways to do things, common sense doesn’t exist for AI.

My favorite is when Claude will build a completely new application to load and inspect a .dll file using reflection instead of just googling the library's interfaces.

Re: LLMs corrupt your documents when you delegate

#108

Post by 8note »

leptons wrote: post #83
danielvaughn wrote: post #44

I've spent the last few months reading a lot of AI-generated code. It's extremely difficult. It's like how psychopaths are eerie because there's nothing behind their eyes. AI-generated code is eerie because there's nothing between the lines. Code is in some sense theory building, and when you read a humans code you can (mostly) feel their theory working in the background. LLMs have no such theory, the code is just fa…

My company is moving to a workflow where we only write Jira tickets, the LLM writes all the code and submits a PR. Then we are supposed to review the code the LLM wrote. I'm looking for a new job.

that doesnt seem particularly horrible, as long as you as the engineer can still go change things in the code package and surrounding infrastructure to improve the output, and make sure that the agent is actually making the right stuff the first time you see the outputs

eg. setting up better feedback loops, improving CI/CD, breaking changes up at the right scale, etc.

you i assume also can then put in more work up front, doing simulations of solutions, lean proofs, and so on?

more engineering, less plumbing

Re: LLMs corrupt your documents when you delegate

#109

Post by tmaly »

When AI generates code, we have the ability to easily verify it and test it.

The same is not so easy with free form text. I have been thinking about this mainly around when agents write plans or edit plans, but I think figuring out how to do this in general would be a huge breakthrough.

Logical English was one idea I came across and Runcible https://runcible.com/ was another idea I recently stumbled on.

Re: LLMs corrupt your documents when you delegate

#110

Post by y3ahd0g »

Yeah so I run my agents as a different user that do not have write perms to my /home

Then I can diff what they wrote with my copy

Users are the OG container. On Linux it's possible to constrain a user to a network namespace, cgroups.

BPF can be used like docker compose to ensure a service running under a user is running

TL;DR a lot of the userspace cruft we import to run software has been rolled into the kernel over the last 10-15 years.

Ignore the terminology "user". Under the hood all the same constraint and boundary setting you want exists without downloading the entire internet

Post reply on HN