Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs corrupt your documents when you delegate

arxiv.org

Post reply on HN
Print view View original

71–80 of 235 posts

Re: LLMs corrupt your documents when you delegate

#71

Post by XMPPwocky »

figarus314 wrote: post #57
CamperBob2 wrote: post #51

Earlier quoted context omitted.

I don't know what a "booster" is, but if a model can solve original math problems, then it's reasoning. If you can come up with a way to do math without reasoning, that would be, in a sense, even more interesting than AI.

A model solving original math problems may look like human reasoning, but internally the model is choosing the next token based on what it has learned about probability around various patterns and structures. The model knows about correlations between problems, proof techniques and answer structures, and when it "reasons" it's selecting a high probability trajectory through that learned knowledge. A calculator is dif…

> The difference is that when a human reasoner goes to solve a problem, they'll think "this kind of proof usually goes this way" - following an explicit rule enforcement.

How is this different from "probabilistic pattern selection"?

Re: LLMs corrupt your documents when you delegate

#72

Post by isityettime »

ClikeX wrote: post #60
ekidd wrote: post #30

Earlier quoted context omitted.

Where this result is actually interesting and relevant is when a coding agent splits a large source file into multiple smaller files. Opus + Claude Code will try to recite long sections of source code from memory into each of the new files, instead of using some sort of copy/paste operation like a human would. Moving a file is a bit easier. LLMs may sometimes try to recite the file from memory. But if you tell them t…

> And for in-place edits, you can review "git diff" for surprises. I don't let AI touch git anyway, and I always review the diff after it generated stuff. If it modifies my documentation, I always want to check if it messed with the text instead of just added formatting.

This. I know the LLM agents often have their own little diff viewers and edit approval workflows, but for a high volume of code, I cannot imagine actually reviewing everything without leaning on much more capable Git tooling.

I use Magit, and up until I started using LLM agents it was mostly a nice-to-have that I relied on casually. (I was definitely under-utilizing its power.) But for reviewing, selectively staging, and selectively rejecting the changes of an LLM agent? I feel like I'd die without it. Idk how others manage.

Re: LLMs corrupt your documents when you delegate

#73

Post by majormajor »

isityettime wrote: post #70
timacles wrote: post #14

Least shocking thing I've read about LLMs recently. They are essentially like that one JPEG meme, where each pass of saving as JPEG slightly degrades the quality until by the end its unrecognizable. Except with LLMs, the starting point is intent. Each pass of the LLMs degrades the intent, like in the case of a precise scientific paper, just a little bit of nuance, a little bit of precision is lost with a re-wording h…

I've definitely experienced this while coding with LLMs. Often, after a flurry of feature work in which I thought I was being reasonably careful but moving very fast, I take a closer look at some small piece of code and go "holy shit". Then I have to spend a few hours going over everything and carefully reworking parts where things didn't quite go how I'd like, where I was unclear, or where the LLM's brainworms kicke…

Yeah, a lot of "it doesn't matter how the code looks" convos seem to be ignoring that we know what happens over time when you just make tactical the-tests-still-pass changes over and over and over again. Slowly some of those tests get corrupted without noticing. And you never had the ENTIRE spec (and all the edge-case but user-relied-on-things) covered anyway. And then new dev gets way harder.

Re: LLMs corrupt your documents when you delegate

#74

Post by buffaloPizzaBoy »

I typically tell my agents to only treat document writing as a last "rendering" pass. LLMs are so good at taking sparse knowledge and compiling it, that I prefer to store knowledge as composable ideas/facts.

What has worked well in practice is giving the agent a directory, and tell it to make independent markdown files for facts/findings it locates - with each file having front-matter for easy search-ability.

This de-complects most tasks from "research AND store iteratively in a final document format" to more cohesive tasks "research a set of facts and findings which may be helpful for a document", and "assemble the document".

Only a partial mitigation, but find it leads to more versatile re-use of findings, same as if a human was working.

Re: LLMs corrupt your documents when you delegate

#75

Post by fiddlerwoaroof »

isityettime wrote: post #70
timacles wrote: post #14

Least shocking thing I've read about LLMs recently. They are essentially like that one JPEG meme, where each pass of saving as JPEG slightly degrades the quality until by the end its unrecognizable. Except with LLMs, the starting point is intent. Each pass of the LLMs degrades the intent, like in the case of a precise scientific paper, just a little bit of nuance, a little bit of precision is lost with a re-wording h…

I've definitely experienced this while coding with LLMs. Often, after a flurry of feature work in which I thought I was being reasonably careful but moving very fast, I take a closer look at some small piece of code and go "holy shit". Then I have to spend a few hours going over everything and carefully reworking parts where things didn't quite go how I'd like, where I was unclear, or where the LLM's brainworms kicke…

My experience mostly matches this: I think of a piece of development work having three phases:

1. Prototype 2. Initial production implementation 3. Hardening

My experience with LLMs is that they solve “writer’s block” problems in the prototyping phase at the expense of making phases 2+3 slower because the system is less in your head. They also have a mixed effect on ongoing maintenance: small tasks are easier but you lose some of the feel of the system.

Re: LLMs corrupt your documents when you delegate

#77

Post by mrcartmeneses »

timacles wrote: post #14

Least shocking thing I've read about LLMs recently. They are essentially like that one JPEG meme, where each pass of saving as JPEG slightly degrades the quality until by the end its unrecognizable. Except with LLMs, the starting point is intent. Each pass of the LLMs degrades the intent, like in the case of a precise scientific paper, just a little bit of nuance, a little bit of precision is lost with a re-wording h…

Further, could we think of intent as some ordered state, and over time the LLM introduces entropy, eventually resulting in something akin to free-association?

Re: LLMs corrupt your documents when you delegate

#78

Post by sebastiennight »

somewhatgoated wrote: post #38
trollbridge wrote: post #16

Earlier quoted context omitted.

I see people cut and paste from Excep into a chat, as an image, and ask it to sum up numbers.

I’ve seen people drink their own recycled piss and inject coffee into their ass - what’s your point?

In the first half, I thought you were an astronaut, but the second half has me double-guessing myself.

Re: LLMs corrupt your documents when you delegate

#79

Post by senordevnyc »

figarus314 wrote: post #57
CamperBob2 wrote: post #51

Earlier quoted context omitted.

I don't know what a "booster" is, but if a model can solve original math problems, then it's reasoning. If you can come up with a way to do math without reasoning, that would be, in a sense, even more interesting than AI.

A model solving original math problems may look like human reasoning, but internally the model is choosing the next token based on what it has learned about probability around various patterns and structures. The model knows about correlations between problems, proof techniques and answer structures, and when it "reasons" it's selecting a high probability trajectory through that learned knowledge. A calculator is dif…

I don’t think there’s any evidence that “human reasoning” isn’t also based on probabilistic pattern selection.

Re: LLMs corrupt your documents when you delegate

#80

Post by originalvichy »

isityettime wrote: post #70
timacles wrote: post #14

Least shocking thing I've read about LLMs recently. They are essentially like that one JPEG meme, where each pass of saving as JPEG slightly degrades the quality until by the end its unrecognizable. Except with LLMs, the starting point is intent. Each pass of the LLMs degrades the intent, like in the case of a precise scientific paper, just a little bit of nuance, a little bit of precision is lost with a re-wording h…

I've definitely experienced this while coding with LLMs. Often, after a flurry of feature work in which I thought I was being reasonably careful but moving very fast, I take a closer look at some small piece of code and go "holy shit". Then I have to spend a few hours going over everything and carefully reworking parts where things didn't quite go how I'd like, where I was unclear, or where the LLM's brainworms kicke…

This is definitely most annoying when dealing with software or standards with slightly illogical or hard to grasp cases. Recently, I worked on one of the software community's favourite spaces, timezones, and kept getting myself and my LLM context polluted with the confusion that arises when using POSIX standard timezone notation and common human-readable formats.

This blog probably covers my exact headache [0]. In summary, "Etc/GMT+6" actually means UTC-6. I was developing a one-off helper script to massively create calendars to a web app via API, and when trying to validate my CSV+Python script's results, I kept getting confused as to when do the CSV rows have correct data and when does the web app UI have correct data. LLM probably developed the Python script in a manner that translated this on-the-fly, but my human-readable "Calendar name" column which had "Etc/GMT+6" would generate a -6 in the web app. This probably would not have been a problem with explicit locations specified, but my use case would not allow for that.

When trying to debug if something is wrong, the thinking trace was going into loops trying to figure out if the "problem" is coming from my directions, the code's bugs, or the CSV having incorrect data.

Learning: when facing problems like this, try using the well-known "notepad file" methods to track problems like this, so that if the over-eager LLM starts applying quick code fixes – although YOU were the "problem's" source – it will be easier to undo or clean up code that was added to the repository during a confusing debug session. For me, it has been difficult to separate "code generated due to more resilient improvements" vs. "code generated during debugging that sort of changed some specific step of the script".

(Do note that I am not an advanced software engineer, my practices are probably obvious to others. My repos are mainly comprised of sysadmin style shell/python helper code! :-) )

[0]https://blacksheepcode.com/posts/til_etc_timezone_is_backwar...

Post reply on HN