Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs corrupt your documents when you delegate

arxiv.org

Post reply on HN
Print view View original

161–170 of 235 posts

Re: LLMs corrupt your documents when you delegate

#161

Post by AlienRobot »

ofjcihen wrote: post #114
simonw wrote: post #20

I'm suspicious of their results with regards to tool usage. It's unsurprising that round-tripping long content through an LLM results in corruption. Frequent LLM users already know not to do that. They claim that tool use didn't help, which surprised me... but they also said: > To test this, we implemented a basic agentic harness (Yao et al., 2022) with file reading, writing, and code execution tools (Appendix M). We…

I agree with most of what you wrote except for this: >Frequent LLM users already know not to do that. And I think that’s the biggest problem. Amidst the current push to utilize LLMs across orgs and groups there are a large (if even say majority) of people that are using them every day but who have never approached anything as technical as a “harness” before let alone an entire setup. For them the behavior mentioned h…

Exactly. When I use a scissor, I don't want the scissor to not work just because I'm not a "frequent scissor user," and then get told by someone who makes their breakfast with scissors that I'm doing it wrong. Most people will not be "frequent" anything users.

Re: LLMs corrupt your documents when you delegate

#163

Post by Art9681 »

simonw wrote: post #20

I'm suspicious of their results with regards to tool usage. It's unsurprising that round-tripping long content through an LLM results in corruption. Frequent LLM users already know not to do that. They claim that tool use didn't help, which surprised me... but they also said: > To test this, we implemented a basic agentic harness (Yao et al., 2022) with file reading, writing, and code execution tools (Appendix M). We…

Any rando can publish research nowadays. It means nothing. Just like "X country published N research papers last year". It is noise. In a world where it was required to attach age, experience level, and country of origin to every comment, research paper, or post on the internet, it would shatter the conviction we mistakenly have towards the information we receive.

This team is inexperienced and it shows.

The noise to signal ratio will get worse, even in "academia". Brace yourselves. The kids are growing up in this new world.

Re: LLMs corrupt your documents when you delegate

#164

Post by xstas1 »

buffaloPizzaBoy wrote: post #74

I typically tell my agents to only treat document writing as a last "rendering" pass. LLMs are so good at taking sparse knowledge and compiling it, that I prefer to store knowledge as composable ideas/facts. What has worked well in practice is giving the agent a directory, and tell it to make independent markdown files for facts/findings it locates - with each file having front-matter for easy search-ability. This de…

Sounds like a good system. To use the analogy from ths other comment, this would be like running an image through JPEG compression twice.

The issue happens then if you're updating the individual research files on a regular basis. (Or making a long series of commits on a starting code base.) Every edit has a chance of doing a drive-by cleanup on nearby lines. Over a long enough timeline, it'll ablate your logic into something featureless, like if you compress an image too many times.

Re: LLMs corrupt your documents when you delegate

#165

Post by xstas1 »

leptons wrote: post #141
8note wrote: post #108

Earlier quoted context omitted.

that doesnt seem particularly horrible, as long as you as the engineer can still go change things in the code package and surrounding infrastructure to improve the output, and make sure that the agent is actually making the right stuff the first time you see the outputs eg. setting up better feedback loops, improving CI/CD, breaking changes up at the right scale, etc. you i assume also can then put in more work up fr…

The change is turning me from someone who writes pretty good reliable code, to someone who has to read and review pretty bad code. If you think this is an improvement, you're nuts. It is inserting a pretty unreliable middle-man know for errors and hallucinations, that often just goes down and stops working for reasons we can't control into a workflow that has worked well for a decade, and we're paying extra to really…

Turning you into a "reverse centaur" to borrow a term from Cory Doctorow

Re: LLMs corrupt your documents when you delegate

#166

Post by crooked-v »

adampunk wrote: post #12
polskibus wrote: post #5

Earlier quoted context omitted.

By „with each pass” do you mean within the same session, or with new session (context window) each time?

Each edit, even with unrelated edits. I had a README referring to something as "the cathedral of s*t" (some HN commentators don't care for the swearing, which is systemically bad news but w/e) and the robot would lift that phrase out in drive-bys, repeatedly. Occasionally it would report the action, sometimes it would not bother to report it. It never reached into the README on an unrelated doc edit, but if it was to…

That kind of passive-aggressive pseudo-moralizing is a common feature of all the current 'frontier' models. Try to do something like get one to summarize A Song of Ice and Fire text and it's likely to try and covertly sand off all the 'offensive' rough edges without even saying it's doing so.

Re: LLMs corrupt your documents when you delegate

#167

Post by jrflowers »

threethirtytwo wrote: post #160
rune-dev wrote: post #154

Earlier quoted context omitted.

I think you’re living in a bubble if you think the average user of AI even knows what a harness is The vast majority of people are literally going to chatGPT, pasting in their document and asking for edits.

This will change too man. Maybe I am in a bubble but with how fast things are changing, it won’t be too long before the bubble becomes reality. Either way we should be doing experiments on the actual capabilities of AI not about the stupidest possible way to use AI because it helps validate your own negative bias against AI. Additionally as software engineers using agentic AI… which HN basically is… this experiment i…

> This will change too man. Maybe I am in a bubble but with how fast things are changing, it won’t be too long before the bubble becomes reality.

You can’t get mad at an experiment for not happening in the future.

> Either way we should be doing experiments on the actual capabilities of AI

They simulated common end user behavior

>because it helps validate your own negative bias against AI.

We’ve gone from “this study is flawed because language models don’t do that” to “this study is flawed because while language models do do that, I don’t think that they will in the future” to “data that could support a bias other than my own is bad”

Re: LLMs corrupt your documents when you delegate

#168

Post by LPisGood »

Art9681 wrote: post #162

Remind yourselves that most research papers are written by career students with no real world practical experience. That is all.

Spending some time in and around applied research labs and seeing how poorly the sausage looks before it gets made into a paper is quite distressing.

I’m sure there are labs out there doing excellent work (especially those focused on theory), but most of the applied research I’ve seen up close and personal is very poor indeed.

Re: LLMs corrupt your documents when you delegate

#169

Post by pcwelder »

simonw wrote: post #20

I'm suspicious of their results with regards to tool usage. It's unsurprising that round-tripping long content through an LLM results in corruption. Frequent LLM users already know not to do that. They claim that tool use didn't help, which surprised me... but they also said: > To test this, we implemented a basic agentic harness (Yao et al., 2022) with file reading, writing, and code execution tools (Appendix M). We…

It's worth noting that Claude Code itself doesn't use the `insert` tool. (It also uses custom edit tool not the suite's predefined str_replace)

Also as a person developing agentic code tools since before Claude Code, I'm skeptical if str_replace provides accuracy improvement over just full rewrite.

Back in the day when SOTA models would do lazy coding like `// ... rest of the code ...`, full rewrite wasn't easy. Search/replace was fast, efficient and without the lazy coding. However, it came with slight accuracy drop.

Today that accuracy drop might be minimal/absent, but I'm not sure if it could lead to improvements like preventing doc corruption.

Re: LLMs corrupt your documents when you delegate

#170

Post by nunez »

8note wrote: post #108
leptons wrote: post #83

Earlier quoted context omitted.

My company is moving to a workflow where we only write Jira tickets, the LLM writes all the code and submits a PR. Then we are supposed to review the code the LLM wrote. I'm looking for a new job.

that doesnt seem particularly horrible, as long as you as the engineer can still go change things in the code package and surrounding infrastructure to improve the output, and make sure that the agent is actually making the right stuff the first time you see the outputs eg. setting up better feedback loops, improving CI/CD, breaking changes up at the right scale, etc. you i assume also can then put in more work up fr…

my (gender neutral) dude.

WAKE UP.

Literally anyone can write a Jira ticket. US engineers are expensive. What do you think will happen when the powers that enacted this policy decide that the ticket to merged into prod rate is acceptable to them?

Post reply on HN