Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs corrupt your documents when you delegate

arxiv.org

Post reply on HN
Print view View original

61–70 of 235 posts

Re: LLMs corrupt your documents when you delegate

#61

Post by bigstrat2003 »

jvuygbbkuurx wrote: post #52
devmor wrote: post #39

Earlier quoted context omitted.

If you’re using LLMs for agentic work it is absolutely essential that you have a robust set of tools for them to use and the correct instructions to prompt their use. The LLM will come up with stupid ways to do things, common sense doesn’t exist for AI.

Isn't this the whole reason they became viable in the last 6 months? The system prompt and harness is improving. It's less and less essential every day to roll your own.

They still aren't viable. Nothing changed within the last 6 months.

Re: LLMs corrupt your documents when you delegate

#63

Post by glaslong »

Twirrim wrote: post #27
timacles wrote: post #14

Least shocking thing I've read about LLMs recently. They are essentially like that one JPEG meme, where each pass of saving as JPEG slightly degrades the quality until by the end its unrecognizable. Except with LLMs, the starting point is intent. Each pass of the LLMs degrades the intent, like in the case of a precise scientific paper, just a little bit of nuance, a little bit of precision is lost with a re-wording h…

A coworker talks about LLMs as "bullshit" layers. Not exactly dismissing them or being derogatory about them, but emphasising that each time you feed something through an LLM, what comes out the other side may not be what you expect/want. Like that guy at the pub sharing what he'd seen online somewhere, after a few pints. Might be accurate, but carries notable risk it's not. So e.g., don't use an LLM to call an API t…

I like this framing. At least as "nondeterministic" vs "deterministic" layers for the folks who flinch at "bullshit." Also "broadly capable but lossy" versus "limited capability but reliable."

Building structures of dependencies, the interface between each pair seems to collapse to the lesser of the two. So there's a ton of work right now going into TLA+, structured io, etc to force even a bit of reliability back into the LLM/program boundaries. To have any hope of chaining multiple LLM dependencies in a stack without the whole thing toppling chaotically.

Re: LLMs corrupt your documents when you delegate

#64

Post by rhubarbtree »

carterschonwald wrote: post #31

this is literally just “leave a child at the work computer with a real doc open playing office”. otoh it is good to design benchmarks tonground these things. on the flip side if you’re literally just using a bare bones harness on top of a stochastic parrot, of course stochastic errors accumulate. theres a lot of ways for improving text faithfulness through harness tool designs, and my incremental experiments seem pro…

It’s not a stochastic parrot.

Re: LLMs corrupt your documents when you delegate

#65

Post by deadbabe »

embedding-shape wrote: post #54
jvuygbbkuurx wrote: post #52

Earlier quoted context omitted.

Isn't this the whole reason they became viable in the last 6 months? The system prompt and harness is improving. It's less and less essential every day to roll your own.

I don't think there is a single reason. Models are improving, so are the harnesses, prompts and we who use them a lot also get more proficient and learn where they can be used effectively vs not, so lots of improvements all over the ecosystem, brought together. Latest big change is probably how feasible local models are becoming, like Qwen 3.6 and Gemma 4, they're no longer easily getting stuck in loops and repetitio…

> we who use them a lot also get more proficient and learn where they can be used effectively vs not

I think it’s always been obvious where an LLM could be used effectively and where it cannot, if you understand how they work and don’t see them as magical.

The “increase in proficiency” is mostly people coming back to reality and being more intentional about LLM usage. There are no surprise discoveries here. One does not need to use an LLM a lot to get effective with them. A total noob could become effective on day 1 with proper guidance.

Re: LLMs corrupt your documents when you delegate

#66

Post by glaslong »

danielvaughn wrote: post #44

I've spent the last few months reading a lot of AI-generated code. It's extremely difficult. It's like how psychopaths are eerie because there's nothing behind their eyes. AI-generated code is eerie because there's nothing between the lines. Code is in some sense theory building, and when you read a humans code you can (mostly) feel their theory working in the background. LLMs have no such theory, the code is just fa…

Thank you I've had trouble articulating this sense, but it's strong. An uncanny valley.

Re: LLMs corrupt your documents when you delegate

#67

Post by rmwaite »

What I find fascinating about LLMs is that a lot of their failures seem strikingly similar to the failures that humans struggle with. I’m not sure what this “means” but I think it’s interesting that we can theoretically fix these failures for LLMs but for humans it is much harder. You pretty much need to educate / indoctrinate people for their entire lives and even then it’s messy and unpredictable and prone to failure—just like LLMs.

Re: LLMs corrupt your documents when you delegate

#68

Post by ieieue »

Terr_ wrote: post #56
CamperBob2 wrote: post #51

Earlier quoted context omitted.

I don't know what a "booster" is, but if a model can solve original math problems, then it's reasoning. If you can come up with a way to do math without reasoning, that would be, in a sense, even more interesting than AI.

My dear sir, the entire universe is made of things that "do math without reasoning!" It's the default, and if we're lucky we harness pieces of it to discern something we're interested in.

[flagged]

Re: LLMs corrupt your documents when you delegate

#69

Post by ieieue »

figarus314 wrote: post #57
CamperBob2 wrote: post #51

Earlier quoted context omitted.

I don't know what a "booster" is, but if a model can solve original math problems, then it's reasoning. If you can come up with a way to do math without reasoning, that would be, in a sense, even more interesting than AI.

A model solving original math problems may look like human reasoning, but internally the model is choosing the next token based on what it has learned about probability around various patterns and structures. The model knows about correlations between problems, proof techniques and answer structures, and when it "reasons" it's selecting a high probability trajectory through that learned knowledge. A calculator is dif…

[flagged]

Re: LLMs corrupt your documents when you delegate

#70

Post by isityettime »

timacles wrote: post #14

Least shocking thing I've read about LLMs recently. They are essentially like that one JPEG meme, where each pass of saving as JPEG slightly degrades the quality until by the end its unrecognizable. Except with LLMs, the starting point is intent. Each pass of the LLMs degrades the intent, like in the case of a precise scientific paper, just a little bit of nuance, a little bit of precision is lost with a re-wording h…

I've definitely experienced this while coding with LLMs. Often, after a flurry of feature work in which I thought I was being reasonably careful but moving very fast, I take a closer look at some small piece of code and go "holy shit". Then I have to spend a few hours going over everything and carefully reworking parts where things didn't quite go how I'd like, where I was unclear, or where the LLM's brainworms kicked in.

Quality is really important to me in its own right, but I also worry about this exact "repeated compression" problem: when my codebase is clean and I have an up-to-date mental model, an LLM can quickly help me churn out some feature work and still leave the codebase in a reasonable state. But as the LLM dirties up the codebase, its past mistakes or misunderstandings compound, and it's likely to flub more and more things. So I have to go back and "restore" things to a correct state before I feel comfortable using the LLM again.

Post reply on HN