Live data from Hacker News

Antislop: A framework for eliminating repetitive patterns in language models

arxiv.org

111–119 of 119 posts

Re: Antislop: A framework for eliminating repetitive patterns in language models

#111
post #79

Earlier quoted context omitted.

I don't think this is true. The LLMs use this construction noticeably more frequently than normal people, and I too feel the annoyance when they do, but if you look around I think you'll find it's pretty common in many registers of human natural english.

And each of us has patterns. I bet if you read a million of my posts, you would be annoyed with my writing idiosyncrasies too.

Yes, this is absolutely part of it, and I think an underappreciated harm of LLMs is the homogeneity. Even to the extent that their writing style is adequate, it is homogeneous in a way that quickly becomes grating when you encounter LLM-generated text several times a day. That said, I think it's fair to judge LLM writing style not to be adequate for most purposes, partly because a decent human writer does a better job of consciously keeping their prose interesting by varying their wording and so forth.

Re: Antislop: A framework for eliminating repetitive patterns in language models

#112
post #79

Earlier quoted context omitted.

This is the main thing that immediately tells me something is AI. This form of reasoning was much less common before ChatGPT.

I don't think this is true. The LLMs use this construction noticeably more frequently than normal people, and I too feel the annoyance when they do, but if you look around I think you'll find it's pretty common in many registers of human natural english.

Not sure what the downvotes are for -- it's trivial to find examples of this contruction from before 2023, or even decades ago. I'm not disagreeing that LLMs overuse this construction (tbh it was already something of a "writing smell" for me before LLMs started doing it, because it's often a sign of a weakly motivated argument).

Re: Antislop: A framework for eliminating repetitive patterns in language models

#113

I've been using ChatGPT fairly regularly for about a year. Mostly as an editor/brainstorming-partner/copy-reviewer. Lots of things have changed in that year, but the things that haven't are: * So, so many em-dashes. All over the place. (I've tried various ways to get it to stop. None of them have worked long term). * Random emojis. * Affirmations at the start of messages. ("That's a great idea!") With a brief pause w…

I am reasonably sure affirmations are a feature, not a bug. No matter how much I might disagree.

Also pretty sure it is a feature because the general population wants to have pleasant interactions with their ChatGPT and OpenAI's user feedback research will have told them this helps. I know some non-developer type people which mostly talk to ChatGPT about stuff like

- how to cope with the sadness of losing their cat

- ranting about the annoying habits of their friends

- finding all the nice places to eat in a city

etc.

They do not want that "robot" personality and they are the majority.

Re: Antislop: A framework for eliminating repetitive patterns in language models

#114

Earlier quoted context omitted.

I am reasonably sure affirmations are a feature, not a bug. No matter how much I might disagree.

Also pretty sure it is a feature because the general population wants to have pleasant interactions with their ChatGPT and OpenAI's user feedback research will have told them this helps. I know some non-developer type people which mostly talk to ChatGPT about stuff like - how to cope with the sadness of losing their cat - ranting about the annoying habits of their friends - finding all the nice places to eat in a cit…

Agreed on all points.

I also recall reading a while back that it's also a dopamine trigger. If you make people feel better using your app, they keep coming back for another fix. At least until they realize the hollow nature of the affirmations and start getting negative feelings about it. Such a fine line.

Re: Antislop: A framework for eliminating repetitive patterns in language models

#115
post #80

This is the epitome of patching symptoms rather than treating the disease. Even if you suppress the obvious syntactic slop like 'it's not X but Y', you have no reason to believe you've fixed mode-collapse on higher more important levels like semantics and creativity. (For example, Claude LLMs have always struck me as mode-collapsed on a semantic level: they don't have the blatant verbal tics of 4o but somehow they st…

Those higher level kinds of mode collapse are hard to quantify in an automated way. To fix that, you would need interventions upstream, at pre & post training. This approach is targeted to the kinds of mode collapse that we can meaningfully measure and fix after the fact, which is constrained to these verbal tics. Which doesn't fix higher level mode collapse on semantics & creativity that you're identifying -- but I…

> but I think fixing the verbal tics is still important and useful.

I don't. I think they're useful for flagging the existence of mode-collapse and also providing convenient tracers for AI-written prose. Erasing only the verbal tics with the equivalent of 's/ - /; /g' (look ma! no more 4o em dashes!) is about the worst solution you could come up with and if adopted would lead to a kind of global gaslighting. The equivalent of a vaccine for COVID which only suppresses coughing but doesn't change R, or fixing a compiler warning by disabling the check.

If you wanted to do useful research here, you'd be doing the opposite. You'd be figuring out how to make the verbal expressions even more sensitive to the underlying mode-collapse, to help research into fixing it and raising awareness. (This would be useful even on the released models, to more precisely quantify their overall mode-collapse, which is poorly captured by existing creative writing benchmarks, I think, and one reason I've had a hard time believing things like Eqbench rankings.)

Re: Antislop: A framework for eliminating repetitive patterns in language models

#116
post #87

Earlier quoted context omitted.

I notice this less with GPT-5 and GPT-5-Codex but it has a new problem: it'll write a sentence that mostly makes sense but have one or two strange word choices that nobody would use in that situation. It tends to use a lot of very dense jargon that makes it hard to read, spitting out references to various algorithms and concepts in places that don't actually make sense for them to be. Also yesterday Codex refused a t…

> refused a task from me because it would be too much work Was this after many iterations? Try letting it get some "sleep". Hear me out... I haven't used Codex, so maybe not relevant, but with Claude I always notice a slow degradation in quality, refusals, and " " placeholders with iterations within the same context window. One time, after making a mistake, it apologized and said something like "that's what I get for…

That's hilarious.

Re: Antislop: A framework for eliminating repetitive patterns in language models

#117
post #85

Earlier quoted context omitted.

Yeah, that’s a really insightful point, and you’ve kind of hit the nail on the head…

yesterday it told me the "juice wasn't worth the squeeze."

I got a rock.

Re: Antislop: A framework for eliminating repetitive patterns in language models

#118
post #19

Earlier quoted context omitted.

You can take my em-dashes from my cold, dead hands—I use them all the time.

On iOS in particular the longer dash variants are easy to access — via long pressing dash. Anecdotally, I use them less often these days, because of the association with AI.

On MacOS or iPadOS keyboard, option - and option shift - give n and m dashes respectively.
Post reply on HN