Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Antislop: A framework for eliminating repetitive patterns in language models

arxiv.org

Post reply on HN
Print view View original

91–100 of 119 posts

Re: Antislop: A framework for eliminating repetitive patterns in language models

#91

Post by razodactyl »

Der_Einzige wrote: post #88
gwern wrote: post #80

This is the epitome of patching symptoms rather than treating the disease. Even if you suppress the obvious syntactic slop like 'it's not X but Y', you have no reason to believe you've fixed mode-collapse on higher more important levels like semantics and creativity. (For example, Claude LLMs have always struck me as mode-collapsed on a semantic level: they don't have the blatant verbal tics of 4o but somehow they st…

Wow it’s pretty sad to see one of my idols be kind of a hater on this. :(

How come? It was a valid take on the situation. Critical feedback is vital to success.

Re: Antislop: A framework for eliminating repetitive patterns in language models

#93

Post by furyofantares »

Forgeties79 wrote: post #28
atourgates wrote: post #15

I've been using ChatGPT fairly regularly for about a year. Mostly as an editor/brainstorming-partner/copy-reviewer. Lots of things have changed in that year, but the things that haven't are: * So, so many em-dashes. All over the place. (I've tried various ways to get it to stop. None of them have worked long term). * Random emojis. * Affirmations at the start of messages. ("That's a great idea!") With a brief pause w…

> Affirmations at the start of messages. ("That's a great idea!") With a brief pause when 5 launched. But it's back and worse than ever now. What a great point! I also can’t stand it. I get it’s basically a meme to point it out - even South Park has mocked it - but I just cannot stand it. In all seriousness it’s so annoying. It is a tool, not my friend, and considering we are already coming from a place of skepticism…

> Stop wasting my tokens and time with fake friendship!

They could hide it so that it doesn't annoy you, but I think it's not a waste of tokens. It's there so the tokens that follow are more likely to align with what you asked for. It's harder for it to then say "This is a lot of work, we'll just do a placeholder for now" or give otherwise "lazy" responses, or to continue saying a wrong thing that you've corrected it about.

I bet it also probably makes it more likely to gaslight you when you're asking something it's just not capable of, though.

Re: Antislop: A framework for eliminating repetitive patterns in language models

#94

Post by sam-paech »

gwern wrote: post #80

This is the epitome of patching symptoms rather than treating the disease. Even if you suppress the obvious syntactic slop like 'it's not X but Y', you have no reason to believe you've fixed mode-collapse on higher more important levels like semantics and creativity. (For example, Claude LLMs have always struck me as mode-collapsed on a semantic level: they don't have the blatant verbal tics of 4o but somehow they st…

Those higher level kinds of mode collapse are hard to quantify in an automated way. To fix that, you would need interventions upstream, at pre & post training.

This approach is targeted to the kinds of mode collapse that we can meaningfully measure and fix after the fact, which is constrained to these verbal tics. Which doesn't fix higher level mode collapse on semantics & creativity that you're identifying -- but I think fixing the verbal tics is still important and useful.

Re: Antislop: A framework for eliminating repetitive patterns in language models

#95

Post by psyclobe »

atourgates wrote: post #15

I've been using ChatGPT fairly regularly for about a year. Mostly as an editor/brainstorming-partner/copy-reviewer. Lots of things have changed in that year, but the things that haven't are: * So, so many em-dashes. All over the place. (I've tried various ways to get it to stop. None of them have worked long term). * Random emojis. * Affirmations at the start of messages. ("That's a great idea!") With a brief pause w…

I was able to get it to briefly change that initial You’re right!! By telling it to say something else like Yarr Mayte. Stuck for a while.

Re: Antislop: A framework for eliminating repetitive patterns in language models

#96

Post by anbotero »

antoniojtorres wrote: post #20
atourgates wrote: post #15

I've been using ChatGPT fairly regularly for about a year. Mostly as an editor/brainstorming-partner/copy-reviewer. Lots of things have changed in that year, but the things that haven't are: * So, so many em-dashes. All over the place. (I've tried various ways to get it to stop. None of them have worked long term). * Random emojis. * Affirmations at the start of messages. ("That's a great idea!") With a brief pause w…

The emoji thing is so bad. You can see it all over github docs and other long form docs. All section headers will have emojis and so on. Strange.

It drives me crazy. It happens with Claude models too. I even created an instruction to avoid them in a CLAUDE.md, and the miserable thing from time to time still does it.

Why?!

Re: Antislop: A framework for eliminating repetitive patterns in language models

#97

Post by quietbritishjim »

thraxil wrote: post #48
antoniojtorres wrote: post #20

Earlier quoted context omitted.

The emoji thing is so bad. You can see it all over github docs and other long form docs. All section headers will have emojis and so on. Strange.

Obviously nothing solid to back this up, but I kind of feel like I was seeing emojis all over github READMEs on JS projects for quite a while before AI picked it up. I feel like it may have been something that bled over from Twitch streaming communities.

Yeah and this explains why you see it in LLMs in the first place. They had to learn it from somewhere.

Re: Antislop: A framework for eliminating repetitive patterns in language models

#99

Post by vintermann »

thraxil wrote: post #48
antoniojtorres wrote: post #20

Earlier quoted context omitted.

The emoji thing is so bad. You can see it all over github docs and other long form docs. All section headers will have emojis and so on. Strange.

Obviously nothing solid to back this up, but I kind of feel like I was seeing emojis all over github READMEs on JS projects for quite a while before AI picked it up. I feel like it may have been something that bled over from Twitch streaming communities.

The name of HuggingFace is a reminder that it was a thing long before the current crop of LLMs.

Re: Antislop: A framework for eliminating repetitive patterns in language models

#100

Post by vintermann »

Forgeties79 wrote: post #56
cyanydeez wrote: post #52

Earlier quoted context omitted.

Guys. It's basically because among the all well researched data, the amount of garbage is infinitely more. If AI wants to be useful (it's not going to atm), real people need to cull all the banalities that facebook, reddit & forums have generated. Because what you're noticing is things we typically elide over in discussions with actual humans.

It is far more polite than any social media platform or forum I’ve ever seen lol

Yeah, and earlier incarnations too... I remember AI dungeon used to cuss people out and even "leave the chat" when people acted annoying.
Post reply on HN