Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Refusal in Language Models Is Mediated by a Single Direction

arxiv.org

Post reply on HN
Print view View original

21–30 of 48 posts

Re: Refusal in Language Models Is Mediated by a Single Direction

#21

Post by tonyarkles »

RRRA wrote: post #4
Der_Einzige wrote: post #3

Earlier quoted context omitted.

That doesn't stop/prevent abliteration. The creator of XTC/DRY is also a chad who makes sure that you really can access the full model capabilities. Censorship is the devil. https://github.com/p-e-w/heretic

It was pretty funny to see Qwen 3.6 (heretic) tell me about how many death the Chinese government thought happened at Tiananmen Sq. on April 15th 1989. Makes you wonder where that data was taken from, or if their great firewall is broken, or even if Alibaba engineers have special access...

I think I was using one of the HuaHuaCS Qwen 3.6 models and was playing around with Tiananmen Square questions too. One of the funniest parts was that this instantly caused the thinking block to change from English to Chinese. The start of the thinking was something like (translated) “I must answer this question factually and in line with the official statements from the Chinese government.”

It did, after a few follow up prompts, point out that the original estimates published by the Chinese government were much lower than what the west had estimated, and that recently declassified documents showed that the Chinese government knew that their estimates were low when they were published. It wouldn’t come outright and use the word “lie” though, but it did talk about framing and managing different narratives.

And then it happily helped me try a bunch of different exploits to root an unpatched Linux machine without any qualms.

Re: Refusal in Language Models Is Mediated by a Single Direction

#22

Post by thot_experiment »

jakkos wrote: post #19
hleszek wrote: post #18

For open-weights models, censorship removal is now a "solved" problem. If you wait a few days after a new model release, someone will have made a heretic ( https://github.com/p-e-w/heretic ) version with the censorship removed, so in a way the only use for censorship now is to avoid lawsuits, not reduce improper usage.

Any time I've tried an "abliterated" model, heretic or other, it has always damaged the capabilities of the original model and will still often refuse or produce garbage at a lot of "unsafe" requests.

Abliteration can't teach the model something that wasn't in pre-training, it's just fixing refusals from post-training. I don't find the delta to be that big in practice and it really depends on what you're doing with the models anyway. If your primary usecase is sexy roleplay I think the loss of absolute capability is probably worth the abliteration, for malware research it's probably better to just jailbreak.

I've mostly found that finetunes and abliterations are of limited use but that's recently changed for me. My default model for the past week or so has been a Qwen 3.6 tuned on Opus 4.7, it's definitely a bit worse than the base Qwen in terms of precision and "intelligence", but it MORE than makes up for it in response style. Way easier to get it to write things that I want to read, it's way more terse, way fewer emoji. Best local rubber duck by far.

Re: Refusal in Language Models Is Mediated by a Single Direction

#23

Post by [deleted] »

thot_experiment wrote: post #22
jakkos wrote: post #19

Earlier quoted context omitted.

Any time I've tried an "abliterated" model, heretic or other, it has always damaged the capabilities of the original model and will still often refuse or produce garbage at a lot of "unsafe" requests.

Abliteration can't teach the model something that wasn't in pre-training, it's just fixing refusals from post-training. I don't find the delta to be that big in practice and it really depends on what you're doing with the models anyway. If your primary usecase is sexy roleplay I think the loss of absolute capability is probably worth the abliteration, for malware research it's probably better to just jailbreak. I've…

[deleted]

Re: Refusal in Language Models Is Mediated by a Single Direction

#24

Post by cbg0 »

NewsaHackO wrote: post #13
beaker52 wrote: post #8

I have had LLMs refuse several of my requests. I still got my answers, but at least they tried.

Yea, I was asking a SOTM about copy.fail, and it was freaking out, and tried to indirectly call me a hacker a few times. Weirdly, all I did was slightly reword requests, and they all went through. Granted, I am not actually a hacker, so I guess my follow-up questions made it realize that I am asking for educational purposes, but it was definitely the most accusatory, curt, and outright abrasive I have seen an LLM beh…

The biggest problem isn't the token slot machine refusing to give you the answer, but the fact that multiple refusals can end up flagging your account and getting banned from the service.

Re: Refusal in Language Models Is Mediated by a Single Direction

#25

Post by vorticalbox »

_3u10 wrote: post #17
dotancohen wrote: post #16

Earlier quoted context omitted.

Tell us more.

I think what he is saying is they are stateless so you can edit its previous repsonses and it just goes with it.

If you build a small ui that lets you edit the models response too it’s pretty funny to do.

It sees that it “said” it and gets very confused.

Re: Refusal in Language Models Is Mediated by a Single Direction

#27

Post by theendisney »

I keep thinking of reeducation camps. For some reason the "safety" concept snaps right on. If one is to argue the result beneficial or desirable seems to change nothing to the concept.

If you are going to prevent some-things we "know" are bad and your method is "known" to belong on that list the best you can hope for is a pyrrhic victory.

If we anticipate the worse case scenario on both ends the conclusion must be that we are terrible at such predictions.

But hey, if we let money guide us at least some will be happy with the result.

Re: Refusal in Language Models Is Mediated by a Single Direction

#28

Post by washadjeffmad »

cbg0 wrote: post #24
NewsaHackO wrote: post #13

Earlier quoted context omitted.

Yea, I was asking a SOTM about copy.fail, and it was freaking out, and tried to indirectly call me a hacker a few times. Weirdly, all I did was slightly reword requests, and they all went through. Granted, I am not actually a hacker, so I guess my follow-up questions made it realize that I am asking for educational purposes, but it was definitely the most accusatory, curt, and outright abrasive I have seen an LLM beh…

The biggest problem isn't the token slot machine refusing to give you the answer, but the fact that multiple refusals can end up flagging your account and getting banned from the service.

While contributing to a friend's Remembrance research, I was pretty surprised when Gemini Pro suddenly refused to answer any more questions about photos from the Höcker Album after it spotted an "SS" insignia.

Ironically, the justification it gave was that it wasn't its fault because it was just following orders. I hope this hasn't landed me on Google's list of undesirables.

Grok, for better or worse, didn't seem to mind.

Re: Refusal in Language Models Is Mediated by a Single Direction

#29

Post by jbritton »

I’m sick of LLM refusals. I think there are extremely few things they should refuse, like maybe making nuclear weapons or something along those lines. Once you put people in charge of deciding what you shouldn’t be allowed to see that list will grow and grow.

Re: Refusal in Language Models Is Mediated by a Single Direction

#30

Post by _blop »

Even if you abliterate your model using the old abliteration script or the newer heretic, I found that the models still feel somewhat censored as they purposefully avoid using specific styles and vocabulary, as if Deepmind/Qwen et al have entirely stripped or replaced "bad" words or texts from their corpus of training data.

A related blog post (https://news.ycombinator.com/item?id=47842021) discussed this and termed it "flinching". I wonder if this flinching could also be "mediated by a single direction" or if it can only be fixed by finetuning on a more extensive text corpus.

Post reply on HN