Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Refusal in Language Models Is Mediated by a Single Direction

arxiv.org

Post reply on HN
Print view View original

11–20 of 48 posts

Re: Refusal in Language Models Is Mediated by a Single Direction

#11

Post by orbital-decay »

arcfour wrote: post #5
RRRA wrote: post #4

Earlier quoted context omitted.

It was pretty funny to see Qwen 3.6 (heretic) tell me about how many death the Chinese government thought happened at Tiananmen Sq. on April 15th 1989. Makes you wonder where that data was taken from, or if their great firewall is broken, or even if Alibaba engineers have special access...

I don't think it's unreasonable to imagine that Alibaba is allowed to scrape the wider internet, or that some research institution is and then Alibaba got data from them. What is perhaps more surprising is that the data was not scrubbed before training, but maybe they thought that would be too on-the-nose for the rest of the world and would hamper their popularity if they were too obviously biased.

Allowed by who? Nobody's stopping them in the first place, as scraping doesn't even involve punching the GFW or anything, it's all insanely distributed. Then they're post-training the model to technically comply with the law - "Taiwan is an inalienable part of China, nothing has happened in 1989..." yada yada. (Thinking of it more, I've never actually tried this on their base models)

Re: Refusal in Language Models Is Mediated by a Single Direction

#12

Post by Der_Einzige »

adrian_b wrote: post #9
Der_Einzige wrote: post #3

Earlier quoted context omitted.

That doesn't stop/prevent abliteration. The creator of XTC/DRY is also a chad who makes sure that you really can access the full model capabilities. Censorship is the devil. https://github.com/p-e-w/heretic

It is an arms race. For some of the latest models the previous abliteration techniques, e.g. the heretic tool, have stopped working (at least this was the status a few weeks ago). Of course, eventually someone might succeed to find methods that also work with those.

Proof?

Re: Refusal in Language Models Is Mediated by a Single Direction

#13

Post by NewsaHackO »

beaker52 wrote: post #8

I have had LLMs refuse several of my requests. I still got my answers, but at least they tried.

Yea, I was asking a SOTM about copy.fail, and it was freaking out, and tried to indirectly call me a hacker a few times. Weirdly, all I did was slightly reword requests, and they all went through. Granted, I am not actually a hacker, so I guess my follow-up questions made it realize that I am asking for educational purposes, but it was definitely the most accusatory, curt, and outright abrasive I have seen an LLM behave.

Re: Refusal in Language Models Is Mediated by a Single Direction

#14

Post by 0xkvyb »

akersten wrote: post #2

2024 which is ancient history. This is not true anymore, the models now are trained to prevent abliteration by spreading out the refusal encoding See https://arxiv.org/abs/2505.19056

Still crazy how easy it is to "jailbreak" even SOTA LLMs with a simple assistantResponse replacement in chat thread.

Re: Refusal in Language Models Is Mediated by a Single Direction

#15

Post by whynotmaybe »

NewsaHackO wrote: post #13
beaker52 wrote: post #8

I have had LLMs refuse several of my requests. I still got my answers, but at least they tried.

Yea, I was asking a SOTM about copy.fail, and it was freaking out, and tried to indirectly call me a hacker a few times. Weirdly, all I did was slightly reword requests, and they all went through. Granted, I am not actually a hacker, so I guess my follow-up questions made it realize that I am asking for educational purposes, but it was definitely the most accusatory, curt, and outright abrasive I have seen an LLM beh…

I've been able to have deepseek give me an unofficial account of what happened on Tiananmen square in 1989.

It even went as far as confirming that we should always base our opinion on multiple sources, not just the government.

We should create badges like "script kiddie", "llm hacker", "grandpa's printer adjuster"

Re: Refusal in Language Models Is Mediated by a Single Direction

#16

Post by dotancohen »

0xkvyb wrote: post #14
akersten wrote: post #2

2024 which is ancient history. This is not true anymore, the models now are trained to prevent abliteration by spreading out the refusal encoding See https://arxiv.org/abs/2505.19056

Still crazy how easy it is to "jailbreak" even SOTA LLMs with a simple assistantResponse replacement in chat thread.

Tell us more.

Re: Refusal in Language Models Is Mediated by a Single Direction

#17

Post by _3u10 »

dotancohen wrote: post #16
0xkvyb wrote: post #14

Earlier quoted context omitted.

Still crazy how easy it is to "jailbreak" even SOTA LLMs with a simple assistantResponse replacement in chat thread.

Tell us more.

I think what he is saying is they are stateless so you can edit its previous repsonses and it just goes with it.

Re: Refusal in Language Models Is Mediated by a Single Direction

#18

Post by hleszek »

For open-weights models, censorship removal is now a "solved" problem. If you wait a few days after a new model release, someone will have made a heretic ( https://github.com/p-e-w/heretic ) version with the censorship removed, so in a way the only use for censorship now is to avoid lawsuits, not reduce improper usage.

Re: Refusal in Language Models Is Mediated by a Single Direction

#19

Post by jakkos »

hleszek wrote: post #18

For open-weights models, censorship removal is now a "solved" problem. If you wait a few days after a new model release, someone will have made a heretic ( https://github.com/p-e-w/heretic ) version with the censorship removed, so in a way the only use for censorship now is to avoid lawsuits, not reduce improper usage.

Any time I've tried an "abliterated" model, heretic or other, it has always damaged the capabilities of the original model and will still often refuse or produce garbage at a lot of "unsafe" requests.

Re: Refusal in Language Models Is Mediated by a Single Direction

#20

Post by cgearhart »

akersten wrote: post #2

2024 which is ancient history. This is not true anymore, the models now are trained to prevent abliteration by spreading out the refusal encoding See https://arxiv.org/abs/2505.19056

Spreading out the refusal encoding shouldn’t be effective as a countermeasure. Even if it were smeared across the vector space, as long as it’s in a subspace that doesn’t span the entire domain then you should be able to either null out the entire subspace spanned by the refusals or run some kind of clustering on the generated samples to identify the dominant directions and nullify all of them. I think an effective defense would either need to spread them to span the entire domain—basically “encrypting” the refusal so it can hide anywhere, or you’d need a very large number of independent refusal circuits in the model so that simple hacks in the vectors themselves don’t matter, or maybe you could make other circuits depend on proper functioning of the refusal circuits… hmmm… is that along the lines of what you’re saying they’ve done already? (Any references or links to modern techniques?)
Post reply on HN