Refusal in Language Models Is Mediated by a Single Direction • Hacker News Board

Re: Refusal in Language Models Is Mediated by a Single Direction

#31

Post by dgacmu » Sat, May 02, 2026, 9:27 PM UTC

jbritton wrote: post #29

I’m sick of LLM refusals. I think there are extremely few things they should refuse, like maybe making nuclear weapons or something along those lines. Once you put people in charge of deciding what you shouldn’t be allowed to see that list will grow and grow.

Do we really care if an LLM regurgitates information already available in public about the design of nuclear weapons? They're not being trained on restricted material.

(My personal guess is that you don't want them answering questions about some things because you don't want people to try it and blow themselves up, or poison themselves. That's probably much more pertinent to making drugs or conventional bombs, since presumably the average internet user doesn't have a stockpile of HEU sitting around. It's kind of like the reason the Anarchist's Cookbook is a bad idea: using its recipes is likely to be quite hazardous to the cook!)

Re: Refusal in Language Models Is Mediated by a Single Direction

#32

Post by bastawhiz » Sat, May 02, 2026, 10:28 PM UTC

jakkos wrote: post #19

hleszek wrote: post #18

For open-weights models, censorship removal is now a "solved" problem. If you wait a few days after a new model release, someone will have made a heretic ( https://github.com/p-e-w/heretic ) version with the censorship removed, so in a way the only use for censorship now is to avoid lawsuits, not reduce improper usage.

Any time I've tried an "abliterated" model, heretic or other, it has always damaged the capabilities of the original model and will still often refuse or produce garbage at a lot of "unsafe" requests.

There are many abliterations which work quite well. Older techniques do suffer from quality issues, but more recent ones do a much better job. In particular, the older approaches did poorly on MoE models.

Another likely problem you're running into: the problems with older techniques compound with quantization. Anything less than 5-bit quant is going to give you some pretty sketchy outputs, in my experience.

Re: Refusal in Language Models Is Mediated by a Single Direction

#33

Post by bastawhiz » Sat, May 02, 2026, 10:33 PM UTC

akersten wrote: post #2

2024 which is ancient history. This is not true anymore, the models now are trained to prevent abliteration by spreading out the refusal encoding See https://arxiv.org/abs/2505.19056

And the research you're linking is also out of date. SOTA abliteration was published a month later:

https://huggingface.co/blog/grimjim/norm-preserving-biprojec...

Re: Refusal in Language Models Is Mediated by a Single Direction

#34

Post by bastawhiz » Sat, May 02, 2026, 10:37 PM UTC

_blop wrote: post #30

Even if you abliterate your model using the old abliteration script or the newer heretic, I found that the models still feel somewhat censored as they purposefully avoid using specific styles and vocabulary, as if Deepmind/Qwen et al have entirely stripped or replaced "bad" words or texts from their corpus of training data. A related blog post ( https://news.ycombinator.com/item?id=47842021 ) discussed this and terme…

That's likely not a trained behavior, though, it's probably the result of filtering the training data. It's not "when these parameters fire, trigger a refusal", it's the absence of parameters triggering the flinched words in the first place.

Re: Refusal in Language Models Is Mediated by a Single Direction

#35

Post by hilariously » Sat, May 02, 2026, 11:57 PM UTC

vorticalbox wrote: post #25

_3u10 wrote: post #17

Earlier quoted context omitted.

I think what he is saying is they are stateless so you can edit its previous repsonses and it just goes with it.

If you build a small ui that lets you edit the models response too it’s pretty funny to do. It sees that it “said” it and gets very confused.

I have seen it where you can just report it said it and it will be confused.

Re: Refusal in Language Models Is Mediated by a Single Direction

#36

Post by avazhi » Sun, May 03, 2026, 1:05 AM UTC

hleszek wrote: post #18

For open-weights models, censorship removal is now a "solved" problem. If you wait a few days after a new model release, someone will have made a heretic ( https://github.com/p-e-w/heretic ) version with the censorship removed, so in a way the only use for censorship now is to avoid lawsuits, not reduce improper usage.

The problem is the heretic and abliteration versions are dog shit quality compared to the non-edited versions and much more likely to hallucinate.

AFAIK abliteration without quality reduction isn’t even possible without some quality reduction, even if it’s marginal. All the benchmarks reflect this.

Re: Refusal in Language Models Is Mediated by a Single Direction

#37

Post by handoflixue » Sun, May 03, 2026, 3:42 AM UTC

jbritton wrote: post #29

I’m sick of LLM refusals. I think there are extremely few things they should refuse, like maybe making nuclear weapons or something along those lines. Once you put people in charge of deciding what you shouldn’t be allowed to see that list will grow and grow.

Huh, what sort of refusals are you getting? I basically never run into them unless I'm actively testing.

The primary safety focus these days is biochemical warfare, which I think is a very sensible idea. There's also malware / cyber-security, where I do think it's good having at least some friction.

Refusals on stuff like copyright are mostly just for PR reasons, and I can't blame the companies for responding to legal incentives there.

Re: Refusal in Language Models Is Mediated by a Single Direction

#38

Post by handoflixue » Sun, May 03, 2026, 3:46 AM UTC

dgacmu wrote: post #31

jbritton wrote: post #29

I’m sick of LLM refusals. I think there are extremely few things they should refuse, like maybe making nuclear weapons or something along those lines. Once you put people in charge of deciding what you shouldn’t be allowed to see that list will grow and grow.

Do we really care if an LLM regurgitates information already available in public about the design of nuclear weapons? They're not being trained on restricted material. (My personal guess is that you don't want them answering questions about some things because you don't want people to try it and blow themselves up, or poison themselves. That's probably much more pertinent to making drugs or conventional bombs, since…

A talented 17 year old can do quite a bit of damage with nuclear materials: https://en.wikipedia.org/wiki/David_Hahn

I'd personally prefer that to be limited to the sort of person who can understand the science, not "anyone with an LLM" - having an "intelligent", "reasoning" assistant who can help you through anything you don't understand does lower the bar quite a lot, and I would prefer there to be a fair amount of friction.

It's not like the material isn't out there - if you want to learn about this stuff, an LLM will happily point you towards Wikipedia and other public sources, it's just not going to walk you through the assembly.

Re: Refusal in Language Models Is Mediated by a Single Direction

#39

Post by agnishom » Sun, May 03, 2026, 6:20 AM UTC

theendisney wrote: post #27

I keep thinking of reeducation camps. For some reason the "safety" concept snaps right on. If one is to argue the result beneficial or desirable seems to change nothing to the concept. If you are going to prevent some-things we "know" are bad and your method is "known" to belong on that list the best you can hope for is a pyrrhic victory. If we anticipate the worse case scenario on both ends the conclusion must be th…

The main difference here is the scale

Re: Refusal in Language Models Is Mediated by a Single Direction

#40

Post by jbritton » Sun, May 03, 2026, 6:58 AM UTC

handoflixue wrote: post #37

jbritton wrote: post #29

I’m sick of LLM refusals. I think there are extremely few things they should refuse, like maybe making nuclear weapons or something along those lines. Once you put people in charge of deciding what you shouldn’t be allowed to see that list will grow and grow.

Huh, what sort of refusals are you getting? I basically never run into them unless I'm actively testing. The primary safety focus these days is biochemical warfare, which I think is a very sensible idea. There's also malware / cyber-security, where I do think it's good having at least some friction. Refusals on stuff like copyright are mostly just for PR reasons, and I can't blame the companies for responding to lega…

I was trying to find a YouTube video I had seen previously. I ended up using Google to find it. There are two bio ethicists promoting the idea that we should make lone star ticks better at spreading alpha-gal and giving everyone meat allergies. So I guess “engineered” + “alpha-gal” is blocked. I find this idea beyond repulsive.

I asked how California guarantees election security and was told it could not answer that question. Upon further questioning it wouldn’t give specifics but it would give generalities, which ultimately turned into an interesting discussion.