Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Alignment is not free: How model upgrades can silence your confidence signals

variance.co

Post reply on HN
Print view View original

21–30 of 71 posts

Re: Alignment is not free: How model upgrades can silence your confidence signals

#21

Post by exe34 »

andai wrote: post #16
exe34 wrote: post #15

Earlier quoted context omitted.

> it’s it similar to humans. when restricted in terms of what they can or cannot say, they become more conservative and cannot really express all sorts of ideas. This reminds me of the time when I was a child, and my parents decreed that all communications would henceforth happen in English. I became selectively mute. I responded yes/no, and had nothing further to add and ventured no further information. The decree l…

What did you use to communicate before that? Were you fluent in English?

No, it was a local creole. And no, I was learning it at school.

Re: Alignment is not free: How model upgrades can silence your confidence signals

#22

Post by im3w1l »

andai wrote: post #17
khafra wrote: post #11

Earlier quoted context omitted.

"LLM whisperer" folks will confidently claim that base models are substantially smarter than fine-tuned chat models; with qualitative differences in capabilities. But you have to be an LLM whisperer to get useful work out of a base model, since they're not SFT'ed, RLHF'ed, or RLAIF'ed into actually wanting to help you.

How can I learn more about this? Is it like in the early GPT-3 days, when you had to give it a bunch of examples and hope it catches the pattern?

Back in those days I would either create a little scene with a knowledgeable person and someone with a question. Or I would start writing a monologue and generate a continuation for it.

Re: Alignment is not free: How model upgrades can silence your confidence signals

#26

Post by itchyjunk »

mlin4589 wrote: post #5
Centigonal wrote: post #4

Very interesting! The one thing I don't understand is how the author made the jump from "we lost the confidence signal in the move to 4.1-mini" and "this is because of the alignment/steerability improvements." Previous OpenAI models were instruct-tuned or otherwise aligned, and the author even mentions that model distillation might be destroying the entropy signal. How did they pinpoint alignment as the cause?

Good question! We do know from OpenAI's system card from GPT-4 that the post-trained RLHF model is significantly less calibrated compared to the pre-trained model, so it's a matter of speculation that something similar is occurring. However, it's more of a hunch more than anything. I would be curious if it's possible to reproduce this behavior, or the impact of distillation on calibration. Disclaimer: I wrote this bl…

Could you please elaborate what less or more calibrated means here? Thanks!

Re: Alignment is not free: How model upgrades can silence your confidence signals

#27

Post by sega_sai »

cyanydeez wrote: post #25
sega_sai wrote: post #23

Can we have models also return a probability, reflecting how accurate the statements it made is ?

Sure, but then you need probability stats on the probability stats.

I am not sure what you mean. The idea is that the network should return the text, and a confidence expressed as probability. When trained, the log-score should be optimized. (i'm not sure it would actually work given how the training is structured, but something like this would be useful)

Re: Alignment is not free: How model upgrades can silence your confidence signals

#28

Post by Scene_Cast2 »

itchyjunk wrote: post #26
mlin4589 wrote: post #5

Earlier quoted context omitted.

Good question! We do know from OpenAI's system card from GPT-4 that the post-trained RLHF model is significantly less calibrated compared to the pre-trained model, so it's a matter of speculation that something similar is occurring. However, it's more of a hunch more than anything. I would be curious if it's possible to reproduce this behavior, or the impact of distillation on calibration. Disclaimer: I wrote this bl…

Could you please elaborate what less or more calibrated means here? Thanks!

For binary labels: you take a slice of labeled data. The mean of the ML model prediction on this data is different from the mean of the label. In practice, often a synonym for "loss is worse / could be better".

Not sure if that's what the GP meant, I only worked with binary labels stuff.

Re: Alignment is not free: How model upgrades can silence your confidence signals

#29

Post by malfist »

hansvm wrote: post #7
malfist wrote: post #6

Earlier quoted context omitted.

How are you defining "creativity" in context with a statistical model?

> defined as syntactic and semantic diversity

That's not creativity, that's entropy.

It would make sense that fine tuning and alignment reduce diversity in the response, that's the goal.

Re: Alignment is not free: How model upgrades can silence your confidence signals

#30

Post by redman25 »

sega_sai wrote: post #27
cyanydeez wrote: post #25

Earlier quoted context omitted.

Sure, but then you need probability stats on the probability stats.

I am not sure what you mean. The idea is that the network should return the text, and a confidence expressed as probability. When trained, the log-score should be optimized. (i'm not sure it would actually work given how the training is structured, but something like this would be useful)

It's not that simple how would the model know when it knows? Removing hallucination has to be a post-training thing because you need to test the model against what it actually knows first in order to provide training examples of what it knows and doesn't know and how to respond in those circumstances.
Post reply on HN