Earlier quoted context omitted.
> > The goal is to be as unbiased as possible given that limited amount of effort. > So you are therefore biased Yes, but significantly less than before. Which is the goal. > You have a finite set of resources, and you are choosing to allocate them in a particular way. That is bias. That "particular way" is to give more weight to opinions that are under-represented in your training data and give less weight to opinio…
I object specifically to this: > Saying "there's no way to avoid the bias" is just an excuse to get away with being biased And specifically to the implied idea that there is an unbiased target that is achievable.
I don't think we can achieve 0% bias, I agree with you on that. But I think that, if you decide to spend some amount of effort (let's call this amount "X"), it is possible to reduce bias compared to if you spent zero amount of effort.
And that furthermore, if you spend a "Y" amount of effort where XObviously, at a certain point this would have diminishing returns, so presumably there is some sweet point where, even though you can't be 100% unbiased, you can at least say you made a reasonable effort to be unbiased and that your remaining sources of bias are unintentional (and probably, almost just as likely to go in one direction vs another).
To bring the conversation back to the original topic, I think ChatGPT/InstructGPT is actually being actively biased towards one political side as a side effect of RLHF being done by people from OpenAI, even if this bias is being introduced unintentionally.
It would be much better, for example, if OpenAI could somehow accomplish RLHF using a sample of its users as the AI trainers.
It would still be a far cry from reaching 0% bias asymptotically, but it would already be an improvement, I think, as I think its users are a more representative sample of the population than OpenAI employees.