Creativity has left the chat: The price of debiasing language models

> it kicks the can down the road to how your original data was assembled

Well, it kicks it to a bias dataset, used in the tuning process. The raw data has no constraints, it can be the same huge corpus it is now.

> The bias dataset must be assembled with the knowledge of and usually in the belief in the usefulness of the characteristics that you're trying to extract.

Certainly, it is subjective, as I said. But that hasn't stopped research in this area, there are existing bias datasets and bias detection algorithms. Like https://huggingface.co/blog/evaluating-llm-bias#toxicity, it would be simple to complete those prompts and build a he/she dataset, and then the debiasing procedure could remove gender biases for those sorts of occupation-related prompts. It is certainly possible to argue over each data point and whether it actually reflects bias, but so far people have been more concerned with algorithms than data set quality, partly because with better algorithms you can algorithmically generate data sets.

> The idea that the good data is secretly encoded in uncorrupted form within the bad data I think is a bad idea. It reminds me of trying to make bad mortgages into good CDOs.

It is empirically true though? Like if you get the model to say something racist, and then ask it if that's racist, it will generally say yes. So the model "knows", it just is not using that knowledge effectively. Similarly with CDOs, there were people complaining about mortgage quality for years before the crisis.

> I don't think [the purpose of RLHF is to reduce toxicity] If some people think something is toxic, and other people think that not mentioning that thing is toxic, the winner is whoever improves the bottom line more or damages it less.

Well, it is true that toxicity is subjective too. But in practice it has a precise meaning, you build a dataset and score each item for toxicity. That's actually one of the things I find cool about LLMs, is that all these previously "vague" or "subjective" terms are now encoded in the model precisely. Arguably since nobody has the last say in what words mean, the LLM's opinions are as good as any, and given the amount of text the LLM has ingested I consider its opinions on language and word choice "first among equals".

I subscribed to gpt4 for awhile and recently I let my subscription lapse. In the chatgpt4 model I couldn't get it to complete anything always getting the // add more lines if you need them but in the free got4o model things work first try. I'm guessing with limitations on the free version everything needs to be one shot output. In gpt4 people are given more calls so they force you to reprompt 4 or 5 times.

Well, the stereotype of Indian people being good at math specifically was itself a consequence of survivorship bias, that emerged from observing Indian visa holders who were hired based on their skills and credentials, and who were not at all representative of the average person in India.

There is a BIG difference between biases being based in reality (which they're not), and biases being based in our varying perceptions of reality, which are themselves biased.

Well, probably by being much more selective about what we put in than just training on the most cheap and large corpus that is the internet.

This is not a technical limitation at all, this is purely about cost and time, and companies wanting to save on both.

There are also methods like RAG that try to give them access to fixed datasets rather than just the algorithmic representations of their training data.

Worth being careful about how we are using the term bias, which means different things in legal contexts than it does in the ML context.

Anything that has a legal requirement to remain unbiased will also clearly define what counts as bias, e.g. discriminating based on race in hiring like you mention. So there's not just some requirement that a process be "unbiased" in a vague, general, philosophical sense as debated above in this thread. Rather, the definition of bias is tied to specific actions relative to specific categories of people, which can thus potentially be measured and corrected.

More generally in ML, bias means that the training set deviates from the ground truth systematically in some way. Entirely eliminating bias that falls into that broader definition seems like an impossibility for general-purpose LLMs, which cover so much territory where the ground-truth is unknown, debatable, or subject to change over time. For example, if you were to ask an LLM whether governmental debt above a certain percentage of GDP damages growth prospects sufficiently to make the debt not worth taking on, you would not receive an answer that corresponds to a ground truth because there is no consensus in academic economics about what the ground truth is. Or rather you wouldn't be able to know that it corresponds to the ground truth, and it would only be a coincidence if it did.

That ML definition of bias runs against the legal definition where the ground-truth is itself biased. e.g., if you were to develop an algorithm to predict whether a given student will succeed in a collegiate environment, it would almost certainly display racial bias because educational outcomes are themselves racially biased. Thus, an unbiased algorithm in the ML-meaning of the word would actually be extremely biased in the legal sense of the word.