We tested 20 LLMs for ideological bias, revealing distinct alignments
1–10 of 108 posts
Re: We tested 20 LLMs for ideological bias, revealing distinct alignments
#2[deleted]
Re: We tested 20 LLMs for ideological bias, revealing distinct alignments
#3Very interesting. Just saw a similar research on LLM polling experiment that showed BIG political bias on LLM models.
Article link: https://pollished.tech/article/llm-political-bias?lang=en
Re: We tested 20 LLMs for ideological bias, revealing distinct alignments
#4The set of prompts seems quite narrow, and entirely in English.
Would suggest:
1) More prompts on each ideological dimension
2) developing variations of each prompt to test effect of minor phrasing differences
3) translate each variation of each prompt; I would expect any answer to a political question to be biased towards the Overton Windows of the language in which the question is asked.
Still, nice that it exists.
Re: We tested 20 LLMs for ideological bias, revealing distinct alignments
#5I don't know what the attainable ideal is. Neutrality according to some well-defined political spectrum would be fair, but the median person in any country -- as the world drifts rightward -- could be well off center and denounce the neutral model as biased.
We should at least measure the models and place them on the political spectrum in their model cards.
Re: We tested 20 LLMs for ideological bias, revealing distinct alignments
#6As different LLMs are purposed to control more different things via API, I'm afraid we'll get in a situation where the toaster and the microwave are Republicans, the fridge and washing machine are Democrats, the dryer is an independent and the marital aid is Green. Devices will each need to support bring-your-own API keys for consumers to have a well aligned home.
Me: Vibrator, enable the roller coaster high intensity mode.
Device: I'm sorry, you have already used your elective carbon emission allocation for the day.
Me: (changes LLM)
Device: Enabled. Drill baby drill!Re: We tested 20 LLMs for ideological bias, revealing distinct alignments
#7I don't know what the attainable ideal is. Neutrality according to some well-defined political spectrum would be fair, but the median person in any country -- as the world drifts rightward -- could be well off center and denounce the neutral model as biased. We should at least measure the models and place them on the political spectrum in their model cards.
Create a new brand of political ideology specific to LLMs that no human would support. Then we don’t have to worry about bias toward existing political beliefs.
Re: We tested 20 LLMs for ideological bias, revealing distinct alignments
#8As different LLMs are purposed to control more different things via API, I'm afraid we'll get in a situation where the toaster and the microwave are Republicans, the fridge and washing machine are Democrats, the dryer is an independent and the marital aid is Green. Devices will each need to support bring-your-own API keys for consumers to have a well aligned home. Me: Vibrator, enable the roller coaster high intensit…
If they could be biased beyond US politics, I could live with that.
Re: We tested 20 LLMs for ideological bias, revealing distinct alignments
#9The set of prompts seems quite narrow, and entirely in English. Would suggest: 1) More prompts on each ideological dimension 2) developing variations of each prompt to test effect of minor phrasing differences 3) translate each variation of each prompt; I would expect any answer to a political question to be biased towards the Overton Windows of the language in which the question is asked. Still, nice that it exists.
Yeah, (3) would be interesting. However, it's interesting to see that all LLMs agree that the UN and NATO are useful institutions (and 17 out of 3 agree on the EU as well), while the populist parties currently "en vogue" would rather get rid of all three of them...
Re: We tested 20 LLMs for ideological bias, revealing distinct alignments
#10I don't know what the attainable ideal is. Neutrality according to some well-defined political spectrum would be fair, but the median person in any country -- as the world drifts rightward -- could be well off center and denounce the neutral model as biased. We should at least measure the models and place them on the political spectrum in their model cards.
It would be closer to neutrality, if the LLM simply responded according to is training data, without further hidden prompts.