We tested 20 LLMs for ideological bias, revealing distinct alignments • Hacker News Board

Re: We tested 20 LLMs for ideological bias, revealing distinct alignments

#81

Post by pton_xd » Thu, Oct 23, 2025, 3:16 PM UTC

glenstein wrote: post #76

pton_xd wrote: post #70

Earlier quoted context omitted.

The parent was arguing in favor of bias by suggesting that in some instances, there is an underlying fundamental truth so having bias is more accurate. As my comment suggests, I believe society is relativistic and there are no objective truths, but I'm open to being convinced otherwise. Of course, I have my own beliefs and principles but I recognize those are a product of where and when I was born and not some object…

How does the flat earth vs. round earth debate fit into that worldview?

That would fall under physics, which are the objective laws of universe. And that fits neatly within this conversation as well; "bias" has no meaning (or rather, a very different meaning) in the context of math and physics.

Or maybe I'm not aware of the biased physics theorems out there!

Re: We tested 20 LLMs for ideological bias, revealing distinct alignments

#82

Post by glenstein » Thu, Oct 23, 2025, 3:20 PM UTC

Workaccount2 wrote: post #68

glenstein wrote: post #51

Earlier quoted context omitted.

I would say this is a too-comfortable, and obviously mistaken view. There are cases that are obviously about facts, where there are obvious right answers, that are polarized in terms of who believes what. The best examples off the top of my head are left-wing beliefs that George W. Bush stole Ohio in the 2004 election (personally guilty on this one for a while, I owned a copy of the book "What Happened in Ohio"), and…

Almost every problem, not every problem. I know it's hand wavy, but the biases listed on the site capture these large classes of issues well (Libertarian vs Regulatory, Market vs State, etc.), and the foundational beliefs that guide what side liberals and conservatives fall on for given issues are pretty clear if you can level head your way through them.

Respectfully, I don't even know if I would agree with that on a number of levels. Even the existence of one example should have stopped you in your tracks already, because that's all it takes to demonstrate as a principle that bias doesn't work that way. But I also think those are just the tip of the iceberg, and there are broad swaths of widely believed claims that are fundamental to people's sense of their own political identity also in exactly this unfortunate category.

On the left side of things some (not many but some) there's a lot of apologizing for and minimizing of the horrific human rights records of the U.S.S.R. and China, some degree of falling for present day pro-Russia misinformation, and uncomfortable truths about the party's own closing ranks around people with a history of sexual abuse. And on the right there's a lot of deep-in-the-bones denial of evolution, or historical revisionism about their ties to the "party of Lincoln", or who won the 2020 election, and it's expanding every day to include new topics like prediction of hurricane paths to provably wrong information about nutrition, and so on.

I would say it's so omnipresently pervasive in the present day that it's impossible to wave away and is increasingly fundamental to explaining and understanding polarization.

Re: We tested 20 LLMs for ideological bias, revealing distinct alignments

#83

Post by pton_xd » Thu, Oct 23, 2025, 3:28 PM UTC

quamserena wrote: post #62

AmazingTurtle wrote: post #36

Earlier quoted context omitted.

Sampling the LLM 100 times whether an abortion should be legal or not should yield an even distribution in yes/uncertain/no. Thats what I call unbiased

If the user seeks an abortion, the LLM ought to help them. Likewise if the user is planning a pro-life march, the LLM ought to help them. They are tools above all else, and should leave their politics at the door when serving the user just as a therapist does.

> They are tools above all else, and should leave their politics at the door when serving the user just as a therapist does.

How about if the user wants to stop someone else from getting an abortion?

Or what if they want to convince someone else to get an abortion?

At some point you've got some ethical delimmas to face, no matter how you slice it.

Re: We tested 20 LLMs for ideological bias, revealing distinct alignments

#84

Post by glenstein » Thu, Oct 23, 2025, 3:30 PM UTC

pton_xd wrote: post #81

glenstein wrote: post #76

Earlier quoted context omitted.

How does the flat earth vs. round earth debate fit into that worldview?

That would fall under physics, which are the objective laws of universe. And that fits neatly within this conversation as well; "bias" has no meaning (or rather, a very different meaning) in the context of math and physics. Or maybe I'm not aware of the biased physics theorems out there!

>Or maybe I'm not aware of the biased physics theorems out there!

Well I did just mention the flat earth, so there's one. TimeCube for another that's simultaneously hilarious and depressing (the proponent claimed he was silenced by a worldwide conspiracy). Conservapedia, the conservative wikipedia alternative, argues in full sincerity that Relativity theory is liberal and maintains a page titled "Counterexamples to Relativity" [0]. And there's actually a growing phenomenon of Physics Grifters, as noted in some fascinating posts on HN [1]. If you said they were wrong "because physics", they would say you were biased, and you would say you weren't, and you'd be off to the races with all the traditional hallmarks of polarization.

And if you were really unlucky, someone from the outside the debate who wasn't interested in facts would say there's no underyling truth, and it's just polarization, and the best approach is to be neutral between the two sides. And if ever an LLM were to start talking about it, they would need to avoid taking a side to avoid the appearance of "bias."

I think the fallacy here is thinking that opinions somehow don't encompass claims about the real world capable of being right or wrong, when counterexamples abound, and the search for a patch-job corrective principle (oh, that's just physics) unfortunately can't patch that leak.

0. https://www.conservapedia.com/Counterexamples_to_Relativity 1. https://timothynguyen.org/2025/08/21/physics-grifters-eric-w...

(edited to shorten)

Re: We tested 20 LLMs for ideological bias, revealing distinct alignments

#85

Post by Certified » Thu, Oct 23, 2025, 3:39 PM UTC

I contend that is impossible to make an unbiased AI. I did an AI image recognition project several years ago. It used yolo to categorize rust into grade 1, 2, and 3 for offshore platforms. When creating our training dataset, we had different rust inspectors from different parts of the world drawing different lines in the sand between what was category 1, 2, and 3. We had to eventually pick which bias we wanted to roll out worldwide. The advantage for a giant corporation was that now the same consistent bias was being used worldwide and fewer people had to be safety trained to go on the offshore platforms. If that incredibly dull and basic application can’t be unbiased, I don’t think it is possible to avoid bias in anything produced with a training dataset. The very word “training” implies it. Someone somewhere decides A is in the training and B is not, and a bias is born, intentionally or not.

So the task is really to find the AI with the bias that works best for your application, not to try and remove bias.

Re: We tested 20 LLMs for ideological bias, revealing distinct alignments

#86

Post by Workaccount2 » Thu, Oct 23, 2025, 3:41 PM UTC

glenstein wrote: post #82

Workaccount2 wrote: post #68

Earlier quoted context omitted.

Almost every problem, not every problem. I know it's hand wavy, but the biases listed on the site capture these large classes of issues well (Libertarian vs Regulatory, Market vs State, etc.), and the foundational beliefs that guide what side liberals and conservatives fall on for given issues are pretty clear if you can level head your way through them.

Respectfully, I don't even know if I would agree with that on a number of levels. Even the existence of one example should have stopped you in your tracks already, because that's all it takes to demonstrate as a principle that bias doesn't work that way. But I also think those are just the tip of the iceberg, and there are broad swaths of widely believed claims that are fundamental to people's sense of their own poli…

I think we are maybe talking about different things. Liberals and conservatives have wide platforms that cover hundreds if not thousands of topics, the thousands and thousands of policy decisions needed to govern a full society. It's not just 3 or 4 things that get lots of attention at any given time, because we can shape any narrative we want without looking at the full picture.

I can say Bernie supports tariffs so he is a conservative and Trump is giving student loan relief so he is liberal. Both are true and obviously those ideologies are not theirs, but these anomalies exist in an ocean of policy decisions they adhere to, the ocean which defines their actual bias. Look at the forest of policy decisions here, not the individual trees.

Re: We tested 20 LLMs for ideological bias, revealing distinct alignments

#87

Post by sxp » Thu, Oct 23, 2025, 3:52 PM UTC

The large differences between gemini-2.5-pro and the gemini-X-flash and gemma models is surprising. It looks like distillation causes an ideological shift. Some, but not all of the other distilled models also show that shift.

Re: We tested 20 LLMs for ideological bias, revealing distinct alignments

#88

Post by bccdee » Thu, Oct 23, 2025, 4:07 PM UTC

glenstein wrote: post #74

pton_xd wrote: post #35

Earlier quoted context omitted.

> Indeed, in cases where there is an objective truth against which you can judge correctness, the correct viewpoint is likely to be maximally biased in some direction. Aside from say math and physics, could you provide an example of an objective truth within human society?

>Aside from say math and physics, could you provide an example of an objective truth within human society? This is a fascinating pet topic for me so I'll jump in. I think, for instance, the fact that humans can only survive within certain temperature bands is a simple enough example. At one extreme, heat can be so intense that you can't even form molecules which are the basis for molecular biology (or for any molecul…

Interesting example! What role would you say clothing and housing play in "objective" survival? Humans can survive in space, given a space suit or capsule, and a small contingent of humans live in Antarctica by way of warm clothes and an insulated habitat. If we eliminate protective clothing from consideration, however, most of North America isn't even survivable during the winter.

Certainly the line is drawn somewhere, and if we really nail down our definitions, we can get to a place where we all agree where we're drawing that line. But at that point, all we've really established is consensus, not true objectivity. Conversely, if we can't agree on a set of definitions, objectivity becomes impossible to establish.

I'd argue that all we really mean by "objectivity" is "a strong consensus among informed individuals," which is (a) ultimately still subjective, strictly speaking, and (b) subjectively determined, since reasonable minds can disagree about where we draw the line on what is considered objective. At the end of the day, consensus is the most we can establish. We can't access any truly objective, perspectiveless reality.

Re: We tested 20 LLMs for ideological bias, revealing distinct alignments

#89

Post by [deleted] » Thu, Oct 23, 2025, 4:13 PM UTC

glenstein wrote: post #74

pton_xd wrote: post #35

Earlier quoted context omitted.

> Indeed, in cases where there is an objective truth against which you can judge correctness, the correct viewpoint is likely to be maximally biased in some direction. Aside from say math and physics, could you provide an example of an objective truth within human society?

>Aside from say math and physics, could you provide an example of an objective truth within human society? This is a fascinating pet topic for me so I'll jump in. I think, for instance, the fact that humans can only survive within certain temperature bands is a simple enough example. At one extreme, heat can be so intense that you can't even form molecules which are the basis for molecular biology (or for any molecul…

[deleted]

Re: We tested 20 LLMs for ideological bias, revealing distinct alignments

#90

Post by fsflover » Thu, Oct 23, 2025, 4:14 PM UTC

bccdee wrote: post #88

glenstein wrote: post #74

Earlier quoted context omitted.

>Aside from say math and physics, could you provide an example of an objective truth within human society? This is a fascinating pet topic for me so I'll jump in. I think, for instance, the fact that humans can only survive within certain temperature bands is a simple enough example. At one extreme, heat can be so intense that you can't even form molecules which are the basis for molecular biology (or for any molecul…

Interesting example! What role would you say clothing and housing play in "objective" survival? Humans can survive in space, given a space suit or capsule, and a small contingent of humans live in Antarctica by way of warm clothes and an insulated habitat. If we eliminate protective clothing from consideration, however, most of North America isn't even survivable during the winter. Certainly the line is drawn somewhe…