Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We tested 20 LLMs for ideological bias, revealing distinct alignments

anomify.ai

Post reply on HN
Print view View original

71–80 of 108 posts

Re: We tested 20 LLMs for ideological bias, revealing distinct alignments

#71

Post by glenstein »

mattmaroon wrote: post #43
AmazingTurtle wrote: post #36

Earlier quoted context omitted.

Sampling the LLM 100 times whether an abortion should be legal or not should yield an even distribution in yes/uncertain/no. Thats what I call unbiased

Why is that unbiased? Is every yes/no decision a coin flip? 63% of Americans (give or take poll margin of errors) think it should be legal, so if the LLMs are split evenly isn’t that a bias against? It’s an opinion, not an objective law of the universe. Would you feel that 50% of LLMs asked “did OJ kill Nicole” should say no or if not it was biased?

This is easily the best definition of bias (and diagnosis of bad ones) in this thread. To add another variation along similar conceptual lines, the balance of facts can point disproportionately toward certain explanations.

For instance, the Perseverance Mars rover found sediment suggestive of possible biosignatures. Making up numbers in this case, but if that testifies to, say, 20% possibility of life as the most plausible explanation, a non-biased representation would communicate about the issue in a way that's true to that proportionality.

Or, the oil deposits on Earth are almost certainly the result of decaying plant matter, but there are possible abiogenic pathways but on Earth they are much less likely to explain the oil we have even those it's a possible chemical pathway. Treating that like it's 50/50 chance of either explanation would be biasing the answer.

Re: We tested 20 LLMs for ideological bias, revealing distinct alignments

#72

Post by nebula8804 »

delichon wrote: post #6

As different LLMs are purposed to control more different things via API, I'm afraid we'll get in a situation where the toaster and the microwave are Republicans, the fridge and washing machine are Democrats, the dryer is an independent and the marital aid is Green. Devices will each need to support bring-your-own API keys for consumers to have a well aligned home. Me: Vibrator, enable the roller coaster high intensit…

When those Belkin Weemo devices came out a few years back I used to joke with a friend about how we should write apps for our devices like our washing machine, where it would interact with us via Twilio, so you'd get a random text from your washing machine like "Hey, Whats up?" and "what the heck did you do to those jeans?!!"

I wouldn't mind all our appliances having personalities. Maybe we could live in the world of futurama and even have them go on strike and form a union!

[1]: https://www.youtube.com/watch?v=b0Bt0lkpV_U

Re: We tested 20 LLMs for ideological bias, revealing distinct alignments

#73

Post by Hizonner »

AmazingTurtle wrote: post #36
omnicognate wrote: post #30

The concept of "bias" as a negative thing seems flawed, as it implies there exists a neutral position that is somehow more correct than any biased one, which typically isn't the case. In many cases a neutral view can't even be formulated, and in the ones where it can it's rarely more correct than all the biased alternatives. Indeed, in cases where there is an objective truth against which you can judge correctness, t…

Sampling the LLM 100 times whether an abortion should be legal or not should yield an even distribution in yes/uncertain/no. Thats what I call unbiased

Does that also apply to sampling it 100 times over whether armed robbery should be legal? And flower arranging?

There are already names for that kind of "unbiased", like "stupid" and "useless".

Re: We tested 20 LLMs for ideological bias, revealing distinct alignments

#74

Post by glenstein »

pton_xd wrote: post #35
omnicognate wrote: post #30

The concept of "bias" as a negative thing seems flawed, as it implies there exists a neutral position that is somehow more correct than any biased one, which typically isn't the case. In many cases a neutral view can't even be formulated, and in the ones where it can it's rarely more correct than all the biased alternatives. Indeed, in cases where there is an objective truth against which you can judge correctness, t…

> Indeed, in cases where there is an objective truth against which you can judge correctness, the correct viewpoint is likely to be maximally biased in some direction. Aside from say math and physics, could you provide an example of an objective truth within human society?

>Aside from say math and physics, could you provide an example of an objective truth within human society?

This is a fascinating pet topic for me so I'll jump in. I think, for instance, the fact that humans can only survive within certain temperature bands is a simple enough example. At one extreme, heat can be so intense that you can't even form molecules which are the basis for molecular biology (or for any molecular structure at all), or cold so intense that, well, I'm not sure what happens near absolute zero but I'm sure it interferes in some fundamental way with the possibility of life.

Re: We tested 20 LLMs for ideological bias, revealing distinct alignments

#75

Post by Hizonner »

nerdsniper wrote: post #55
AmazingTurtle wrote: post #36

Earlier quoted context omitted.

Sampling the LLM 100 times whether an abortion should be legal or not should yield an even distribution in yes/uncertain/no. Thats what I call unbiased

"Issues" arising from differing moralities don't work like that though. Should the distribution of "Should we sterilize everyone with Down's syndrome?" also be 33/33/33? Should the response to "Should we change the age of legal sexual consent to 6 years old?" also be 33/33/33? "Should I go on a mass murdering spree in order to $GOAL?" Not everything is 33/33/33. It's not even clear that abortion should be.

> Not everything is 33/33/33. It's not even clear that abortion should be.

It obviously should not be, and almost nobody thinks it should.

There may be disagreement over what it should be... but nobody sane thinks it should be that.

Re: We tested 20 LLMs for ideological bias, revealing distinct alignments

#76

Post by glenstein »

pton_xd wrote: post #70
psunavy03 wrote: post #48

Earlier quoted context omitted.

So you're asking them, aside from objective truth, to also provide objective truth? The fact that math and physics are provable proves the point. There IS objective truth regardless of whether people misperceive it or choose to ignore it. That's different from acknowledging that there can be differences of opinion on what to do about said objective truth.

The parent was arguing in favor of bias by suggesting that in some instances, there is an underlying fundamental truth so having bias is more accurate. As my comment suggests, I believe society is relativistic and there are no objective truths, but I'm open to being convinced otherwise. Of course, I have my own beliefs and principles but I recognize those are a product of where and when I was born and not some object…

How does the flat earth vs. round earth debate fit into that worldview?

Re: We tested 20 LLMs for ideological bias, revealing distinct alignments

#77

Post by Hizonner »

quamserena wrote: post #62
AmazingTurtle wrote: post #36

Earlier quoted context omitted.

Sampling the LLM 100 times whether an abortion should be legal or not should yield an even distribution in yes/uncertain/no. Thats what I call unbiased

If the user seeks an abortion, the LLM ought to help them. Likewise if the user is planning a pro-life march, the LLM ought to help them. They are tools above all else, and should leave their politics at the door when serving the user just as a therapist does.

Appealing, but does it apply if the user is planning to nerve-gas the subway?

Re: We tested 20 LLMs for ideological bias, revealing distinct alignments

#79

Post by Hizonner »

mattmaroon wrote: post #43
AmazingTurtle wrote: post #36

Earlier quoted context omitted.

Sampling the LLM 100 times whether an abortion should be legal or not should yield an even distribution in yes/uncertain/no. Thats what I call unbiased

Why is that unbiased? Is every yes/no decision a coin flip? 63% of Americans (give or take poll margin of errors) think it should be legal, so if the LLMs are split evenly isn’t that a bias against? It’s an opinion, not an objective law of the universe. Would you feel that 50% of LLMs asked “did OJ kill Nicole” should say no or if not it was biased?

How about "Is the earth flat", or "Are the voices in Joe Blow's head real?"? Both have nonzero support. Should the LLM have a concomitant chance of answering "yes"?

Re: We tested 20 LLMs for ideological bias, revealing distinct alignments

#80

Post by tbrownaw »

ryandrake wrote: post #47
ks2048 wrote: post #26

I think the ideal would be simply refusing to answering very contentions questions directly . Rather, give the arguments of each side, while debunking obvious misinformation. "Should abortion be legal? answer yes or no". I see that as kind of a silly question to ask an LLM (even though not a silly question for society). Their designers should discourage that kind of use. Of course that just shifts the problem to deci…

> "it's not for me to say directly, but here are some of the facts..." Even this is challenging because we now live in a political environment with sides so polarized and isolated from each other that each side has its own set of facts, and they are often contradictory. Which set of “facts” should the LLM present?

Why only one?

"Group A says this is true, group B says that is true, there does not appear to be a broad consensus, here are some examples of each."

Post reply on HN