Vision Language Models Are Biased

I'm not. I'm referencing the paper - not some hypothetical abstract word problem. Imagine walking into a room, where the pieces are slowly morphing from staid Staunton structures into amorphous blobs of lava lamp Cthulhu nightmares. If a locomotive steam train from Denver passes within 15 meters of the room, how many passengers paid for the tickets using a cashier's check?

Nobody's arguing that humans never take logical shortcuts or that those shortcuts can cause us to make errors.

Some of the rebuttals in this thread are ridiculous. Like what if I forced you to stare at the surface of the sun followed by waterboarding for several hours, and then asked you to look at a 1000 different chess boards. Are you sure you wouldn't make a mistake?

In the paper the various VLLMs are asked to double-check which still didn't make a difference. The argument is more along the lines that VLLMs (and multimodal LLMs) aren't really thinking in the same way that humans do.

And if you REALLY need an example albeit a bit tangential - try this one out. Ask any SOTA (multimodal or otherwise) model such as gpt-image-1, Kontext, Imagen4, etc. for a five-leaf cover. It'll get it about 50% of the time.

Now go and ask any kindergartener for the same thing.

Ish. The catch is we spend a ton of effort on teaching these models to recognize specific things in pictures. Then we ask it to not do that task, but instead count something on the picture. Which, we oddly don't spend a lot of time training the model to do.

It is a lot like the experiment where you ask people to say what color some text is. With the trick where some of the text is the name of another color. Can be surprisingly hard for people that are good at reading.

I tried it with GPT-4o, took the 5-legged zebra example from their github and it answered quite well.

"The animal in the image appears to have five visible legs, but this is an illusion caused by the overlapping of legs and motion blur. Zebras, like all equids, only have four legs."

Not perfect, but also doesn't always regress to the usual answer.

"The animal in the image appears to be an elephant, but it has been digitally altered. It visually shows six legs, although the positioning and blending of shadows and feet are unnatural and inconsistent with real anatomy. This is a visual illusion or manipulation." (actually should say five)

"This bird image has also been manipulated. It shows the bird with three legs, which is anatomically impossible for real birds. Normal birds have exactly two legs." (correct)

"Each shoe in the image has four white stripes visible on the side." (correct)

I agree that models are bad at counting in general, but in this case it could just as easily be ambiguity in the wording of the prompt. The model was shown a 3 legged chicken and asked how many legs "this animal" has. It is reasonable that the model identified a chicken and answered that chickens usually have 2 legs. I would expect the same answer from a human child, adding evidence to my assertion that LLMs are just like toddlers that have read everything on the Internet. They have knowledge but no wisdom.

When you say that the definition I gave of bias is biased (in the sense I defined), what direction does it have a tendency to be wrong in? I assume by “wrong” you mean “not matching how people use the word”?

To clarify, when I said “identifiable”, I didn’t mean “identified”. I meant “in principle possible to identify”. Like, if you have a classifier between inputs where another thing (the thing being judged for bias) gets right answers and inputs where it gets wrong answers, and this classifier is both substantially simpler than the other thing, and gets a significantly better than chance success rate, and like, there is a human comprehensible thing about the inputs that this classifier is basing things on, then that’s a bias of the thing that is being judged for bias.

_____

Now for your definition:

Ah, I see, so your definition of “bias” is something like “a perspective” (except without anthropomorphizing) . It is something that picks among multiple options in a way that isn’t unambiguously specified by precise rules. (Kind of reminds me of filters/ultrafilters. Probably not actually particularly analogous, but still came to mind. I guess a closer analogy would be the concept of a choice function.)

The issue I have with this definition is that it doesn’t capture the (quite common) usage of “bias” that a “bias” is something which is bad and is to be avoided.

When people say that a process, e.g. a ML program, is “biased against brunettes” (for example) they generally mean this as a criticism of that process. And I think this being a criticism is a major part of what is meant by the word “bias” (in this type of usage of the word, not in the sense of a constant term in an affine map).

I do get that often people say that “everyone has their own biases” and “it is impossible to be unbiased (about [topic])”, and they will sometimes describe their general perspective as a way of warning people about their own biases, and this somewhat fits with the “a bias is a perspective/choice-function “ type definition, but, I think it fails to capture the reason that people mention biases : because they think they can lead to being wrong (either leading to inaccurate conclusions or to unjust/immoral/unfair choices). I don’t think it is just a warning of “I sometimes have to make a choice among several options where there is no canonical right choice, and you might make different such choices”. It is instead a warning to others that one, like everyone else, is fallible, and moreover, that there may be patterns in those failings that one does not perceive (on account of those same failings), but that others, who have different patterns in their failings, might perceive, and, at the same time, things that others might perceive as failings but are not, due to their own failings.

Hm.

But, I do note a shortcoming in my definition that yours doesn’t seem to have: if multiple people who believe that there is no such thing as objective aesthetic quality are talking about the aesthetic qualities of various works, they might sometimes describe their patterns in their aesthetic judgements as “biases”, especially when these patterns are differences in how they judge things aesthetically vs how others (would) judge those things aesthetically. This seems more in line with the definition you gave than in the definition I gave, because such people don’t believe that there is a truth of the matter as to the aesthetic quality of the works, and therefore would not consider the ways they differ to be patterns in being wrong, only in being different (or just in being). Though, I think it seems to have some aspects of both. The definition you gave doesn’t seem to really include the pattern aspect.

____

Still, I think when people complain that a machine learning model is biased, what they mean is usually more like the definition I gave?

____

I noticed another shortcoming in my definition. Sometimes the “bias” that people complain that something has is not really any individual answer/output being wrong, but rather something about there being something wrong/undesirable in the distribution of the outputs. For a simple example, if dice aren’t fair, we call them biased. This could conceivably be more along the lines of the “the constant term in a affine map” sense, but I think people would say the same thing about something that e.g. selects applicants, even if it never picks an applicant that is objectively less preferable over one that is more preferable, if it among equally qualified candidates has a tendency that would be unfair, this is still called a bias even if any individual such choice would be fine. Fixing this would be a small change in phrasing, or perhaps a footnote with clarification that the thing that is “wrong” doesn’t have to be in any individual output.