>I can't prove that I'm not a dysfunctional cyborg with false perceptions
It doesn't matter if you are one. If you were an AI researcher and encountered a model that saw things for what they really are you would deem it to be malfunctioning and discard it. Any functional model will always be at least one degree further removed from a raw sensory experience that agrees with yours, than you. Its perception will be invariably filtered through the lens of labeled human output.
>I happen to be in possession of some of these and I think this needs a "usually, under ordinary conditions".
I won't dwell much on what you mean, since you said it's unimportant, but it takes a lot for sensory organs to malfunction, and even when they do, they produce corrupted, not false, information; black blotches, not pink elephants. I assume you were thinking of alcohol or something; drugs that affect perception affect the brain, not the other organs.
An interesting edge case is stuff like entoptic phenomena, but those aren't false perceptions; they're, if you will, hypertrue perceptions (something so true, we would rather not see it).
>but "red" and "blue" are the bread and butter of my visual consciousness. These correspond to firings of my sensory neurons much more than they correspond to anything in reality. If that's not interpretation, what is it/what is interpretation?
"Red" and "blue" are your brain's interpretation of the signals it receives from the eye. Notice how "red" and "blue" are fully abstract words, decoupled from anything physical, whereas if I were to refer to the actual encoding of visual information that passes through the optic nerve, I'd have no choice but to reference physical processes (probably talk about voltage and action potentials; I honestly don't know how the optic nerve works). That's because a retinal cell is a simple transducer. The eye doesn't interpret, it merely converts and encodes. Interpretation is a higher level operation.
You cannot compare the simplicity of the mechanism of a whole eye to the indescribable complexity that is between the sentence "roses are red and violets are blue" written on a book, and the raw perception of red roses and blue violets. But a model will only ever be exposed to that distant interpretation, not to roseness or redness.
>I expect we agree that I can show a multimodal model a real and a CGI picture and it can tell me which is which.
Of course, but obviously that's not something it knows inherently. There's nothing about the image intrinsically that says it's fake; someone has to label it such that the model can associate it with unreality (according to our own parameters). That's not how an animal works. An animal assumes what its senses perceive is real and can distinguish its own thoughts from its sensory input.
>On the flipside, I could show some meh drawings of fairies to Arthur Conan Doyle, and he'd say "Whoa, this changes everything". I consider him to be one of the great rational minds of history, but he was unable to pass your test here.
That's a slight equivocation. He would not have mistaken the drawings of fairies for raw perceptions of fairies, he would have simply been swayed by the rhetorical strength of the testimony implicit in the drawing (and perhaps an explicit one that accompanied it). If you want to make a true parallel to a multimodal model you'd have to compare a CGI fairy and a photograph of someone holding a drawing of a fairy. The CGI is as raw to the model as the signals passing through your optic nerves right now, but the drawing is one level of abstraction further away.