Positional preferences, order effects, prompt sensitivity undermine AI judgments

I'm not sure I understand your objection (or if it's even an objection), but just to illustrate what I mean - this is literally how the chat interfaces are implemented (or at least initially they were).

You're not talking with the model, you're talking with some entity that the model is asked to simulate. The system is just cleverly using your input and the statistic to output something that looks like chat with an assistant.

Whether that's real AI or not doesn't really matter. I didn't mean to make it sound like this is not real, just to point out where are the current shortcomings coming from.

It is an objection. I'm not sure if you consider the whole subfield of machine learning that is classification non-existing, or just the fact that LLMs can produce classifications, but either way, I do object.

The objection against the former is trivial and self evident, and was more where my sudden upset came from. *

Against the latter, the model trying to make the overall text that is its context window approach a chat exchange, by adjusting its own output within it accordingly, doesn't make a hypothetical request for classification in there not performed. You either classify or you don't. If it's doing it in a "misguided" way, it doesn't make it not performed. If it's doing it under the pretense of roleplay, it still doesn't make it not performed. Same is true if the LLM is actually secretly a human operator, or if the LLM is just spewing random tokens. Either you got a classified output of your input or you didn't. It doesn't "look like" anything. I can understand if maybe you mean that the designated notion of the person in the exchange it's trying to approximate for is going to affect the classifications it provides when requested one in the context window, but since these models are trained to "act agentic", I'm not sure if that's a useful thing to ponder (as there's no other way to get anything out of them).

I object to the whole "AI is just statistics" notion too. In several situations you want it to do something completely different than what the dataset would support just through rote statistics. That's where you get actual value out of them. One could conveniently recategorize that as just "advanced statistics", or "higher level" statistics, but I think that's a very perverse way of defining statistics. There's very clearly more mathematics involved in LLMs than just statistics. Just the other day, there was a post here trying to regard LLMs as "just topology". Clearly neither of these can be true at the same time, which was consequently explored in the thread too.

> You're not talking with the model, you're talking with some entity

I'm not suggesting I'm actually talking with anyone or anything in particular beyond the antropomorphization.

* What I meant by "people saying AI is not real" is that people claim to regard that the current generation of AI products are not real "artificial intelligences", because they seem to think that it's either "SkyNet" and "Detroit: Become Human", or nothing. Unsurprisingly, these folks don't tend to talk much about OCR, image segmentation and labeling, optical flow, etc. And just like the half a century old field of Artificial Intelligence isn't just some new marketing con that just spawned into existence, classification algorithms aren't some novel snakeoil either.

Edit: typing this all out about how I'm aware I'm not actually having conversations with anyone or anything gave me a feeling of realization. This is not good, because intellectually I was always aware of this, meaning I got subconsciously parasocial with these products and services over time. Really concerned about this all of a sudden lol.

This post was about LLMs so I was specifically referring to LLMs.

When I say "they immitate what classification looks like," I don't mean that the classification somehow isn't real, I'm referring to the specific technique of how it's done.

If you ask LLM "Is this sentence offensive: ...?", the task that it's doing is not simply "test whether this sentence is offensive." It's something like "generate what a plausible answer to this question looks like," part of which is answering the question (usually).

This means that if you ask this question in a way that is more often used with an expectation of a certain answer, LLM will use that as a signal to bias the answer, because "that's how the answers to these questions look like." which is the problem highlighted in the article.

> I feel that a lot of information they present is purposely misinterpreted by their teams for media or pr/clout or who knows what reasons.

I think it's just the culture of machine learning research at this point. Academics are better about it, but still far from squeaky clean. It can't be squeaky clean, because if you aren't willing to make grand overinflated claims to help attract funding, someone else will be, and they'll get the research funding, so they'll be the ones who get to publish research.

It's like an anthropic principle of AI research. (rimshot)