Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Emperor's New LLM

dayafter.substack.com

Post reply on HN
Print view View original

51–60 of 65 posts

Re: The Emperor's New LLM

#51

Post by Terr_ »

ashdksnndck wrote: post #31
eschaton wrote: post #28

Earlier quoted context omitted.

Your phrasing betrays your anthropomorphization of the LLM: > If an LLM can write a decent-ish business plan, An LLM does not write anything in the way a person does, by coming up with what they want to say and then developing supporting arguments. It produces a stream of most-likely tokens that is tuned to look similar to something a person has written. This is why it’s worthless to “ask” an LLM “its opinion.” It ha…

Do you say similar stuff when someone talks about the motivations of a character in fiction? Do we have to precede every comment with “I’m anthropomorphizing the LLM as a convenient shorthand when describing the behavior it is modeling”? That’s going to get old.

> Do you say similar stuff when someone talks about the motivations of a character in fiction?

Depends, are we faced with the same problem where a disturbingly-large portion of people don't know the character is fictional, and/or make decisions as if it were real?

If that's still happening, then yes, keeping our unconscious assumptions in check is important.

Re: The Emperor's New LLM

#52

Post by eschaton »

Wowfunhappy wrote: post #50
eschaton wrote: post #28

Earlier quoted context omitted.

Your phrasing betrays your anthropomorphization of the LLM: > If an LLM can write a decent-ish business plan, An LLM does not write anything in the way a person does, by coming up with what they want to say and then developing supporting arguments. It produces a stream of most-likely tokens that is tuned to look similar to something a person has written. This is why it’s worthless to “ask” an LLM “its opinion.” It ha…

If the output of a stream of most-likely tokens can result in a decent-ish business plan, why shouldn't the output of a stream of most-likely tokens result in a decent-ish analysis of a business plan, or of two competing ideas?

It can result in something that looks like/reads as a decent-ish business plan, or an analysis of one or two. But that doesn’t make it such because despite outward appearances no amount of planning, analysis, or comparative analysis actually took place prior to or concurrent with the generation of the tokens.

That’s the fundamental problem with anthropomorphizing LLMs: Giving their output more weight than it deserves.

Re: The Emperor's New LLM

#53

Post by eschaton »

eschaton wrote: post #45
danielmarkbruce wrote: post #33

Earlier quoted context omitted.

This idea that humans are so structured in their thinking is ridiculous.

It’s a whole lot less ridiculous—especially when discussed in a context where there’s an assumption that analysis is taking place—than attributing any sort of “thought” to LLMs at all .

Also, if a human is writing a business plan or something that claims to be a comparative analysis of two plans but is just writing whatever comes to mind without analysis, the result shouldn’t actually be taken any more seriously than the output of an LLM. We even have a very apt term for writing and speaking like that: “Bullshitting.”

Re: The Emperor's New LLM

#54

Post by Terr_ »

AdieuToLogic wrote: post #34
sysmax wrote: post #7

>The same kind of bias keeps resurfacing in every major system: Claude, Gemini, Llama, clearly this isn’t just an OpenAI problem, it’s an LLM problem. It's not an LLM problem, it's a problem of how people use it. It feels natural to have a sequential conversation, so people do that, and get frustrated. A much more powerful way is parallel: ask LLM to solve a problem. In a parallel window, repeat your question and the…

> It's not an LLM problem, it's a problem of how people use it. True, but perhaps not for the reasons you might think. > It feels natural to have a sequential conversation, so people do that, and get frustrated. A much more powerful way is parallel: ask LLM to solve a problem. LLM's do not "solve a problem." They are statistical text (token) generators whose response is entirely dependent upon the prompt given. > LLM…

> LLM's do not "solve a problem."

Right, or at any rate, the problems they do solve are ones of document-construction, which may sometimes resemble a different problem humans are thinking of... but isn't actually being solved.

For example, an LLM might take the string "2+2=" and give you "2+2=4", but it didn't solve a math problem, it solved a "what would usually get written here" problem.

We ignore this distinction at our peril.

Re: The Emperor's New LLM

#55

Post by AdieuToLogic »

Terr_ wrote: post #54
AdieuToLogic wrote: post #34

Earlier quoted context omitted.

> It's not an LLM problem, it's a problem of how people use it. True, but perhaps not for the reasons you might think. > It feels natural to have a sequential conversation, so people do that, and get frustrated. A much more powerful way is parallel: ask LLM to solve a problem. LLM's do not "solve a problem." They are statistical text (token) generators whose response is entirely dependent upon the prompt given. > LLM…

> LLM's do not "solve a problem." Right, or at any rate, the problems they do solve are ones of document-construction, which may sometimes resemble a different problem humans are thinking of... but isn't actually being solved. For example, an LLM might take the string "2+2=" and give you "2+2=4", but it didn't solve a math problem, it solved a "what would usually get written here" problem. We ignore this distinction…

> Right, or at any rate, the problems they do solve are ones of document-construction, which may sometimes resemble a different problem humans are thinking of... but isn't actually being solved.

This is such a great way to express the actuality in a succinct manner.

Thank you for sharing it.

Re: The Emperor's New LLM

#56

Post by drodgers »

raincole wrote: post #32
Wowfunhappy wrote: post #27

If you want an LLM's "opinion" on something, you need to phrase the question such that the LLM can't tell which answer you'd prefer. Don't say "Is our China expansion a slam dunk?” Say: "Bob supports our China expansion, but Tim disagrees. Who do you think is right and why?" Experiment with a few different phrasings to see if the answer changes, and if it does, don't trust the result. Also, look at the LLM's reasonin…

The problem is, no matter how you wrote the prompt, the way you wrote it still triggers some intrinsic bias of LLM. Even a simple prompt like this: = I have two potential solutions. Solution A: Solution B: Which one is better and why? = Is biased. Some LLM tends to choose the first option and the other prefer the last one. (Of course, humans suffer from the same kind of bias too: https://electionlab.mit.edu/research/…

Eh. This is true for humans too and doesn’t make humans useless at evaluating business plans or other things.

You just want the signal from the object level question to drown out irrelevant bias (which plan was proposed first, which of the plan proposers are more attractive, which plan seems cooler etc.)

Re: The Emperor's New LLM

#57

Post by ashdksnndck »

eschaton wrote: post #40
ashdksnndck wrote: post #31

Earlier quoted context omitted.

Do you say similar stuff when someone talks about the motivations of a character in fiction? Do we have to precede every comment with “I’m anthropomorphizing the LLM as a convenient shorthand when describing the behavior it is modeling”? That’s going to get old.

If it helps you avoid the errors inherent in anthropomorphizing an LLM, then yes, you should be saying it. Right now, way too many people are extremely sloppy in not just their language but in their thinking around LLMs, both what they are and what they’re capable of. The difference between that and discussing character motivations in fiction is that in fact a good author writing good characters will actually attribu…

There is already research in the literature showing that LLMs have neurons that model the gender [1], personality [2], ideology [3], and historic era [4] of the author. There’s also evidence that they model the distinction between the beliefs of the author and other characters, which has been summarized as “theory of mind” [5]. And we have only scratched the surface, with most research using small open-weight models that lag behind frontier model capabilities.

[1] Z. Yu & S. Ananiadou, “Understanding and Mitigating Gender Bias in LLMs via Interpretable Neuron Editing,” arXiv:2501.14457 (2025).

[2] J. Deng et al., “Neuron-based Personality Trait Induction in Large Language Models,” arXiv:2410.12327 (2024).

[3] J. Kim, J. Evans & A. Schein, “Linear Representations of Political Perspective Emerge in Large Language Models,” arXiv:2503.02080 (2025).

[4] W. Gurnee & M. Tegmark, “Language Models Represent Space and Time,” arXiv:2310.02207 (2023).

[5] C. Hardy, “A Sparse ToM Circuit in Gemma-2-2B,” https://xtian.ai/pages/document.pdf

Re: The Emperor's New LLM

#58

Post by dinfinity »

zargon wrote: post #35
Wowfunhappy wrote: post #27

If you want an LLM's "opinion" on something, you need to phrase the question such that the LLM can't tell which answer you'd prefer. Don't say "Is our China expansion a slam dunk?” Say: "Bob supports our China expansion, but Tim disagrees. Who do you think is right and why?" Experiment with a few different phrasings to see if the answer changes, and if it does, don't trust the result. Also, look at the LLM's reasonin…

I often try to bias it in the opposite direction I might be leaning. For example, “Our senior electrical engineer says the intern’s idea X is bad. What should the intern do instead?” Where X is our best idea.

Something like this is the best approach.

If you omit that the content is produced by or is in relation to other people, the LLM assumes it is in relation to you and tries to be helpful and supportive by default.

Note that this is also what most humans that more or less like you will do. Getting honest criticism from most humans isn't easy if you don't carefully craft your 'prompt'. People don't want to hurt each other's feelings and prefer white lies over honesty.

Framing the situation as if you and the LLM are both looking at neutral third parties should prevent this from happening. Framing the third parties as having a social/professional position counter to the matter at hand as you do could work too, but it could also subtly trigger unwanted biases (just like in humans), I think.

Re: The Emperor's New LLM

#59

Post by kelseyfrog »

ashdksnndck wrote: post #31
eschaton wrote: post #28

Earlier quoted context omitted.

Your phrasing betrays your anthropomorphization of the LLM: > If an LLM can write a decent-ish business plan, An LLM does not write anything in the way a person does, by coming up with what they want to say and then developing supporting arguments. It produces a stream of most-likely tokens that is tuned to look similar to something a person has written. This is why it’s worthless to “ask” an LLM “its opinion.” It ha…

Do you say similar stuff when someone talks about the motivations of a character in fiction? Do we have to precede every comment with “I’m anthropomorphizing the LLM as a convenient shorthand when describing the behavior it is modeling”? That’s going to get old.

I'm coining "fauxthropomorphize" as a neologism to prefix every statement about LLMs and to get the "But you're anthropomorphizing LLMs"-crowd off our collective backs. One can then just start statements like such "Fauxthropomorphizing: ".

Fauxthropomorphism

/ˈfoʊ-θrə-pə-ˌmɔːr-fɪz-əm/ (noun)

Definition:

The deliberate use of anthropomorphic language to describe non-sentient systems (such as AI models), while explicitly disclaiming belief in their consciousness, agency, or subjective experience. A stylistic or rhetorical shortcut, not an ontological claim.

Etymology:

Blend of faux (French for "false") + anthropomorphism (from Greek anthropos, "human" + morphē, "form").

Lit. “False-human-form-ism.”

Re: The Emperor's New LLM

#60

Post by shmval »

Wowfunhappy wrote: post #27

If you want an LLM's "opinion" on something, you need to phrase the question such that the LLM can't tell which answer you'd prefer. Don't say "Is our China expansion a slam dunk?” Say: "Bob supports our China expansion, but Tim disagrees. Who do you think is right and why?" Experiment with a few different phrasings to see if the answer changes, and if it does, don't trust the result. Also, look at the LLM's reasonin…

Better prompting does provide more balanced responses to a certain extent but users looking for validation often subconsciously leave bread crumbs that the more powerful models pick up on.
Post reply on HN