[flagged]
LLMs are not the black box you were promised
11–20 of 48 posts
Re: LLMs are not the black box you were promised
#12> Ask it "what is the capital of the state containing Dallas" and you can observe, in order:
> the Dallas feature goes active,
> which causes the Texas feature to light up,
> which then causes Austin to light up.
> It seems fairly clear that this is tracing semantic relationships between high-level concepts — and in doing so, performing a kind of pseudo-symbolic inference, similar to what some philosophers would describe as "higher reasoning."
Uhhh no reasoning is required for Austin to follow Texas after Dallas, let alone "higher reasoning".
This is really grasping as straws
Re: LLMs are not the black box you were promised
#13[dead]
Re: LLMs are not the black box you were promised
#14just curious.. are there languages that are better or more efficient to build LLM's with other than English?
Re: LLMs are not the black box you were promised
#15I don't recall being promised a black box. Are we certain llms didn't write this article and just came up with one of those It's Not Whatever-random-thing pithy zinger kind of things that they're prone to?
Re: LLMs are not the black box you were promised
#16"lack of metacognitive insight" is interesting, because I have observed people acting this way too. I have even observed it in myself
It is a characteristic of neural nets that they do not have insight into their own functioning.
It is arguably a characteristic of any intelligent system, that at least some part of it must be opaque to itself, but the previous sentence is more defensible than a generalized claim.
If you don't understand what that means, tell me from your own metacognitive insight what parts of your brain are being used to read this. Not because of learned knowledge about what parts of the brain do what, through your own insight in your own functioning. You can't, because you don't have any.
This isn't just that human rationalize a lot. This is below that. This is that even if you notice yourself rationalizing, which is something you can train yourself to do, you have no access to the underlying computations/processes of the rationalization itself, or the process of noticing you are rationalizing.
There is arguably still a sense that we experience in which we humans could reasonably say "No, I'm pretty sure I used addition-with-carry to answer you", so that is perhaps not the easiest example to think about the experience of. But there will always be some question of "how did you do that" to which you can give no answer because the answer is in the firing of the neural net itself and you, who is in one way or another the product of that firing, do not have access to that. How did you quickly catch that ball that someone unexpectedly threw at you? You just did, as far your neural net is concerned.
(Also, while I've expressed this in terms of your conscious experience, this doesn't have anything to do with "consciousness". Neural nets in general do not get this feedback and do not and can not have arbitrary metacognition about their own functioning. This is an artifact of my writing text to address conscious beings.)
Re: LLMs are not the black box you were promised
#17If you asked me last year(2025) I would have still said LLMs are a silly toy. As of Jan 2026 I have come to accept that LLMs are at least part of the puzzle of how intelligence works. They are at this point better than the majority of humans at various intellectual tasks. It may not be or ever be a 1:1 but good enough ran the world already before llms. There is not even a formal definition of what intelligence is so…
"How can we understand what an LLM is "thinking"? It's clearly very valuable to do so — it could enable steering model behavior, detecting dangerous intent, and more."
Well that is complete any utter bollocks, dribbled in para three or so, and obviously written by a next token guesser.
LLMs are tools and I'm pretty sure if I let you loose on some of my tools, you might lose an extremity unless I kept an eye on you.
I have an on prem Qwen3.6-35B-A3B-UD-Q4_K_XL working on a box in the office and its quite handy for a chat.
Re: LLMs are not the black box you were promised
#18just curious.. are there languages that are better or more efficient to build LLM's with other than English?
[dead]
Re: LLMs are not the black box you were promised
#19"lack of metacognitive insight" is interesting, because I have observed people acting this way too. I have even observed it in myself
Isn't that a metacognitive insight itself?
I am not incapable of metacognition, but I still have observed (and keep observing) cases where I seem to lack real metacognitive insight into something.
Re: LLMs are not the black box you were promised
#20"lack of metacognitive insight" is interesting, because I have observed people acting this way too. I have even observed it in myself
It is a characteristic of neural nets that they do not have insight into their own functioning. It is arguably a characteristic of any intelligent system, that at least some part of it must be opaque to itself, but the previous sentence is more defensible than a generalized claim. If you don't understand what that means, tell me from your own metacognitive insight what parts of your brain are being used to read this.…
Yep. An effect cannot in itself reason about its cause. Some effects can suggest causes though. For example you tend to have memory of an algorithm you just executed, when the usage was at least somewhat conscious or intended, which can lead you to be able to guess which algorithm you used (and potentially even correctly)