Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs are not the black box you were promised

jay.ai

Post reply on HN
Print view View original

31–40 of 48 posts

Re: LLMs are not the black box you were promised

#31

Post by coppsilgold »

LoganDark wrote: post #2

"lack of metacognitive insight" is interesting, because I have observed people acting this way too. I have even observed it in myself

I believe it's a great deal worse than that. All the metacognitive insight we do have may just be confabulation and we are fooled into believing that we have it because the process for conjuring it is good at finding a plausible answer.

When you read about and observe the split-brain patient experiments the appropriate response is abject horror at the implications.

Re: LLMs are not the black box you were promised

#33

Post by dawnerd »

airstrike wrote: post #12

> Ask it "what is the capital of the state containing Dallas" and you can observe, in order: > the Dallas feature goes active, > which causes the Texas feature to light up, > which then causes Austin to light up. > It seems fairly clear that this is tracing semantic relationships between high-level concepts — and in doing so, performing a kind of pseudo-symbolic inference, similar to what some philosophers would desc…

Even better, I just tried in chatgpt and it just googled it and told me. That’s not reasoning, that’s offloading a task and taking five times as long and way more energy than if I just typed it into Google myself.

Re: LLMs are not the black box you were promised

#35

Post by camelmel »

LLM written article. It's also not accurate; the fact that language models have human-interpretable representations and neurons has been known since BERT.

Circuits research also does not come from Anthropic. Mech interp is a huge field in academia and most of the core circuit analysis papers were from OpenAI/GDM/academia. However, Anthropic tends to produce a lot of blog posts where they draw poorly supported but hype-able analogies between LLMs and biological intelligence. It's wild.

For a better understanding of mech interp and circuits, including what we actually do know about LLM internals, I would recommend reading this paper: https://arxiv.org/pdf/2501.16496

Re: LLMs are not the black box you were promised

#36

Post by svachalek »

an0malous wrote: post #28
citizenpaul wrote: post #9

If you asked me last year(2025) I would have still said LLMs are a silly toy. As of Jan 2026 I have come to accept that LLMs are at least part of the puzzle of how intelligence works. They are at this point better than the majority of humans at various intellectual tasks. It may not be or ever be a 1:1 but good enough ran the world already before llms. There is not even a formal definition of what intelligence is so…

They’re the language part of the puzzle, which seems to require some basic world modeling but it can’t make novel models unless there’s an example in its training data. I think engineering and mathematical thought requires spatial reasoning, when I model problems I see them as 3D shapes. Like the economy is a series of tubes that money flows through and collect in buckets, programming state is little boxes that hold…

Most LLMs are multimodal now, able to map visual concepts to language and vice versa. If OpenAI's recent Erdos solution was faking math, it faked it very well.

Re: LLMs are not the black box you were promised

#37

Post by svachalek »

lucisferre wrote: post #29
citizenpaul wrote: post #9

If you asked me last year(2025) I would have still said LLMs are a silly toy. As of Jan 2026 I have come to accept that LLMs are at least part of the puzzle of how intelligence works. They are at this point better than the majority of humans at various intellectual tasks. It may not be or ever be a 1:1 but good enough ran the world already before llms. There is not even a formal definition of what intelligence is so…

They are mostly "faster" than the majority of humans. They are rarely better than experienced and talented humans at the majority of tasks they are able to do. They are better on both scales on a small thin slice of work tasks.

They're not better than the best humans at practically anything. However I doubt there's a person alive that could outperform an LLM on a broad suite of tasks like Humanity's Last Exam and the vast majority of people probably couldn't answer a single question on it.

Re: LLMs are not the black box you were promised

#38

Post by pessimizer »

an0malous wrote: post #28
citizenpaul wrote: post #9

If you asked me last year(2025) I would have still said LLMs are a silly toy. As of Jan 2026 I have come to accept that LLMs are at least part of the puzzle of how intelligence works. They are at this point better than the majority of humans at various intellectual tasks. It may not be or ever be a 1:1 but good enough ran the world already before llms. There is not even a formal definition of what intelligence is so…

They’re the language part of the puzzle, which seems to require some basic world modeling but it can’t make novel models unless there’s an example in its training data. I think engineering and mathematical thought requires spatial reasoning, when I model problems I see them as 3D shapes. Like the economy is a series of tubes that money flows through and collect in buckets, programming state is little boxes that hold…

Seems like LLMs are that. A bunch of most probable word associations is a network, and you can build a physical model of a network, or build a network that allows you to reason about a physical model. Whether it's just a flowchart or workflow diagram, or an X-dimensional matrix with vectors moving through it.

But the only way to map the network in an LLM is experimentally. You have to prompt it, and see how the coefficients fall in order to construct your most likely walk through the training data.

I think that LLMs can and do come up with novel things through exhaustion, just by applying the relationships between some set of entities to entirely different sets of entities because an accumulation of earlier context pushed the probability of those entities being mentioned, and they were able to easily replace a selection of entities that were more associated with those nearer connective, relationship words.

I think that as such LLMs are good at generating metaphors, and a lot of innovation comes from going "What if As worked like Bs?" Just go through all the As and Bs, toss the ones that don't make any sense and test the ones that seem like they might.

Re: LLMs are not the black box you were promised

#39

Post by bediger4000 »

an0malous wrote: post #28
citizenpaul wrote: post #9

If you asked me last year(2025) I would have still said LLMs are a silly toy. As of Jan 2026 I have come to accept that LLMs are at least part of the puzzle of how intelligence works. They are at this point better than the majority of humans at various intellectual tasks. It may not be or ever be a 1:1 but good enough ran the world already before llms. There is not even a formal definition of what intelligence is so…

They’re the language part of the puzzle, which seems to require some basic world modeling but it can’t make novel models unless there’s an example in its training data. I think engineering and mathematical thought requires spatial reasoning, when I model problems I see them as 3D shapes. Like the economy is a series of tubes that money flows through and collect in buckets, programming state is little boxes that hold…

I don't believe you can say that "LLM" is part of intelligence. No single human is exposed to as much text as any LLM model ingests, not by many orders of magnitude, and humans still perform cognition and generate new language.

Re: LLMs are not the black box you were promised

#40

Post by _jayhack_ »

camelmel wrote: post #35

LLM written article. It's also not accurate; the fact that language models have human-interpretable representations and neurons has been known since BERT. Circuits research also does not come from Anthropic. Mech interp is a huge field in academia and most of the core circuit analysis papers were from OpenAI/GDM/academia. However, Anthropic tends to produce a lot of blog posts where they draw poorly supported but hyp…

Hello, I am the author - this is not an LLM-generated article, I wrote this by hand and had an LLM adapt it from a thread on X. You can see the original thread here: https://x.com/mathemagic1an/status/2035850046735098065

> the fact that language models have human-interpretable representations and neurons has been known since BERT... Circuits research also does not come from Anthropic... The article does not claim Anthropic invented the field, rather that they have had important contributions to it. This is intended as an overview into a specific set of ideas that are working for mechanistic interpretability. Not a formal literature review.

Post reply on HN