Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI: Accelerated Incompetence

slater.dev

Post reply on HN
Print view View original

271–280 of 287 posts

Re: AI: Accelerated Incompetence

#271

Post by crazylogger »

acquisitionsilk wrote: post #268
Lazarus_Long wrote: post #243

Earlier quoted context omitted.

I assume you ignored "teleology" because you concede the point, otherwise feel free to take it. " Is there an “inventiveness test” that humans can pass but LLMs don’t?" Of course, any topic where there is no training data available and that cannot be extrapolated by simply mixing the existing data. Of course that is harder to test on current unknowns and unknown unknowns. But it is trivial to test on retrospective kn…

Now there's a sensible point. Can someone do that, so that we can put some of these points definitely to bed?

I don't think it will settle things even if we did manage to train an 1800 LLM with sufficient size.

LLMs are blank slates (like an uncultured primitive human being - albeit LLM comes with knowledge built-in, but builtin knowledge is mostly irrelevant here). LLM output is purely a function of the input (context), so agentic systems' capabilities do not equal underlying LLM's capabilities.

If you ask such an LLM "overturn Newtonian physics, come up with a better theory", of course the LLM won't give you relativity just like that. The same way an uneducated human has no chance of coming up with relativity either.

However, ask it this:

``` You are Einstein ... ... Recent experiments have put these ideas to doubt, ...... Any idea why this occurs? ```

and provide it with tools to find books, speak with others, run experiments, etc. Conceivably, the result will be different.

Again, we pretty much see this play out in coding agents:

Claude the LLM has no prior knowledge of my codebase so of course it has zero chance of solving a bug in it. Claude 4 is a blank slate.

Claude Code the agentic system can:

- look at a screenshot.

- know what the overarching goal is from past interactions & various documentation it has generated about the codebase, as well as higher-level docs describing the company and products.

- realize the screenshot is showing a problem with the program.

- form hypothesis / ideate why the bug occurs.

- verify hypotheses by observing the world ("the world" to Claude Code is the codebase it lives in, so by "observing" I mean it reads the code).

- run experiments: modify code then run a type check or unit test (although usually the final observation is outsourced to me, so I am the AI's tool as much as the other way around.)

Re: AI: Accelerated Incompetence

#272

Post by tempodox »

> LLMs give me finished thoughts, polished and convincing, but none of the intellectual growth that comes from developing them myself

Even if all the wonders were true that people love to believe in about LLMs, you cannot get around this argument.

Re: AI: Accelerated Incompetence

#273

Post by croemer »

This claim is talking out of proportion, LLMs do push back:

> Input Risk. An LLM does not challenge a prompt which is leading or whose assumptions are flawed or context is incomplete. Example: An engineer prompts, "Provide a thread-safe list implementation in C#" and receives 200 lines of flawless, correct code. It's still the wrong answer, because the question should have been, "How can I make this code thread-safe?" and whose answer is "Use System.Collections.Concurrent" and 1 line of code. The LLM is not able to recognize an instance of the XY problem because it was not asked to.

When I prompt Gemini 2.5 Pro with "Provide a thread-safe list implementation in C#" it _does_ push back and suggest using the standard library instead (in addition to providing the code of course). First paragraph of the LLM response:

> You can achieve a thread-safe list in C# by using the lock statement to synchronize access to a standard List. Alternatively, you can use concurrent collection classes provided by .NET, such as ConcurrentBag or ConcurrentQueue, depending on the specific access patterns you need.

https://g.co/gemini/share/7ac7b9238b28

Re: AI: Accelerated Incompetence

#274

Post by mjburgess »

TeMPOraL wrote: post #266
mjburgess wrote: post #259

Earlier quoted context omitted.

Is this based on analogising LLMs to animal mental capacities, or based on a scientific study of these capacities? ie., is this confirmation bias, or science? One can always find a kind of confirmation bias analysis here, which "saves the appearances", ie., one can always say "take a measurement set of people's mental capacities, given in their linguistic behaviour" and find such behaviours apparent in LLMs. This wil…

> Is this based on analogising LLMs to animal mental capacities, or based on a scientific study of these capacities? ie., is this confirmation bias, or science? - On how embeddings work; - On the observation that in very high-dimensional space you can encode a lot of information in relative arrangement of things; - On the observation that the end result (LLMs) are too good at talking and responding like people in nua…

> On the observation that most of the failure modes of LLMs also happen to human

That's assuming that LLMs operate according to how we read their text. What you're doing is reading llm chain-of-thought as-if said by a human, and imparting the implied capacities that would be implied if a human said it. But this is almost certainly not how LLMs work.

LLMs are replaying "linguisitc behaivour" which we take, often accurately, to be dispositive of mental states in people. They are not evidence of mental capacities and states in LLMs, for seemingly obvious reasons. When a person says, "I am hungry" it is, in verdical cases, caused by their hunger. When an LLM says it the cause is something like, "responding appropriately, accoring to a history of appropriate use of such words, on the occasion of a prompt which would, in ordinary historical cases, give this response".

The reason an LLM generates a text prima fascie never involves any associated capacities which would have been required for that text to have been written in the first place. Overcoming this leap of logic requires vastly more than "it seems to me".

> On how embeddings work

The space of necessary capacities is no exhausted by "embedding", by which you mean a (weakly) continuous mapping of historical exemplars into a space. Eg., logical relationships, composition, recursion, etc. are not mental capacities which can be implemented this way.

> We don't know what drives our own mental processes either.

Sure we do. At the level of enumerating mental capacities, their operation and so on, we can give very exhaustive lists. We do not know how even the most basic of these is implemented biologically, save I believe, we can say quite a lot about how properties of complex biological systems generically enable this.

But we have a lot of extremely carefully designed experiments to show the existence of relevant capacities in other animals. None of these experiments can be used on an LLM, because by design, any experiment we would run would immediately reveal the facade: any measurement of the GPU running the LLM and its environmental behaviour shows a total empirical lack of anything which could be experimentally measured.

We are, by the charaltan's design, only supposed to use token-in/token-out as "measuremnt". But this isn't a valid measure, becuase LLMs are constructed on historical cases of linguistic behaviour in people. We know, prior to any experiment, that the one thing designed to be a false measure, is the lingustic behaviour of the LLM.

Its as if we have constructed a digital thermometer to always replay historical temperature readings -- we know, by design, that these "readings" are therefore never indicative of any actual capacity of the device to measure temperature.

Re: AI: Accelerated Incompetence

#275

Post by croemer »

> LLMs as they currently exist cannot master a theory, design, or mental construct because they don't remember beyond their context window.

That's not categorically true: if a theory/design fits in their context window it's possible that they _can_ master it.

LLMs shine for simple tasks with little context. There are plenty of tasks like that.

Re: AI: Accelerated Incompetence

#277

Post by croemer »

sensanaty wrote: post #21

> Input Risk. An LLM does not challenge a prompt which is leading ... (Emphasis mine) This has been the biggest pain point for me, and the frustrating part is that you might not even realize you're leading it a particular way at all. I mean it makes sense with how LLMs work, but a single word used in a vague enough way is enough to skew the results in a bad direction, sometimes contrary to what you actually wanted to…

People have similar biases, it's just a lot easier to test an LLM for biases than it is to test humans so we are more aware of LLM biases.

Re: AI: Accelerated Incompetence

#278

Post by btbuildem »

eikenberry wrote: post #258
btbuildem wrote: post #80

I strongly agree with both the premise of the article, and most of the specific arguments brought forth. That said, I've also been noticing some positive aspects of using LLMs in my day-to-day. For context, I've been in the software trade for about three decades now. One thing working with AI-generated code forces you to do is to read code -- development becomes more a series of code reviews than a first-principles c…

I don't think it is becoming a series of code reviews, more like having something do some prototyping for you. It is great for fixing the blank page problem, but not something you can review and commit as is.

In my experience code reviews involve a fair bit of back-and-forth, iterating with the would-be committer until the code 1) does what it's meant to and 2) does it in an acceptable manner. This parallels the common workflow of trying to get an LLM to produce something useable.

Re: AI: Accelerated Incompetence

#279

Post by dangus »

johnecheck wrote: post #262
dangus wrote: post #226

Earlier quoted context omitted.

Even the phrase "world-changing" might be a bit too strong. It's enabled some acceleration of product prototyping and it has democratized hardware design a little bit. Some little companies are building some buildings using 3D printing techniques. Speaking as someone who owns and uses a 3D printer daily, I think the biggest impact it's had is that it's a fun hobby, which doesn't strike me as "world-changing."

That's fair, but don't sell them short. A 3d printed gun just killed a CEO. Ukrainian drones are using 3d printed parts to drop bombs. Between that and the changed game for hobbyists, the world is meaningfully different. Most world-changing inventions do so subtly. Atom bombs are the exception, not the rule.

I think the phrase "world-changing" implies a lot less subtlety than that.

Re: AI: Accelerated Incompetence

#280

Post by johnecheck »

dangus wrote: post #279
johnecheck wrote: post #262

Earlier quoted context omitted.

That's fair, but don't sell them short. A 3d printed gun just killed a CEO. Ukrainian drones are using 3d printed parts to drop bombs. Between that and the changed game for hobbyists, the world is meaningfully different. Most world-changing inventions do so subtly. Atom bombs are the exception, not the rule.

I think the phrase "world-changing" implies a lot less subtlety than that.

That's fair. My definition is probably broader than most.
Post reply on HN