Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Being “Confidently Wrong” is holding AI back

promptql.io

Post reply on HN
Print view View original

151–160 of 274 posts

Re: Being “Confidently Wrong” is holding AI back

#151

Post by therobots927 »

aleph_minus_one wrote: post #135
jansper39 wrote: post #97

Earlier quoted context omitted.

Honestly when I speak about these sorts of issues I get the feeling that other people view me as some kind of luddite, especially people above me who presumably want to replace as many people with AI as possible. I suppose me pointing out the flaws breaks the illusion of magic that people want AI to have.

> I suppose me pointing out the flaws breaks the illusion of magic that people want AI to have. My impression is rather: there exist two kinds of people who are "very invested in this illusion": 1. People who want to get rich by either investing in or working on AI-adjacent topics. They of course have an interest to uphold this illusion of magic. 2. People who have a leftist agenda ("we will soon all be replaced by A…

I agree with your first point but regarding your second: I’m as far left as it gets and I don’t think that’s true at all. Most of the influencers I follow despise AI and also are highly skeptical of the outrageous claims made by Sam Altman etc. The reality is that the need for things like universal health care exists today. Tens of millions of people can not get medical care in the US. Insurance companies are allowed to deny claims with no justification. That has nothing to do with AI taking jobs BUT it does involve AI because United Health’s denial rate went through the roof right after they started letting AI determine which claims were covered by policy with no human review. So people on the left are talking about AI in contexts that it doesn’t seem you’re aware of

Re: Being “Confidently Wrong” is holding AI back

#152

Post by vidarh »

aleph_minus_one wrote: post #135
jansper39 wrote: post #97

Earlier quoted context omitted.

Honestly when I speak about these sorts of issues I get the feeling that other people view me as some kind of luddite, especially people above me who presumably want to replace as many people with AI as possible. I suppose me pointing out the flaws breaks the illusion of magic that people want AI to have.

> I suppose me pointing out the flaws breaks the illusion of magic that people want AI to have. My impression is rather: there exist two kinds of people who are "very invested in this illusion": 1. People who want to get rich by either investing in or working on AI-adjacent topics. They of course have an interest to uphold this illusion of magic. 2. People who have a leftist agenda ("we will soon all be replaced by A…

The left is generally extremely sceptical to UBI, as its main proponents tend to be classically liberal groups (so not "US liberal") pushing it as a means to contain and limit welfare systems by dropping welfare programs in favour of a general, low UBI.

The more leftist position ever since the days of Marx has been that "right rather than being equal would have to be unqueal" to be equitable given that people have different needs, to paraphrase from Critique of the Gotha Program - UBI is in direct contradiction to socialist ideals of fairness.

The people I see pushing UBI, on the contrary, usually seems motivated either by the classically liberal position of using it to minimise the state, or driven by a fear of threats to the stability of capitalism. Saving capitalism from perceived threats to itself isn't a particularly leftist position.

Re: Being “Confidently Wrong” is holding AI back

#153

Post by gavinray »

delichon wrote: post #142
gavinray wrote: post #133

Earlier quoted context omitted.

There's a guy, Ken Stanley, who wrote the NEAT[0]/HyperNEAT[1] algorithms. His big idea is that evolution/advancements don't happen incrementally, but rather in unpredictable large leaps. He wrote a whole book about it that's pretty solid IMO: "Why Greatness Cannot Be Planned: The Myth of the Objective." [0] https://en.wikipedia.org/wiki/Neuroevolution_of_augmenting_t... [1] https://en.wikipedia.org/wiki/HyperNEAT

https://en.wikipedia.org/wiki/Saltation_(biology)

Neat (no pun intended), TIL there's a word for this

Re: Being “Confidently Wrong” is holding AI back

#154

Post by vidarh »

psadauskas wrote: post #146
vidarh wrote: post #139

Earlier quoted context omitted.

This is a good point, and to drive this home to people, if you have a conversation of this pattern: User: Fix this problem ... Assistant: X User: No, don't do X Assistant: Y User: No, Y is wrong too. Assistant: X It is generally pointless to continue. You now have a context that is full of the assistant explaining to you and itself why X and Y are the right answers, and much less context of you explaining why it is w…

There's also the Pink Elephant Paradox (Whatever you do, DO NOT think about a pink elephant). If you mention X or Y, even if they're preceded by "DO NOT" in all caps, an LLM will still end up with both X and Y into its context, making it more likely it gets used. I'm running out of ways to tell the assistant to not use mocks for tests, it really really wants to use them.

I think in some cases you "just" need to instead up temperature to increase the variety of responses, repeat requests, and use hooks to automatically review and reject bad options.

(And yes, it's a horrible workaround)

Re: Being “Confidently Wrong” is holding AI back

#155

Post by [deleted] »

chpatrick wrote: post #131
staticman2 wrote: post #123

Earlier quoted context omitted.

I really wish people into LLMs would limit themselves to terms from neuroscience or philosophy when descrbing humans. You are in my mind rightfully getting pushback for writing "human experts also output tokens with some statistical distribution. "

That's just a mathematical fact. You have a big opaque box with a slot where you can put text in and you can see text come out. The text that comes out follows some statistical distribution (obviously), and isn't always the same. Can you decide just from that if there's an LLM or a human sitting inside the box? No. So you can't make conclusions about whether the box as a system is intelligent just because it outputs…

[deleted]

Re: Being “Confidently Wrong” is holding AI back

#156

Post by chpatrick »

staticman2 wrote: post #150
chpatrick wrote: post #131

Earlier quoted context omitted.

That's just a mathematical fact. You have a big opaque box with a slot where you can put text in and you can see text come out. The text that comes out follows some statistical distribution (obviously), and isn't always the same. Can you decide just from that if there's an LLM or a human sitting inside the box? No. So you can't make conclusions about whether the box as a system is intelligent just because it outputs…

Okay... I objected to your use of the word token. Humans don't think in tokens or even write in tokens so obviously what you wrote is not a fact. That shouldn't even be controversial, I don't think? You wrote "The text that comes out follows some statistical distribution". At the risk of being over my head here did you mean the text can be described statistically or "follows some statistical distribution". Are these…

> Humans don't think in tokens or even write in tokens so obviously what you wrote is not a fact.

Doesn't matter what they think in. A token can be a letter or a word or a sound. The point is that the box takes some sequence of tokens and produces some sequence of tokens.

> You wrote "The text that comes out follows some statistical distribution". > At the risk of being over my head here did you mean the text can be described statistically or "follows some statistical distribution". Are these two concepts the same thing? I don't think so. > A program by design follows some statistical distribution. A human is doing whatever electrochemical thing it's doing that can be described statistically after the fact.

Again, it doesn't matter how the box works internally. You can only observe what goes in and out and observe its distribution.

> Regardless my point was pretty simple, I know this will never happen but I wish tech people would drop this tech language when describing humans and adopt neuroscience language.

My point is neuroscience or not doesn't matter. People make the claim that "the box just produces characters with some stochastic process, therefore it's not intelligent or correct", and I'm saying that implication is not true because there could just as well be a human in the box.

You can't decide whether a system is intelligent just based of the method with which it communicates.

Re: Being “Confidently Wrong” is holding AI back

#157

Post by AIPedant »

selfmodruntime wrote: post #49
rwmj wrote: post #6

Only thing? Just off the top of my head: That the LLM doesn't learn incrementally from previous encounters. That we appear to have run out of training data. That we seem to have hit a scaling wall (reflected in the performance of GPT5). I predict we'll get a few research breakthroughs in the next few years that will make articles like this seem ridiculous.

Never before did we have a combination of well and poison where the pollution of the first was both as instantaneous and as easily achieved. I‘ve yet to see a convincing article for artificial training data.

It does seem like it helps with math, but in a way that demonstrates the futility of the enterprise: "after training the LLM on 10,000,000 examples of K-8 arithmetic it is now superhuman up to 12 digits, after which it falls off a cliff. Also it demonstrably doesn't understand what 'four' means conceptually and it still fails on many trivial counting problems."

Re: Being “Confidently Wrong” is holding AI back

#158

Post by [deleted] »

vidarh wrote: post #152
aleph_minus_one wrote: post #135

Earlier quoted context omitted.

> I suppose me pointing out the flaws breaks the illusion of magic that people want AI to have. My impression is rather: there exist two kinds of people who are "very invested in this illusion": 1. People who want to get rich by either investing in or working on AI-adjacent topics. They of course have an interest to uphold this illusion of magic. 2. People who have a leftist agenda ("we will soon all be replaced by A…

The left is generally extremely sceptical to UBI, as its main proponents tend to be classically liberal groups (so not "US liberal") pushing it as a means to contain and limit welfare systems by dropping welfare programs in favour of a general, low UBI. The more leftist position ever since the days of Marx has been that "right rather than being equal would have to be unqueal" to be equitable given that people have di…

[deleted]

Re: Being “Confidently Wrong” is holding AI back

#159

Post by hodgehog11 »

roxolotl wrote: post #11

The big thing here is that they can’t even be confident. There is no there there. They are a, admittedly very useful, statistical model. Ascribing confidence to it is an anthropomorphizing mistake which is easy to make since we’re wired to trust text that feels human. They are at their most useful when it is cheaper to verify their output than it is to generate it yourself. That’s why code is rather ok; you can run i…

But as a statistical model, it should be able to report some notion of statistical uncertainty, not necessarily in its next-token outputs, but just as a separate measure. Unfortunately, there really doesn't seem to be a lot of effort going into this.

Re: Being “Confidently Wrong” is holding AI back

#160

Post by dkersten »

traceroute66 wrote: post #32
lucideer wrote: post #8

While the thrust of this article is generally correct, I have two issues with it: 1. The words "the only thing" massively underplays the difficulty of this problem. It's not a small thing. 2. One of the issues I've seen with a lot of chat LLMs is their willingness to correct themselves when asked - this might seem, on the surface, to be a positive (allowing a user to steer the AI toward a more accurate or appropriate…

> is their willingness to correct themselves when asked Except they don't correct themselves when asked. I'm sure we've all been there, many, many, many,many,many times .... - User: "This is wrong because X" - AI: "You're absolutely right ! Here's a production-ready fixed answer" - User: "No, that's wrong because Y" - AI: "I apologise for frustrating you ! Here's a robust answer that works" - User: "You idiot, you ju…

My pet peace is when I point out a problem it responds with acknowledgement and then explaining why it’s wrong. Like… I already know why it’s wrong, since I’m the one that pointed it out!
Post reply on HN