Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scientists should use AI as a tool, not an oracle

aisnakeoil.com

Post reply on HN
Print view View original

61–70 of 113 posts

Re: Scientists should use AI as a tool, not an oracle

#61

Post by threeseed »

duxup wrote: post #24
skywhopper wrote: post #12

Earlier quoted context omitted.

This is incredibly simplistic. Search engine results give a lot of context clues about the reliability of their asserted facts and provide a potential spectrum of answers. LLM-generated answers strip all that away, and give a single authoritatively phrased answer. Even if you’re inclined to disbelieve it, the LLM answer gives you no ability to dig in, refine, or compare. It just is. If you ask a chatbot if it’s sure,…

I think you overestimate people’s ability to sniff out bad data on the internet. Also are you suggesting people fact check an AI by asking it if it is correct? That seems absurd.

But you could trust certain websites being more accurate than others based on their brand, the author, the other content the site had published, who they are linking to and people linking to them etc.

LLMs remove that ability to be discerning about what to trust.

Re: Scientists should use AI as a tool, not an oracle

#62

Post by Kalium »

skrap wrote: post #42

...but why wouldn't they use AI as an oracle? From an outsider's perspective, it seems that there's already plenty of incentive to test the margins of acceptable academic practice in order to produce more papers or publish more quickly. Sadly I feel like it'll become the norm to have a chatbot interpret your results and write your paper rather than using those expensive grad students. I don't have answers; just the l…

We're building this because the ability to make narrow, specific predictions can be narrowly and specifically useful. This works if you have a good understanding of both the tools and the domain you're looking to make predictions in.

Unfortunately, from an outsider perspective, this looks like being widely and generically useful. If you don't understand your tools, you're going to misuse them, and this hype cycle is the result.

Re: Scientists should use AI as a tool, not an oracle

#63

Post by williamcotton »

SrslyJosh wrote: post #28
captainkrtek wrote: post #19

In my professional work, I treat chatgpt as a search engine that I feel I can ask questions of in a natural manner. I often find small flaws in technical solutions it offers, but it can still provide useful starting points to investigate. I rarely trust code it generates (at least for the language I mainly work in) as i’ve seen it make some serious mistakes (eg: using keywords in the language that don’t exist)

> I rarely trust code it generates (at least for the language I mainly work in) as i’ve seen it make some serious mistakes (eg: using keywords in the language that don’t exist) It's only a mistake from your perspective. The model just generates text based the probabilities it learned during training. In that respect, there is no such thing as "incorrect" output because the model doesn't operate at that level of abstr…

“Correctness” is a property of a proposition determined by an observer. Sometimes what is output by an LLM is correct, sometimes not. That an LLM is aware of the output or not means literally nothing.

Re: Scientists should use AI as a tool, not an oracle

#64

Post by threeseed »

logrot wrote: post #18

But surely if it's artificial intelligence then it'd know its limits and would respond appropriately? Oracle use no problem? It is it because it's actually shit but it's the best thing we've seen yet and everyone is just in denial?

It depends on who you mean.

Most normal people look at AI like ChatGPT as an amazing tool and have used it effectively as a replacement for Google, Grammarly etc. And for them it's fine because any mistakes are localised to them.

The problem are those building products on LLMs e.g. Legal, Customer Service who are knowingly misrepresenting the capabilities of what it can do to companies who don't know any better. And I would argue this is fraudulent and where we will see most of the problems.

Re: Scientists should use AI as a tool, not an oracle

#65

Post by [deleted] »

mewpmewp2 wrote: post #29
logrot wrote: post #18

But surely if it's artificial intelligence then it'd know its limits and would respond appropriately? Oracle use no problem? It is it because it's actually shit but it's the best thing we've seen yet and everyone is just in denial?

People constantly misevaluate their own limits though. Why should AI not be allowed to do that?

[deleted]

Re: Scientists should use AI as a tool, not an oracle

#66

Post by fragmede »

gerdesj wrote: post #59
unkulunkulu wrote: post #5

> Unfortunately, most scientific fields have succumbed to AI hype, leading to a suspension of common sense. For example, a line of research in political science claimed to predict the onset of civil war with an accuracy2 of well over 90%, a number that should sound facially impossible. (It turned out to be leakage, which is what got us interested in this whole line of research.) This coupled with people acting on its…

"accuracy2" sigh - the 2 is a superscript to a footnote and not a domain specific term. "facially impossible" ... does that really riff on "on the face of it", or is it farcically misspelt? Garbage in, garbage out 8)

Also from the article:

> Also, ML code tends to vastly more complex and less standardized than traditional statistical modeling.

I mean, hey, it's proof that the text isn't AI generated, since ChatGPT is better at English than that, but it makes it hard to read and I'm not going to buy their book if it's going to be full of errors like that.

Re: Scientists should use AI as a tool, not an oracle

#67

Post by dotnet00 »

Zambyte wrote: post #60
shmatt wrote: post #22

I feel like 90% of AI discussions online these days can be shut down with “a probabilistic syllable generator is not intelligence”

Humans are not fact machines, we are often wrong. Do humans not have intelligence? What do you even mean by "intelligence" when you say a probabilistic syllable generator "is not intelligence"?

Like clockwork, out come the "but humans" deflections. An LLM is not a human-like intelligence. This is patently obvious, such comparisons are nonsensical and just further the problem of people anthropomorphizing a tool and treating it like an oracle.

Re: Scientists should use AI as a tool, not an oracle

#68

Post by Jensson »

mewpmewp2 wrote: post #29
logrot wrote: post #18

But surely if it's artificial intelligence then it'd know its limits and would respond appropriately? Oracle use no problem? It is it because it's actually shit but it's the best thing we've seen yet and everyone is just in denial?

People constantly misevaluate their own limits though. Why should AI not be allowed to do that?

Professionals don't constantly misevaluate their limits, if the AI is to replace a professional it has to know its limits.

Re: Scientists should use AI as a tool, not an oracle

#69

Post by l33t7332273 »

alvah wrote: post #38
godelski wrote: post #17

Earlier quoted context omitted.

To be fair, people did this before ChatGPT. It's just the thing they point to as evidence now, and they'll always find something. The underlying problem is much bigger: 1) people confidently arguing with domain experts about topics that they have little to no experience in. 2) people valuing the opinions of arguers from 1 over experts.

To be extra fair, "domain experts" in some areas have had a bad few years; there are a couple of fields I can think of off the top of my head where the "experts" wheeled out to advise/scare the public are clearly more influenced by politics (or saving their own skin) than science. Replacing trust in experts with trust in LLMs is obviously dumb, but who is Joe Sixpack supposed to turn to?

> there are a couple of fields I can think of off the top of my head where the "experts" wheeled out to advise/scare the public are clearly more influenced by politics (or saving their own skin) than science

This feels like a thinly veiled jab at COVID era public health recommendations. Can you be more clear about which fields you’re referring to?

Re: Scientists should use AI as a tool, not an oracle

#70

Post by userbinator »

People treating tools like they're infallible has been a problem since computers were invented, but IMHO the biggest difference with AI is how confident and convincing it can be in its output. Much like others here, I already have had to convince, very carefully, many otherwise-decently-intelligent people who believed ChatGPT was correct.

Thus I think the biggest success of AI will be the arts, where imprecision is not fatal, and hallucinations turn into entertainment instead of "truths".

Post reply on HN