Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scientists should use AI as a tool, not an oracle

aisnakeoil.com

Post reply on HN
Print view View original

31–40 of 113 posts

Re: Scientists should use AI as a tool, not an oracle

#31

Post by immibis »

SrslyJosh wrote: post #23
unkulunkulu wrote: post #5

> Unfortunately, most scientific fields have succumbed to AI hype, leading to a suspension of common sense. For example, a line of research in political science claimed to predict the onset of civil war with an accuracy2 of well over 90%, a number that should sound facially impossible. (It turned out to be leakage, which is what got us interested in this whole line of research.) This coupled with people acting on its…

> are AI safety folks building models of this pattern? First you need to ask if AI "safety folks" actually understand the technology, and if they are thinking about it objectively. If they believe that we're a few years away from accidentally creating Skynet, they need to put down the crack pipe and go work in another field.

We have already created Skynet. Its name is Capitalism. Or the Internet. One of those things.

Re: Scientists should use AI as a tool, not an oracle

#33

Post by alpinisme »

SrslyJosh wrote: post #20
JeremyBarbosa wrote: post #9

Earlier quoted context omitted.

Sadly it's not even just references, LLMs still hallucinate or at least misrepresent even the most basic of facts. That and the stereotypical GPT-verbage makes it impossible to use for writing anything significant.

> LLMs still hallucinate Keep in mind that there's no difference between what happens inside a model when it "hallucinates" vs. when it generates "correct" output. It's the exact same process.

That’s true, but it’s also true of anything else that makes mistakes, including buggy software. When a buggy sorting algorithm produces a bad ordering it’s doing so “with the exact same process” the good ordering is coming from. Ditto for humans and their slips (although tbh I get a little tired of the analogizing of humans and llms…not that the analogies are wrong, but just that we always analogize human minds with the latest technology: wax writing pads through computers)

Re: Scientists should use AI as a tool, not an oracle

#34

Post by Onawa »

dluan wrote: post #14

Scientists have been obsessed with over-optimzing for FOMO for the past decade - what papers should I read that I don't have time for, what grants should I apply for that I don't know about, what projects should I work on that will give me the best ROI, who in my field is poised to disrupt or make a big leap, etc. Some even think that the end goal is actually an autonomous research agent that can make decisions about…

I understand the point that you're making about overoptimizing for FOMO in science. I wanted to give you another perspective from a scientist working within the US government that doesn't care about playing that game.

Our governmental research agency, and NIH as a whole has TONS of research data that we don't have the manpower to screen and provess. There are also gaps in data that AI/ML could help us simulate. AI research assistants could potentially help us process and evaluate "what questions to ask" by, for example, looking for trends in QSAR (quantitative structure-activity relationship) models for novel chemicals and help us direct our attention to compounds of toxicological interest.

We've also been trying to use the AI research assistants to speed up the process of evaluating the scientific literature for toxicologists who have to make regulatory decisions. Our agency has a backlog of chemicals that we would love to evaluate, but lacks the manpower to do so.

No profit motive or much "clout" interest, at least that I've seen. Just a lot of public servant scientists who need some extra help protecting the public.

Re: Scientists should use AI as a tool, not an oracle

#36

Post by LordKeren »

SrslyJosh wrote: post #28
captainkrtek wrote: post #19

In my professional work, I treat chatgpt as a search engine that I feel I can ask questions of in a natural manner. I often find small flaws in technical solutions it offers, but it can still provide useful starting points to investigate. I rarely trust code it generates (at least for the language I mainly work in) as i’ve seen it make some serious mistakes (eg: using keywords in the language that don’t exist)

> I rarely trust code it generates (at least for the language I mainly work in) as i’ve seen it make some serious mistakes (eg: using keywords in the language that don’t exist) It's only a mistake from your perspective. The model just generates text based the probabilities it learned during training. In that respect, there is no such thing as "incorrect" output because the model doesn't operate at that level of abstr…

While yes, this is the technical reason — it’s important to not overlook how non-technical people see LLMs. And not only that, how they are being marketed.

I’m struggling to think of any comparable technology where the regular median users understanding is both fundamentally wrong— and is being purposefully misinformed.

Re: Scientists should use AI as a tool, not an oracle

#37

Post by raincole »

SrslyJosh wrote: post #20
JeremyBarbosa wrote: post #9

Earlier quoted context omitted.

Sadly it's not even just references, LLMs still hallucinate or at least misrepresent even the most basic of facts. That and the stereotypical GPT-verbage makes it impossible to use for writing anything significant.

> LLMs still hallucinate Keep in mind that there's no difference between what happens inside a model when it "hallucinates" vs. when it generates "correct" output. It's the exact same process.

Uh... yes? I'm not sure why it's some significant insight.

Surely when google gives bad results, it's "the same process" as when it gives good results. And when a book gives wrong information, it's the exact same kind of ink as correct information.

Re: Scientists should use AI as a tool, not an oracle

#38

Post by alvah »

godelski wrote: post #17
TheRoque wrote: post #10

The worst is having random people questioning your expertise because of what ChatGPT told them.

To be fair, people did this before ChatGPT. It's just the thing they point to as evidence now, and they'll always find something. The underlying problem is much bigger: 1) people confidently arguing with domain experts about topics that they have little to no experience in. 2) people valuing the opinions of arguers from 1 over experts.

To be extra fair, "domain experts" in some areas have had a bad few years; there are a couple of fields I can think of off the top of my head where the "experts" wheeled out to advise/scare the public are clearly more influenced by politics (or saving their own skin) than science. Replacing trust in experts with trust in LLMs is obviously dumb, but who is Joe Sixpack supposed to turn to?

Re: Scientists should use AI as a tool, not an oracle

#39

Post by tombert »

SrslyJosh wrote: post #28
captainkrtek wrote: post #19

In my professional work, I treat chatgpt as a search engine that I feel I can ask questions of in a natural manner. I often find small flaws in technical solutions it offers, but it can still provide useful starting points to investigate. I rarely trust code it generates (at least for the language I mainly work in) as i’ve seen it make some serious mistakes (eg: using keywords in the language that don’t exist)

> I rarely trust code it generates (at least for the language I mainly work in) as i’ve seen it make some serious mistakes (eg: using keywords in the language that don’t exist) It's only a mistake from your perspective. The model just generates text based the probabilities it learned during training. In that respect, there is no such thing as "incorrect" output because the model doesn't operate at that level of abstr…

Wait, no, it's "incorrect" in the sense that you asked it to do something, and the thing it gives you doesn't accomplish the task.

I asked it "what is the PS3 game where the full version of To Kill a Mockingbird is in there?" and it responded back with "The Sabateour", when the correct answer would have been "The Darkness". That is incorrect by most definitions of the word, whether or not it's a consequence of the training model doesn't really change that.

I suppose we could get into details about epistemology and ontology about the nature of what an answer "is", but I think it's fair to say that "incorrect" is when it gives you something that doesn't accomplish the task you asked it to do, or rather when it tries to accomplish the task but what it gives you don't work.

Re: Scientists should use AI as a tool, not an oracle

#40

Post by EGreg »

LouisSayers wrote: post #4

Not just scientists, but everyone! My partner recently went a bit nuts writing an article with the help of GPT4. She was very proud of how productive she'd been until I asked if she'd actually searched for the papers GPT4 had referred to. Of course, many of the referred to papers didn't exist...

Hmm. In the future the AI in nefarious hands can retroactively make the papers first, and get them past the censors. Just make up a lot of bullshit and then it’s turtles all the way down lmao
Post reply on HN