Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scientists should use AI as a tool, not an oracle

aisnakeoil.com

Post reply on HN
Print view View original

21–30 of 113 posts

Re: Scientists should use AI as a tool, not an oracle

#21

Post by Mathnerd314 »

flatline wrote: post #15
LouisSayers wrote: post #4

Not just scientists, but everyone! My partner recently went a bit nuts writing an article with the help of GPT4. She was very proud of how productive she'd been until I asked if she'd actually searched for the papers GPT4 had referred to. Of course, many of the referred to papers didn't exist...

That is not writing with the help of GPT 4, that is letting it write for you! I can’t imagine doing anything creative and letting a computer source material for me without having reviewed the material first hand, even if it was accurate. Clearly, this is not where everyone’s head is at, and I suspect your wife’s workflow is more the common case. I’ve said from the outset that in academic settings you should be able t…

I've sort of worked out a workflow. Like say I had to write an essay and take a side for/against something. Then I would ask GPT to write the strongest argument for, and the strongest argument against, telling it to make up whatever sources it wants. Then after reading those, I would have some idea of my own opinions. I would write from scratch but with the GPT for/against pulled up alongside as reference for how to structure the arguments. Then I would put it through GPT again for proofreading and grammar (or just spelling, if there is AI detection software).

It is a bit tricky though, there are definitely points that come up with GPT that people would not think of normally. So in that sense it is still distinguishable from writing solely by oneself, but I would argue the GPT-assisted essays are just better writing and more well-rounded.

Re: Scientists should use AI as a tool, not an oracle

#23

Post by SrslyJosh »

unkulunkulu wrote: post #5

> Unfortunately, most scientific fields have succumbed to AI hype, leading to a suspension of common sense. For example, a line of research in political science claimed to predict the onset of civil war with an accuracy2 of well over 90%, a number that should sound facially impossible. (It turned out to be leakage, which is what got us interested in this whole line of research.) This coupled with people acting on its…

> are AI safety folks building models of this pattern?

First you need to ask if AI "safety folks" actually understand the technology, and if they are thinking about it objectively. If they believe that we're a few years away from accidentally creating Skynet, they need to put down the crack pipe and go work in another field.

Re: Scientists should use AI as a tool, not an oracle

#24

Post by duxup »

skywhopper wrote: post #12
duxup wrote: post #8

We use search that way, don’t see why AI trained on similar content wouldn’t be just variable in terms of reliability.

This is incredibly simplistic. Search engine results give a lot of context clues about the reliability of their asserted facts and provide a potential spectrum of answers. LLM-generated answers strip all that away, and give a single authoritatively phrased answer. Even if you’re inclined to disbelieve it, the LLM answer gives you no ability to dig in, refine, or compare. It just is. If you ask a chatbot if it’s sure,…

I think you overestimate people’s ability to sniff out bad data on the internet.

Also are you suggesting people fact check an AI by asking it if it is correct? That seems absurd.

Re: Scientists should use AI as a tool, not an oracle

#25

Post by throwanem »

wegfawefgawefg wrote: post #16
throwanem wrote: post #11

Earlier quoted context omitted.

How would that work, do you think?

If you knew everyone would ask gpt before doing anything, you would make gpt say what woudl generally be considered the better option. Not going to war, not committing suicide, etc. In this way even if war was the optimal decision according to some other utility function, the behavior of people is directed in a positive way. (Presumably)

Sure, if you also assume people follow whatever advice so given. They won't, even before the covert influence effort becomes popular knowledge, as it inevitably will. This destroys consumer trust in your product after you have successfully made that product indispensable, thus opening up a previously impossible vacuum in epistemology and thus access to power.

Re: Scientists should use AI as a tool, not an oracle

#26

Post by switchbak »

JeremyBarbosa wrote: post #9
LouisSayers wrote: post #4

Not just scientists, but everyone! My partner recently went a bit nuts writing an article with the help of GPT4. She was very proud of how productive she'd been until I asked if she'd actually searched for the papers GPT4 had referred to. Of course, many of the referred to papers didn't exist...

Sadly it's not even just references, LLMs still hallucinate or at least misrepresent even the most basic of facts. That and the stereotypical GPT-verbage makes it impossible to use for writing anything significant.

We were already swimming in a world of bullshit prior to the wide availability of these. I'm not sure what the future holds, but I think intelligent people are going to become very skeptical of virtually all information sources.

I would imagine there's also a raft of people who will use it as a reason to give up on any search for truth.

I still do hold a lot of hope for their eventual capabilities, but I'm also pretty pessimistic on what the direct and Nth order social effects will be.

Re: Scientists should use AI as a tool, not an oracle

#28

Post by SrslyJosh »

captainkrtek wrote: post #19

In my professional work, I treat chatgpt as a search engine that I feel I can ask questions of in a natural manner. I often find small flaws in technical solutions it offers, but it can still provide useful starting points to investigate. I rarely trust code it generates (at least for the language I mainly work in) as i’ve seen it make some serious mistakes (eg: using keywords in the language that don’t exist)

> I rarely trust code it generates (at least for the language I mainly work in) as i’ve seen it make some serious mistakes (eg: using keywords in the language that don’t exist)

It's only a mistake from your perspective. The model just generates text based the probabilities it learned during training. In that respect, there is no such thing as "incorrect" output because the model doesn't operate at that level of abstraction.

Re: Scientists should use AI as a tool, not an oracle

#29

Post by mewpmewp2 »

logrot wrote: post #18

But surely if it's artificial intelligence then it'd know its limits and would respond appropriately? Oracle use no problem? It is it because it's actually shit but it's the best thing we've seen yet and everyone is just in denial?

People constantly misevaluate their own limits though. Why should AI not be allowed to do that?

Re: Scientists should use AI as a tool, not an oracle

#30

Post by bsenftner »

Mathnerd314 wrote: post #21
flatline wrote: post #15

Earlier quoted context omitted.

That is not writing with the help of GPT 4, that is letting it write for you! I can’t imagine doing anything creative and letting a computer source material for me without having reviewed the material first hand, even if it was accurate. Clearly, this is not where everyone’s head is at, and I suspect your wife’s workflow is more the common case. I’ve said from the outset that in academic settings you should be able t…

I've sort of worked out a workflow. Like say I had to write an essay and take a side for/against something. Then I would ask GPT to write the strongest argument for, and the strongest argument against, telling it to make up whatever sources it wants. Then after reading those, I would have some idea of my own opinions. I would write from scratch but with the GPT for/against pulled up alongside as reference for how to…

There is a subtle aspect of LLM AIs that is lost to most people: they are trained on the entirety of the Internet. That means whatever topic you ask these LLM AIs, there are multiple instances of that same information with different levels of seriousness and accuracy in their treatment of the subject.

For example: if one asks a question using street slang, the answer generated will be generated from training data about your subject, but from online sources that used street slang in their conversation about that issue. Likewise, if you use ordinary language for your question, the generated response will be from ordinary language conversations of your topic. However, if your question concerns any type of formalized knowledge, by asking your question using the formal language of experts in that topic, then the generated AI answer will come from training data that used this same formal expert terms, and are most likely to be correct, because they come from discussions of that subject’s matter experts.

Plus, don't use LLMs for fact retrieval, use them as strategy guides. They really excel as strategy advisors.

Post reply on HN