Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scientists should use AI as a tool, not an oracle

aisnakeoil.com

Post reply on HN
Print view View original

81–90 of 113 posts

Re: Scientists should use AI as a tool, not an oracle

#81

Post by antonvs »

tombert wrote: post #39
SrslyJosh wrote: post #28

Earlier quoted context omitted.

> I rarely trust code it generates (at least for the language I mainly work in) as i’ve seen it make some serious mistakes (eg: using keywords in the language that don’t exist) It's only a mistake from your perspective. The model just generates text based the probabilities it learned during training. In that respect, there is no such thing as "incorrect" output because the model doesn't operate at that level of abstr…

Wait, no, it's "incorrect" in the sense that you asked it to do something, and the thing it gives you doesn't accomplish the task. I asked it "what is the PS3 game where the full version of To Kill a Mockingbird is in there?" and it responded back with "The Sabateour", when the correct answer would have been "The Darkness". That is incorrect by most definitions of the word, whether or not it's a consequence of the tr…

> Wait, no, it's "incorrect" in the sense that you asked it to do something, and the thing it gives you doesn't accomplish the task.

You believe you "asked it do something," but that's just you anthropomorphizing the model and your interaction with it. Of course the AI companies encourage that perspective, but it's a factually dubious one at best.

Judging whether a model's output is "correct" involves you imposing an external context on both the prompt and response that the model typically doesn't have access to. It also typically has no ability to test its responses.

This is part of why good prompt engineering can be so important - because what you get out is a function of what you put in, and pretending that the model is a question-answering oracle only takes you so far.

Of course what the AI companies are trying to do is train and prompt the models in such a way that their output is considered "correct" from a user's perspective more often than not. In an interaction with an AI company's salespeople, you might argue about "correctness". But that's not going to help understand what's actually going on.

Re: Scientists should use AI as a tool, not an oracle

#82

Post by elicksaur »

godelski wrote: post #13
unkulunkulu wrote: post #5

> Unfortunately, most scientific fields have succumbed to AI hype, leading to a suspension of common sense. For example, a line of research in political science claimed to predict the onset of civil war with an accuracy2 of well over 90%, a number that should sound facially impossible. (It turned out to be leakage, which is what got us interested in this whole line of research.) This coupled with people acting on its…

This is true for a lot of things, not just AI. But in AI, a guy who didn't get a High School degree and wrote Harry Potter Fan Fiction is one of the leading voices in doomerism. The problem is you can't just "use logic and reason" because simple models are not good enough. The nuance dominates, but that's why we have experts. What's funny to me is that people will confidently argue with experts and others value their…

I don’t trust the machines or the people who make them, and I didn’t have to read the Harry Potter fanfic to know ad hominems are poor arguments. What group does that make me?

Re: Scientists should use AI as a tool, not an oracle

#83

Post by Zambyte »

dotnet00 wrote: post #79
Zambyte wrote: post #74

Earlier quoted context omitted.

You didn't answer the question.

I did, I said they aren't human-like intelligences, so countering with "humans make mistakes, are humans not intelligent?" is drawing a false equivalence between humans and LLMs. Since we do not possess a definition of intelligence that isn't human-like, it would be meaningless to argue if LLMs are intelligent in general. All that can be said is that they are not intelligent in the way that humans are.

The question you answered was rhetorical; obviously humans are intelligent. There is another question that actually has an interesting answer after it. In fact, it's not possible to have a meaningful discussion without answering it. I thought that was obvious :)

Re: Scientists should use AI as a tool, not an oracle

#85

Post by xarope »

flatline wrote: post #15
LouisSayers wrote: post #4

Not just scientists, but everyone! My partner recently went a bit nuts writing an article with the help of GPT4. She was very proud of how productive she'd been until I asked if she'd actually searched for the papers GPT4 had referred to. Of course, many of the referred to papers didn't exist...

That is not writing with the help of GPT 4, that is letting it write for you! I can’t imagine doing anything creative and letting a computer source material for me without having reviewed the material first hand, even if it was accurate. Clearly, this is not where everyone’s head is at, and I suspect your wife’s workflow is more the common case. I’ve said from the outset that in academic settings you should be able t…

personal anecdata: I had written a few paragraphs of factual information, and decided to see what GPT 4 would do with it. So I asked it to rewrite the information several times, using a different voice (e.g. write it with an optimistic view, pessimistic view etc)

EVERY single "fact" was perverted by either mixing with another fact, or misrepresenting by replacing a word like "good" with "superb" or "fantastic" (I guess optimistic means lie-through-your-teeth?)

YMMV, but basically I achieved nothing except a waste of about 30mins and an honest, personal evaluation of the limits of GPT.

Re: Scientists should use AI as a tool, not an oracle

#86

Post by elicksaur »

VS1999 wrote: post #55
SrslyJosh wrote: post #28

Earlier quoted context omitted.

> I rarely trust code it generates (at least for the language I mainly work in) as i’ve seen it make some serious mistakes (eg: using keywords in the language that don’t exist) It's only a mistake from your perspective. The model just generates text based the probabilities it learned during training. In that respect, there is no such thing as "incorrect" output because the model doesn't operate at that level of abstr…

This habit of latching onto one word specifically to ignore what everyone knows is obnoxious, pedantic, and most of the time not even technically correct. It's just stupid quibbling over how words in English can be used to mean different things. And just so you know, the model doesn't "learn" anything, you're just adjusting weights until you get a desired result.

“What everyone knows” - https://www.lesswrong.com/posts/BNfL58ijGawgpkh9b/everybody-...

More broadly, the meaning and usage of specific words are important for these products because they shape how people perceive their utility.

If a thing isn’t “correct” because it has no sense of understanding, and therefore is only “correct” due to projection by the user, then that’s a super important distinction.

Re: Scientists should use AI as a tool, not an oracle

#87

Post by tombert »

antonvs wrote: post #81
tombert wrote: post #39

Earlier quoted context omitted.

Wait, no, it's "incorrect" in the sense that you asked it to do something, and the thing it gives you doesn't accomplish the task. I asked it "what is the PS3 game where the full version of To Kill a Mockingbird is in there?" and it responded back with "The Sabateour", when the correct answer would have been "The Darkness". That is incorrect by most definitions of the word, whether or not it's a consequence of the tr…

> Wait, no, it's "incorrect" in the sense that you asked it to do something, and the thing it gives you doesn't accomplish the task. You believe you "asked it do something," but that's just you anthropomorphizing the model and your interaction with it. Of course the AI companies encourage that perspective, but it's a factually dubious one at best. Judging whether a model's output is "correct" involves you imposing an…

It's actually not "anthropomorphizing the model".

I passed it input in the serialized form known as "English text". I expected a response also in serialized English that I can then decode in my brain to something that comports with reality. If I requested from a web server some JSON giving me my bank balance, and the balance it gave me is not accurately reflecting reality, it's not anthropomorphizing anything to say that it's incorrect, any more than pinging Nginx is.

And to be clear, we can wax philosophical all you want about "correctness", but that's really sidestepping the point: I don't care why it's giving me wrong information.

In my bank example, does it really matter, for the end user, if it's because of some integer overflow error or if it's a null pointer there's just a special `if` statement saying that antonvs account should always print out a different number for your balance.

I think nearly everyone would say that that's incorrect, and it actually wouldn't be clever or insightful for someone to say "no that's just a result of how the computer was programmed! You're imposing a human understanding of correctness on your bank balance!"

Re: Scientists should use AI as a tool, not an oracle

#88

Post by dotnet00 »

Zambyte wrote: post #83
dotnet00 wrote: post #79

Earlier quoted context omitted.

I did, I said they aren't human-like intelligences, so countering with "humans make mistakes, are humans not intelligent?" is drawing a false equivalence between humans and LLMs. Since we do not possess a definition of intelligence that isn't human-like, it would be meaningless to argue if LLMs are intelligent in general. All that can be said is that they are not intelligent in the way that humans are.

The question you answered was rhetorical; obviously humans are intelligent. There is another question that actually has an interesting answer after it. In fact, it's not possible to have a meaningful discussion without answering it. I thought that was obvious :)

Maybe I'm miscommunicating, I know that the question about whether or not humans are intelligent is rhetorical. What I'm trying to say is that "humans make mistakes and are obviously intelligent, so a probabilistic syllable generator can be considered intelligent despite making mistakes" does not necessarily follow because at best they're different kinds of intelligences.

In my rush to be a smartass, I did miss that you also asked what definition of intelligence they were working with though, so I suppose I didn't really add anything besides unnecessary snark (-‿-")

Re: Scientists should use AI as a tool, not an oracle

#89

Post by godelski »

elicksaur wrote: post #82
godelski wrote: post #13

Earlier quoted context omitted.

This is true for a lot of things, not just AI. But in AI, a guy who didn't get a High School degree and wrote Harry Potter Fan Fiction is one of the leading voices in doomerism. The problem is you can't just "use logic and reason" because simple models are not good enough. The nuance dominates, but that's why we have experts. What's funny to me is that people will confidently argue with experts and others value their…

I don’t trust the machines or the people who make them, and I didn’t have to read the Harry Potter fanfic to know ad hominems are poor arguments. What group does that make me?

> I don’t trust the machines or the people who make them

This makes you consistent. I have no problem with this.

> I didn’t have to read the Harry Potter fanfic to know ad hominems are poor arguments

I mostly agree. But my point is that Eliezer Yudkowsky doesn't actually have the qualifications. I want to be clear that academic degrees aren't necessary to qualify someone, just like they aren't necessary to qualify someone as a good programmer. But it is generally harder and the foundation is shakier. In this case, most of his arguments are founded on incorrect assumptions. They are often logical, but it doesn't matter if something is logical if the premise is incorrect.

Post reply on HN