Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scientists should use AI as a tool, not an oracle

aisnakeoil.com

Post reply on HN
Print view View original

51–60 of 113 posts

Re: Scientists should use AI as a tool, not an oracle

#51

Post by choilive »

bsenftner wrote: post #30
Mathnerd314 wrote: post #21

Earlier quoted context omitted.

I've sort of worked out a workflow. Like say I had to write an essay and take a side for/against something. Then I would ask GPT to write the strongest argument for, and the strongest argument against, telling it to make up whatever sources it wants. Then after reading those, I would have some idea of my own opinions. I would write from scratch but with the GPT for/against pulled up alongside as reference for how to…

There is a subtle aspect of LLM AIs that is lost to most people: they are trained on the entirety of the Internet. That means whatever topic you ask these LLM AIs, there are multiple instances of that same information with different levels of seriousness and accuracy in their treatment of the subject. For example: if one asks a question using street slang, the answer generated will be generated from training data abo…

Theres actually even more subtlety here, in all of your examples the "knowledge" should theoretically be embedded nearby each other in the same vector space, so regardless of the style of language used, semantically they should all pull from similar weights, and thus give similar answers. This is one of the reasons why LLMs are so powerful.. because they seemingly understand the semantic relationships of words so regardless if the prompt is posed casually or formally it should give similar answers in terms of factuality. I agree with you that LLMs today should be primarily used for more creative output.

Re: Scientists should use AI as a tool, not an oracle

#52

Post by nomel »

SrslyJosh wrote: post #28
captainkrtek wrote: post #19

In my professional work, I treat chatgpt as a search engine that I feel I can ask questions of in a natural manner. I often find small flaws in technical solutions it offers, but it can still provide useful starting points to investigate. I rarely trust code it generates (at least for the language I mainly work in) as i’ve seen it make some serious mistakes (eg: using keywords in the language that don’t exist)

> I rarely trust code it generates (at least for the language I mainly work in) as i’ve seen it make some serious mistakes (eg: using keywords in the language that don’t exist) It's only a mistake from your perspective. The model just generates text based the probabilities it learned during training. In that respect, there is no such thing as "incorrect" output because the model doesn't operate at that level of abstr…

This is like saying the arguments put forth by a schizophrenic lawyer are rational and correct.

If the context is that it's a tool, correct is defined as reality within the context of the use of that tool. If it's to find facts, it can be incorrect, since the context of a fact is reality. If it's writing a story, then "correct" would be based on continuity, etc.

If you're using it as a tool to generate words related to previous ones, then sure, it's always correct, but that's not probably not a useful tool for most people. But, being a next word predictor doesn't mean it can't also be a useful tool in real world contexts. There are, literally, billions of dollars being spent on pushing them to be more "correct" in more contexts, so it's a useful concept being considered, even though they're "just" next word predictors.

Re: Scientists should use AI as a tool, not an oracle

#54

Post by bamboozled »

immibis wrote: post #31
SrslyJosh wrote: post #23

Earlier quoted context omitted.

> are AI safety folks building models of this pattern? First you need to ask if AI "safety folks" actually understand the technology, and if they are thinking about it objectively. If they believe that we're a few years away from accidentally creating Skynet, they need to put down the crack pipe and go work in another field.

We have already created Skynet. Its name is Capitalism. Or the Internet. One of those things.

I know your downvoted but I don’t think you would be if you had of just said, corporations.

I think oil companies are the greatest existential threat to humanity via lobbying and eventually climate destruction. Second is social media companies. It’s so easy to spread misinformation against a whole populace and it’s going south fast.

There is just too much incentive for those vested in these corporations to just stop.

Re: Scientists should use AI as a tool, not an oracle

#55

Post by VS1999 »

SrslyJosh wrote: post #28
captainkrtek wrote: post #19

In my professional work, I treat chatgpt as a search engine that I feel I can ask questions of in a natural manner. I often find small flaws in technical solutions it offers, but it can still provide useful starting points to investigate. I rarely trust code it generates (at least for the language I mainly work in) as i’ve seen it make some serious mistakes (eg: using keywords in the language that don’t exist)

> I rarely trust code it generates (at least for the language I mainly work in) as i’ve seen it make some serious mistakes (eg: using keywords in the language that don’t exist) It's only a mistake from your perspective. The model just generates text based the probabilities it learned during training. In that respect, there is no such thing as "incorrect" output because the model doesn't operate at that level of abstr…

This habit of latching onto one word specifically to ignore what everyone knows is obnoxious, pedantic, and most of the time not even technically correct. It's just stupid quibbling over how words in English can be used to mean different things. And just so you know, the model doesn't "learn" anything, you're just adjusting weights until you get a desired result.

Re: Scientists should use AI as a tool, not an oracle

#56

Post by godelski »

alvah wrote: post #38
godelski wrote: post #17

Earlier quoted context omitted.

To be fair, people did this before ChatGPT. It's just the thing they point to as evidence now, and they'll always find something. The underlying problem is much bigger: 1) people confidently arguing with domain experts about topics that they have little to no experience in. 2) people valuing the opinions of arguers from 1 over experts.

To be extra fair, "domain experts" in some areas have had a bad few years; there are a couple of fields I can think of off the top of my head where the "experts" wheeled out to advise/scare the public are clearly more influenced by politics (or saving their own skin) than science. Replacing trust in experts with trust in LLMs is obviously dumb, but who is Joe Sixpack supposed to turn to?

I'm not sure which domains you're referring to.

I can think of domains where sensationalist opinions are lifted, but not ones where the general consensus is blatantly false. I can think of plenty of instances where large news organizations have grossly misrepresented conclusions of research.

> but who is Joe Sixpack supposed to turn to?

This, I agree with. It is why I actively voice dissent, as an expert and in areas where I have domain expertise, against so-called science communicators (not all are "so-called") and when the news gets it wrong.

Hell, I'll do this when actual science communicators get it wrong. Like when Niel DeGrassee Tyson is being dumb[0]. He also thinks hydrogen bombs don't have fallout...[1]. They do...

That said, I still don't think this is a reason to distrust scientists. But I think it is important for scientists to speak out when communicators get it wrong. I think this is a common problem and allows the conmen to gain power. But that's not the only force at play. Truth is complex. Approximate truth is bounded in complexity. But lies can be infinitely simple. So we get it wrong when we "reason our way through" something, because typically the base assumptions are wrong. This makes many conmen truly believe the lies that they are selling.

Joe Sixpack can reason through that. But Joe Sixpack can also reason through the concept that if he was easily able to reason through something and that experts disagree, it's pretty likely there's a reason why other than them being dumb and knowing better. Can, but doesn't. And we as the public let that happen. This may seem like an insurmountable problem, but instead it is a problem which just needs sufficient effort. Momentum builds, so the more people that push against this, the more common it'll become. And to be clear, it is perfectly fine to question experts. It is not perfectly fine to confidently disagree while not actually understanding the topic. If you don't know the difference, read a few papers/works in the topic and see if you can understand 90+% of it (if it is CS or Engineering, see if you can replicate).

[0] https://www.youtube.com/shorts/a-PHXGmexxM

[1] https://www.youtube.com/watch?v=QGa4ItIOCRg

Re: Scientists should use AI as a tool, not an oracle

#58

Post by FinchNova12 »

godelski wrote: post #13
unkulunkulu wrote: post #5

> Unfortunately, most scientific fields have succumbed to AI hype, leading to a suspension of common sense. For example, a line of research in political science claimed to predict the onset of civil war with an accuracy2 of well over 90%, a number that should sound facially impossible. (It turned out to be leakage, which is what got us interested in this whole line of research.) This coupled with people acting on its…

This is true for a lot of things, not just AI. But in AI, a guy who didn't get a High School degree and wrote Harry Potter Fan Fiction is one of the leading voices in doomerism. The problem is you can't just "use logic and reason" because simple models are not good enough. The nuance dominates, but that's why we have experts. What's funny to me is that people will confidently argue with experts and others value their…

Not sure if you intended this, but it feels like the first sentence of your argument is more broadly a critique of the credentials of AI Safety proponents. Maybe you are distinguishing between doomers vs broader AI Safety proponents, but if not, I feel like the counterargument is that most people on the CAIS letter (https://www.safe.ai/work/statement-on-ai-risk) interface quite frequently with these AI models and are also (purportedly) seriously concerned about AI safety

Re: Scientists should use AI as a tool, not an oracle

#59

Post by gerdesj »

unkulunkulu wrote: post #5

> Unfortunately, most scientific fields have succumbed to AI hype, leading to a suspension of common sense. For example, a line of research in political science claimed to predict the onset of civil war with an accuracy2 of well over 90%, a number that should sound facially impossible. (It turned out to be leakage, which is what got us interested in this whole line of research.) This coupled with people acting on its…

"accuracy2" sigh - the 2 is a superscript to a footnote and not a domain specific term.

"facially impossible" ... does that really riff on "on the face of it", or is it farcically misspelt?

Garbage in, garbage out 8)

Re: Scientists should use AI as a tool, not an oracle

#60

Post by Zambyte »

shmatt wrote: post #22

I feel like 90% of AI discussions online these days can be shut down with “a probabilistic syllable generator is not intelligence”

Humans are not fact machines, we are often wrong. Do humans not have intelligence?

What do you even mean by "intelligence" when you say a probabilistic syllable generator "is not intelligence"?

Post reply on HN