Scientists should use AI as a tool, not an oracle

> it feels like the first sentence of your argument is more broadly a critique of the credentials of AI Safety proponents

It's a not so thinly veiled critique of Eliezer Yudkowsky.

> Maybe you are distinguishing between doomers vs broader AI Safety proponents,

I do. These are different classes of people. But many doomers mascaraed as AI Safety proponents. Just as many conmen mascarade as ML/AI researchers. I suspect distinguishing the groups is quite difficult for those without domain expertise.

> most people on the CAIS letter (https://www.safe.ai/work/statement-on-ai-risk)

I don't care about the opinion of most of these people (there are some I VERY much do), nor do I think this is a meaningful letter.

Interfacing with a model does not endow one with any level of expertise. If this were true, the whole thread would be ill founded because people using GPT are interfacing with it. Instead, one needs to actually deeply study these models. There are things we know about them, and quite a lot. The term "blackbox" gets thrown around a lot, but that doesn't make everyone's expertise on the matter equally valid. In fact, the more complex something is to understand suggests the fewer number of people are qualified to have a reasonable opinion on the matter. My complaint is we often act as if the opposite is true.[0]

My second big problem with the CAIS letter is it means nothing. All it says is "I don't want to kill all humans." This is a fairly universally agreed upon statement and is in fact the default statement. It does not say anything about the potential risk. That's a completely different matter.

Worse, many of the people who have signed this are literally at the helm of the ships steering us into a dystopian future (which is not covered by this toothless letter). So I'm not sure what meaning this is supposed to have other than pageantry. Do not forget that these are the same exact people pushing and promoting abuse of these tools. I do not blame Average Joe for thinking that GPT is equivalent to Google (which itself cannot be trusted at face value, but this does not make it a useless tool) when that is often the way that it is promoted/advertised. So if you are concerned, I wouldn't use this as evidence.

[0] There's an added problem that you can become above average in any given subject relatively quickly. This is a double edged sword because knowledge is valuable but it often results in one being over confident. And the learning difficulty grows exponentially, which is why there are so few experts in any given subject matter. Because expertise is understanding nuance and complexity. The great irony of the doomers is that they fall back on "unknown unknowns" while not putting effort towards putting a bound on that.

We’re talking about different things.

Let’s look at regular intelligence. Stuff in human brains.

If I ask a human a question like “how many sides does a square have?” and that human says “three”, that’s an objectively wrong answer, and I would say that that person is “wrong”.

The human brain also does not have a narrowly defined specialized purpose and can do a lot of things and yet we are perfectly ok still saying “true” or “false” to the truth values being asserted.

A truth proposition can have a “correct” or “incorrect” value. If a human gives me the wrong bank balance because they misremembered it, it’s still wrong regardless of the actual intelligence associated with it.

So again, I don’t think this kind of pedantry is actually useful because you’re not actually saying anything; you’re saying “these models aren’t trained on being correct, they’re on word patterns”, which was never disputed, but that is orthogonal to the truth value.

I would have thought scientists weren’t going to use these tools to do research considering they as a group are far more exposed to things like peer reviews and critical thinking than general society.

What worries me the most about these AI solutions, however, is their usage in the public sector. They can certainly be useful helpers, like, they can scan images for cancer and if added to existing processes involving humans, often lead to enhanced results. They can’t replace any existing methods, however, as we learned here in Denmark a few years ago. Unfortunately that lesson hasn’t been learned across the public sector. I think medicine and healthcare learned it, but right now, we’re replacing actual human controls, audits and sometimes decision making with AI or an unwarranted trust in AI results. Which is going to lead to some really terrible results considering how bad things like LLMs often are at being lucky in even “common knowledge” situations. It’s further enhanced by how some of the work it’s tasked to do isn’t as black-and-white as writing code is. We use AI tools in our daily work, and they are ok, but as anyone who’s used them for programming probably knows by now, they aren’t exactly great at being lucky. Sometimes they’ll hallucinate solutions that simply do not exist.

This is how they work, and as I said earlier, AIs can be great enhancers. They aren’t replacements though, and if we start treating them like they are, which is very tempting from a change-management and benefit-realisation perspective, we’re just going to get in trouble. This is unfortunately exactly what we’re doing, and why wouldn’t we? Most western public sectors have functioned on at least some form of new public management for two decades by now, sometimes longer. As a result the entire systemic culture is geared toward efficiency and cost reduction, even when it doesn’t really result in either efficiency and cost reduction on a broader perspective.

Now, if scientists are on board. Then what hope does a public bureaucracy have?

For the record I believe it to be immoral to manipulate humanity in this way. And I also believe it might be bad for bussiness.

I was just trying to explain to the guy above what I think the guy above that meant.

That assumes that street slang discussions, using entirely different conceptualizations of ideas, would indeed be embedded nearby one another. Plus, both the street slang and ordinary language will tend to treat the information in a less precise, a less concept discriminating manner (meaning the subtle distinctions between issues may be lost in their discussions). In my tests, I find one indeed needs to use the subject matter expert for precise treatment of formal knowledge and generated answers that are more accurate.