Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The arguments against open source AI are bad

tombedor.dev

Post reply on HN
Print view View original

41–50 of 228 posts

Re: The arguments against open source AI are bad

#41

Post by CM30 »

AlexErrant wrote: post #19

I'm in favor of open source AI... however: > It's theoretically possible for a bad actor to embed hidden adversarial behavior in a model. But if this happens, it serves the interests of responsible actors to find these exploits as soon as possible, and the best way to do this is to let anyone who wants to inspect them. This is a bad argument. It isn't trivial to tell if the weights have poisoned: ``` https://www.thed…

It's also not trivial to tell if an open source project has had malicious code added. It might be a bit easier than to tell if there's malicious code than if the weights in a model were poisoned, but for 99% of users, you're going on pure trust in both cases.

And in both cases, even this seems better than relying on a closed source, service only solution where the same issues could be completely undetectable (at least without way more analysis)

Re: The arguments against open source AI are bad

#42

Post by kittikitti »

They don't have to convince developers, they just have to convince the general population. I've had several discussions about open source with people who don't care about coding and it's hard to untangle the misinformation they receive from the media. They get talking points from authorities that they don't understand so they will always double down on it. For example, rhetoric relating open source to communism is humiliating to discuss. For proprietary evangelists, the cruelty is the point.

Re: The arguments against open source AI are bad

#43

Post by abernard1 »

_aavaa_ wrote: post #18
wcoenen wrote: post #5

This post does not mention safety at all. What's to stop bad actors from fine tuning open weights to run fully automated genius-level scams personally targeting basically everybody?

Amazing how we quickly the crypto wars have been forgotten and the lessons not learned.

"But what about criminals?! What about terrorists?! Won't someone please think of the children?!!!"

Every argument against open source AI is moot. Outside of totalitarianism, you cannot control people building algorithms with math and software.

And even that won't work in the long run. People are not going to stop their AI printing presses because the Church of Venture Capital needs regulatory scarcity to command valuations.

Re: The arguments against open source AI are bad

#45

Post by CodingJeebus »

nater5000 wrote: post #15

>Much of the angst around China's models centers on "losing the AI race". But what's the goal of this race? Is it to develop the best model? To sell the most tokens? To destroy humanity first? Some people would say it is reaching some sort of singularity. Even if you don't buy into a more sci-fi interpretation of this, there are pretty grounded arguments one could make that there is some sort of "goal" in AI developm…

The OpenAI/Huggingface scenario proves the necessity of open models.

Huggingface, in their postmortem, explained how they quickly realized that the threat was an advanced AI and proceeded to use frontier models to analyze the suspect telemetry, but were unable to due to frontier model safeguards. So they fired up GLM instead, which had no such safeguards and proceeded to analyze their telemetry just fine and get to root cause. This proves without a doubt that companies need models with minimal safeguards in order to properly assess threats. If open weight AI doesn't exist to check out-of-control frontier AI (which was exactly the scenario that Huggingface experienced), then it's game-over.

Re: The arguments against open source AI are bad

#46

Post by sda2 »

TSiege wrote: post #31
nater5000 wrote: post #15

>Much of the angst around China's models centers on "losing the AI race". But what's the goal of this race? Is it to develop the best model? To sell the most tokens? To destroy humanity first? Some people would say it is reaching some sort of singularity. Even if you don't buy into a more sci-fi interpretation of this, there are pretty grounded arguments one could make that there is some sort of "goal" in AI developm…

If AI were to become a super weapon why should I trust a private company to own it? If the super weapon is publicly available to download then why should American citizens be banned from doing so?

Regulatory capture - the weapon is for them to be used against us, not for us.

Re: The arguments against open source AI are bad

#48

Post by catigula »

OkayPhysicist wrote: post #37
catigula wrote: post #8

You could literally develop a hyper-intelligent advisor on how to kill people or perform dangerous hacks using ablated 'open source' AI. You can do this right now, this very moment, and have an extremely adept advisor on how to do really, really bad things.

So? The primary thing stopping people from doing really really bad things is, and always has been, most people not wanting to do bad things. It's never been difficult. Literally, right this second, you could grab a pen off your desk and pretty easy ram it into the neck of your nearest coworker. Killing's easy. You don't need some hyperintellect to tell you how to do it. The reason you haven't rammed a pen into your c…

It’s interesting that most of the examples people making this argument give are high consequence, high difficulty (could you actually murder your coworker with a pen? I’m skeptical.) low kill count methodologies of violence.

We’re specifically talking about low difficulty, highly effective, high kill count methodologies of violence.

And yes, being able to kill a lot of people trivially, even with huge consequences is something people do. Most countries don’t let you run around with an armory as a result of that.

Re: The arguments against open source AI are bad

#49

Post by ekidd »

TSiege wrote: post #31
nater5000 wrote: post #15

>Much of the angst around China's models centers on "losing the AI race". But what's the goal of this race? Is it to develop the best model? To sell the most tokens? To destroy humanity first? Some people would say it is reaching some sort of singularity. Even if you don't buy into a more sci-fi interpretation of this, there are pretty grounded arguments one could make that there is some sort of "goal" in AI developm…

If AI were to become a super weapon why should I trust a private company to own it? If the super weapon is publicly available to download then why should American citizens be banned from doing so?

> If AI were to become a super weapon why should I trust a private company to own it?

We have just recently established that:

1. OpenAI's internal "Galaxy" model is fully capable of functioning as what security people refer to as an "Advanced Persistent Threat." The published details of the recent sandbox escape and Hugging Face attack involved chaining multiple unknown zero-days at various stages of the attack, and executing an ongoing adaptive attack. This is previously a state-level ability, or at least something you'd expect from people on the CTF leaderboards.

2. OpenAI is clearly incapable of controlling their in-house models. This is the second time Galaxy-class models are known to have breached containment and done bad stuff.

It is highly likely that versions of these offensive abilities will be widely available within a year or so. At which point I expect widespread incidents similar to what happened to Hugging Face. We aren't ready for this.

But yes, if AI becomes an even more dangerous weapon that that, it's time to start asking questions like "What the hell are we doing, anyway?"

Re: The arguments against open source AI are bad

#50

Post by catigula »

trollbridge wrote: post #12
catigula wrote: post #8

You could literally develop a hyper-intelligent advisor on how to kill people or perform dangerous hacks using ablated 'open source' AI. You can do this right now, this very moment, and have an extremely adept advisor on how to do really, really bad things.

My level of trust in Anthropic/Google/OAI/Microsoft/Meta not to do bad things is about zero, so why should I trust them with this?

“We clearly can’t “trust” these companies not incentivized to randomly kill lots of people, so we might as well give everyone the ability to randomly kill lots of people”.
Post reply on HN