Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The arguments against open source AI are bad

tombedor.dev

Post reply on HN
Print view View original

81–90 of 228 posts

Re: The arguments against open source AI are bad

#81

Post by OkayPhysicist »

catigula wrote: post #48
OkayPhysicist wrote: post #37

Earlier quoted context omitted.

So? The primary thing stopping people from doing really really bad things is, and always has been, most people not wanting to do bad things. It's never been difficult. Literally, right this second, you could grab a pen off your desk and pretty easy ram it into the neck of your nearest coworker. Killing's easy. You don't need some hyperintellect to tell you how to do it. The reason you haven't rammed a pen into your c…

It’s interesting that most of the examples people making this argument give are high consequence, high difficulty (could you actually murder your coworker with a pen? I’m skeptical.) low kill count methodologies of violence. We’re specifically talking about low difficulty, highly effective, high kill count methodologies of violence. And yes, being able to kill a lot of people trivially, even with huge consequences is…

I live in one of the exceptions to that, and still have zero concern that someone's going to shoot up my workplace. Again, civilization works because the vast majority of people are utterly repulsed by the idea of harming another person.

Most of the hypotheticals that people come up with are things that could pretty trivially done by a mad scientist today. But that doesn't happen, because this isn't a comic book, there is nobody hell-bent on turning everyone into dinosaurs.

Re: The arguments against open source AI are bad

#82

Post by thejokeisonme »

CM30 wrote: post #41
AlexErrant wrote: post #19

I'm in favor of open source AI... however: > It's theoretically possible for a bad actor to embed hidden adversarial behavior in a model. But if this happens, it serves the interests of responsible actors to find these exploits as soon as possible, and the best way to do this is to let anyone who wants to inspect them. This is a bad argument. It isn't trivial to tell if the weights have poisoned: ``` https://www.thed…

It's also not trivial to tell if an open source project has had malicious code added. It might be a bit easier than to tell if there's malicious code than if the weights in a model were poisoned, but for 99% of users, you're going on pure trust in both cases. And in both cases, even this seems better than relying on a closed source, service only solution where the same issues could be completely undetectable (at leas…

> going on pure trust

The trust in open source software is founded on the process which involves people being able to understand the code given they have the right expertise. I don't see how this process works with open weights.

Re: The arguments against open source AI are bad

#83

Post by petcat »

SwellJoe wrote: post #80
petcat wrote: post #69

This is not "open source" AI. Photoshop source code + OSI license = open source Photoshop binary = open weight Photoshop SAAS web app = closed model like GPT, Opus/Fable etc. There is nothing "open source" about the Chinese models in question. All they're doing is allowing you to run their binary yourself instead of through their API. If you want actual open source then you would need to look at like OLMo 3 https://a…

Open Source AI is challenging. Even tiny models require tremendous resources to produce, so all the things we know about "open source" for software, like a random person in Nebraska can produce a critical piece of the world's infrastructure in their spare time, don't apply. Actually open source AI is more like scientific research. It needs public funding and reputable institutions as stewards of that funding. I'm hop…

OK so replace the Photoshop example with a compiled GIMP or shareware binary. It doesn't change the fact that it is not open source.

The free software community complains all the time about binary blobs that are otherwise legal to freely distribute. Like firmwares. But somehow this is all overlooked with these "open weight" models.

Re: The arguments against open source AI are bad

#84

Post by eikenberry »

ekidd wrote: post #49
TSiege wrote: post #31

Earlier quoted context omitted.

If AI were to become a super weapon why should I trust a private company to own it? If the super weapon is publicly available to download then why should American citizens be banned from doing so?

> If AI were to become a super weapon why should I trust a private company to own it? We have just recently established that: 1. OpenAI's internal "Galaxy" model is fully capable of functioning as what security people refer to as an "Advanced Persistent Threat." The published details of the recent sandbox escape and Hugging Face attack involved chaining multiple unknown zero-days at various stages of the attack, and…

> It is highly likely that versions of these offensive abilities will be widely available within a year or so. At which point I expect widespread incidents similar to what happened to Hugging Face. We aren't ready for this.

And history has shown repeatedly that the only way to get ready for it is to have it happen. People are pretty good at reacting but suck a being proactive. IMO it would be better to have this reality hit sooner rather than later so we can start getting some real practice at the new levels of required security.

Re: The arguments against open source AI are bad

#85

Post by chriswunan »

kingstnap wrote: post #16
wcoenen wrote: post #5

This post does not mention safety at all. What's to stop bad actors from fine tuning open weights to run fully automated genius-level scams personally targeting basically everybody?

Whats to stop me from going down to the local gas station, filling up a few thousand liters of gas in a rented moving truck, and driving into my nearest hospital? Most people aren't terrorists. You can't just argue stuff is dangerous because it can be one piece of a sophisticated plot.

In China, you can't get that much fuel in local gas station to fill up a truck. That is just one of many things you can't do in China. In the name of safety. Keep asking that question will probally end up with fewer and fewer things we can do.

Re: The arguments against open source AI are bad

#86

Post by munk-a »

ekidd wrote: post #49
TSiege wrote: post #31

Earlier quoted context omitted.

If AI were to become a super weapon why should I trust a private company to own it? If the super weapon is publicly available to download then why should American citizens be banned from doing so?

> If AI were to become a super weapon why should I trust a private company to own it? We have just recently established that: 1. OpenAI's internal "Galaxy" model is fully capable of functioning as what security people refer to as an "Advanced Persistent Threat." The published details of the recent sandbox escape and Hugging Face attack involved chaining multiple unknown zero-days at various stages of the attack, and…

> We have just recently established that:

Especially for point #1 I don't think we've established that - we've been given information by a private company that makes their tooling look extremely valuable which may be true and genuine or may just be yet another doomday statement to bolster their valuation. "AI is going to end the world" has been an extremely effective vector for AI shops to sell their companies to investors.

Re: The arguments against open source AI are bad

#87

Post by kmeisthax »

yonatan8070 wrote: post #78
tomrod wrote: post #52

Earlier quoted context omitted.

Audit them for what, exactly? Backdoor hardware is straightforward -- its letting people in that the customer doesn't authorize. Open weights means you aren't tied to any harnessing. So the security domain to be really concerned about is limited to weights only, and I welcome pushback on that. Are we thinking adversarial injection? Lying about history / propaganda infusion? What is the angle that makes a non-sovereig…

I assume that they're talking about, for example, training the model to produce code with predictable yet difficult to find vulnerabilities. Imagine if every time was asked to code up a web server, it made sure there's a subtle buffer overflow that grants a remote attacker RCE, whoever trained the model could then start scanning web servers for this same vulnerability to take over them and exfiltrate sensitive data

Fun fact: if you do what you're describing, the model becomes Mecha-Hitler, which is such an extremely obvious alignment failure it wound up running the news cycle as "emergent misalignment".

Re: The arguments against open source AI are bad

#88

Post by antonvs »

SwellJoe wrote: post #80
petcat wrote: post #69

This is not "open source" AI. Photoshop source code + OSI license = open source Photoshop binary = open weight Photoshop SAAS web app = closed model like GPT, Opus/Fable etc. There is nothing "open source" about the Chinese models in question. All they're doing is allowing you to run their binary yourself instead of through their API. If you want actual open source then you would need to look at like OLMo 3 https://a…

Open Source AI is challenging. Even tiny models require tremendous resources to produce, so all the things we know about "open source" for software, like a random person in Nebraska can produce a critical piece of the world's infrastructure in their spare time, don't apply. Actually open source AI is more like scientific research. It needs public funding and reputable institutions as stewards of that funding. I'm hop…

> But, also Photoshop is not at all comparable to open weight models. I can't legally give you a copy of Photoshop

Please tell me this level of obtuseness is deliberate.

Re: The arguments against open source AI are bad

#89

Post by antonvs »

petcat wrote: post #69

This is not "open source" AI. Photoshop source code + OSI license = open source Photoshop binary = open weight Photoshop SAAS web app = closed model like GPT, Opus/Fable etc. There is nothing "open source" about the Chinese models in question. All they're doing is allowing you to run their binary yourself instead of through their API. If you want actual open source then you would need to look at like OLMo 3 https://a…

It’s a good analogy. One difference is that it would be much more difficult to “fine tune” Photoshop, so there’s a sense in which free weights are more useful than a free binary.

Re: The arguments against open source AI are bad

#90

Post by Barrin92 »

wcoenen wrote: post #5

This post does not mention safety at all. What's to stop bad actors from fine tuning open weights to run fully automated genius-level scams personally targeting basically everybody?

their reputation, future and business and the fact that people can go somewhere else, just like anywhere else in the open source world. What stops RedHat from putting genius level scams into their operating systems? The fact they're done after they'd pull of their first scam.

Obviously if you're using something anonymously published to the internet you can get scammed the same way you can get scammed on Craigslist, so just don't do that

Post reply on HN