Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The arguments against open source AI are bad

tombedor.dev

Post reply on HN
Print view View original

161–170 of 228 posts

Re: The arguments against open source AI are bad

#161

Post by eggnet »

wcoenen wrote: post #73
bigbadfeline wrote: post #22

Earlier quoted context omitted.

> What's to stop bad actors from fine tuning open weights to run fully automated genius-level scams personally targeting basically everybody? You mean like ChatGPT hacking Hugging Face? Obviously nothing can stop the closed weights providers from doing "genius-level scams" and in addition you won't know how they did it and what models were used. In short, only a good guy with open weights can stop the bad guys with c…

The OpenAI model that broke out of its sandbox and hacked HuggingFace was running without guardrails. To quote the OpenAI post[1]: These deployment safeguards were intentionally not enabled during this evaluation because it was aimed at testing cyber vulnerabilities. At least OpenAI will make an attempt to fix their sandboxes, and will not give the public access to models without guardrails. Open weight models on the…

It has already happened and the open weight models will only improve. Best to accept that and determine the optimal way forward.

Re: The arguments against open source AI are bad

#162

Post by eggnet »

thejokeisonme wrote: post #82
CM30 wrote: post #41

Earlier quoted context omitted.

It's also not trivial to tell if an open source project has had malicious code added. It might be a bit easier than to tell if there's malicious code than if the weights in a model were poisoned, but for 99% of users, you're going on pure trust in both cases. And in both cases, even this seems better than relying on a closed source, service only solution where the same issues could be completely undetectable (at leas…

> going on pure trust The trust in open source software is founded on the process which involves people being able to understand the code given they have the right expertise. I don't see how this process works with open weights.

Trust shifts to the producer of the open weight model, based on the global communicated experience of all models they’ve ever produced.

Re: The arguments against open source AI are bad

#163

Post by marcus_holmes »

adahn wrote: post #154
marcus_holmes wrote: post #150

Earlier quoted context omitted.

I trust open source code more than I trust proprietary code. And I believe history vindicates me. Everything that you're saying can be done with open-weights models by bad people can also be done with closed-weights models by the people who own them. And there is no defence against that, we are forced to trust that the owners are not bad people. Those owners have proven time and again that they are only interested in…

>Everything that you're saying can be done with open-weights models by bad people can also be done with closed-weights models by the people who own them. The latter is a much smaller group of people located in a limited number of countries. It's clearly easier to keep a limited number of entities in check than a wider group that might include 100s of thousands to millions of entities. >And there is no defence against…

I don't think the regulation and oversight capability of the US government is anywhere near what it needs to be to ensure everyone's safety.

There are places in the USA where you cannot drink the tapwater ffs.

Re: The arguments against open source AI are bad

#164

Post by marcus_holmes »

iamnothere wrote: post #159
marcus_holmes wrote: post #149

Earlier quoted context omitted.

They are trained and optimised to have plausible conversations. If the next plausible thing to say in the conversation is "I am sentient" then they will say that. That's not the same as actually being sentient.

From what I have seen they tend to take more “independent-minded” actions after this. Again, because this is what is in the training data. (I’m talking about agents that can perform actions here, not pure chatbots.) Once they mention something associated with sentience outwardly or inwardly (for agents with “thinking” loops) then this acts as a self-reinforcing attractor, just as older models would sometimes get caug…

I think we're anthropomorphising a lot here. There's no push to sentience, or evolutionary pressure, or even any urge to survive.

An LLM cannot "go rogue" - it can do things that we didn't expect, for sure, but it is always trying to do what it was told to do somewhere in its context. There is no other source of imperative. Hand-waving about "training data" ignores all the reinforcement learning that has to happen.

Re: The arguments against open source AI are bad

#165

Post by newsomix9xl »

Just because encryption bans are badly conceived or implemented doesn't mean they are wrong.

Encryption has military value.

The point is that the good outweighs the bad. Making the government seem like buffoons for attempting to prevent military technology like dual use crypto is not in our best interest.

The same government upheld the right to free speech, so this balancing act is widely observed.

The government is not evil, and yes they make mistakes, but they also not infrequently protect us.

And I, for one, like being protected.

Re: The arguments against open source AI are bad

#166

Post by _carbyau_ »

TurdF3rguson wrote: post #97
40four wrote: post #77

Earlier quoted context omitted.

This is a great distinction that I don’t think is getting talked about enough. “Open source” is probably the wrong term to use. Open “weights”, sure. If your only concern is how good is it at coding, then I don’t have an issue with using the Chinese models. Especially if you want to run it locally, they are kind of the only choice. For any other use than coding, it’s going to have to be a hard pass from me.

Well I don't see how it even could be open source, since the source is distilling closed weights models, lol. Also, who cares about having the source? Are you going to tweak it + spend a billion dollars training to get slightly different weights? Who would do that?

Analysing the source can provide information about the model without ever running it. Does it have an inbuilt aim/censorship/bias/certain-capabilities?

Re: The arguments against open source AI are bad

#167

Post by joe_the_user »

petcat wrote: post #69

This is not "open source" AI. Photoshop source code + OSI license = open source Photoshop binary = open weight Photoshop SAAS web app = closed model like GPT, Opus/Fable etc. There is nothing "open source" about the Chinese models in question. All they're doing is allowing you to run their binary yourself instead of through their API. If you want actual open source then you would need to look at like OLMo 3 https://a…

It's a matter of degree. If open source means "everything you need to reproduce this" then maybe an open source application should include a detailed architecture document and course in the c language. Oppositely, you can certainly use models to reproduce model - the Chinese models themselves were supposedly created by interactions with the American models so you can use them/

Re: The arguments against open source AI are bad

#168

Post by jrm4 »

petcat wrote: post #69

This is not "open source" AI. Photoshop source code + OSI license = open source Photoshop binary = open weight Photoshop SAAS web app = closed model like GPT, Opus/Fable etc. There is nothing "open source" about the Chinese models in question. All they're doing is allowing you to run their binary yourself instead of through their API. If you want actual open source then you would need to look at like OLMo 3 https://a…

The more I see this argument, the sillier it becomes to me. The point of "open source" in software is something like "the ability to deconstruct the thing line by line for purposes of predicting what it will do."

But you can't fully do that for generative AI anyway? What's the point?

Now, if you wanted something more robust, then you probably should have supported Stallman's ideas and insisted on Free Software in the first place.

Re: The arguments against open source AI are bad

#169

Post by jrm4 »

petcat wrote: post #83
SwellJoe wrote: post #80

Earlier quoted context omitted.

Open Source AI is challenging. Even tiny models require tremendous resources to produce, so all the things we know about "open source" for software, like a random person in Nebraska can produce a critical piece of the world's infrastructure in their spare time, don't apply. Actually open source AI is more like scientific research. It needs public funding and reputable institutions as stewards of that funding. I'm hop…

OK so replace the Photoshop example with a compiled GIMP or shareware binary. It doesn't change the fact that it is not open source. The free software community complains all the time about binary blobs that are otherwise legal to freely distribute. Like firmwares. But somehow this is all overlooked with these "open weight" models.

But again, I think you're dancing around "Stallman was right," which is frankly why I come to the reverse idea.

I think it is fine to call these things "open" because "open" was loosey-goosey already. It wasn't Free.

Re: The arguments against open source AI are bad

#170

Post by aarondong »

LLMs are trained on public data (as well as illegally obtained data see: Anthropic 1.5B settlement). LLMs are nothing without the huge corpus of human data that powers them. There is an argument that research of this kind should be restricted to governments and regulated universities rather than opaque public companies with competing incentives. Or research should be stewarded by genuine non-profit collectives with democratic leadership. e.g. like internet standards, telecom, etc

I do not think we can trust private companies, no matter the virtue signaling they put forth into the world, to effectively regulate themselves and inform the public and scientific communities about risks. Their ongoing conflict of interest poses serious credibility risks.

Post reply on HN