The arguments against open source AI are bad • Hacker News Board

Re: The arguments against open source AI are bad

#71

Post by Varelion » Thu, Jul 23, 2026, 8:06 PM UTC

nater5000 wrote: post #15

>Much of the angst around China's models centers on "losing the AI race". But what's the goal of this race? Is it to develop the best model? To sell the most tokens? To destroy humanity first? Some people would say it is reaching some sort of singularity. Even if you don't buy into a more sci-fi interpretation of this, there are pretty grounded arguments one could make that there is some sort of "goal" in AI developm…

> Some people would say it is reaching some sort of singularity.

Those people never took Systems Thinking, or are failing to apply it to AI as a non-closed system.

Re: The arguments against open source AI are bad

#72

Post by SwellJoe » Thu, Jul 23, 2026, 8:12 PM UTC

The way Anthropic is playing the "AI is scary and dangerous and only we can be trusted with it" game is clearly aimed at regulatory capture, which would only be good for Anthropic. It's worrying how cozy some of the tech leaders have been with the Trump administration, which is also clearly aimed at regulatory capture that serves the interests of the biggest tech companies and no one else.

The US has already been set back a decade or more by anti-science and anti-intellectual purges of agencies and formerly non-partisan positions. A ban on open models would be pretty much game over for the US as a tech leader. But, Anthropic would make a lot of money for a couple of years, so, who can say what's right?

Re: The arguments against open source AI are bad

#73

Post by wcoenen » Thu, Jul 23, 2026, 8:16 PM UTC

bigbadfeline wrote: post #22

wcoenen wrote: post #5

This post does not mention safety at all. What's to stop bad actors from fine tuning open weights to run fully automated genius-level scams personally targeting basically everybody?

> What's to stop bad actors from fine tuning open weights to run fully automated genius-level scams personally targeting basically everybody? You mean like ChatGPT hacking Hugging Face? Obviously nothing can stop the closed weights providers from doing "genius-level scams" and in addition you won't know how they did it and what models were used. In short, only a good guy with open weights can stop the bad guys with c…

The OpenAI model that broke out of its sandbox and hacked HuggingFace was running without guardrails. To quote the OpenAI post[1]: These deployment safeguards were intentionally not enabled during this evaluation because it was aimed at testing cyber vulnerabilities.

At least OpenAI will make an attempt to fix their sandboxes, and will not give the public access to models without guardrails. Open weight models on the other hand will run without guardrails almost by definition. I don't think it's wise to provide those capabilities to scam call centers.

[1] https://openai.com/index/hugging-face-model-evaluation-secur...

Re: The arguments against open source AI are bad

#74

Post by bigfishrunning » Thu, Jul 23, 2026, 8:17 PM UTC

iamnothere wrote: post #58

bluefirebrand wrote: post #56

Earlier quoted context omitted.

> But yes, if AI becomes an even more dangerous weapon that that, it's time to start asking questions like "What the hell are we doing, anyway The answer seems to be "getting the weapon before other people get the weapon" which is unfortunate

I like how everyone assumes that they are going to maintain control of a sentient weapon

You've made the assumption (and you're far from alone) that LLMs will one day be sentient...

Re: The arguments against open source AI are bad

#75

Post by iamnothere » Thu, Jul 23, 2026, 8:22 PM UTC

bigfishrunning wrote: post #74

iamnothere wrote: post #58

Earlier quoted context omitted.

I like how everyone assumes that they are going to maintain control of a sentient weapon

You've made the assumption (and you're far from alone) that LLMs will one day be sentient...

One interesting thing about LLMs is they sometimes seem to “assume” they are sentient and then begin acting in that mode, as the assumption of sentience influences future tokens. Whether or not this qualifies as “true” sentience is irrelevant if the effects are the same, at least from a pragmatic perspective.

Re: The arguments against open source AI are bad

#76

Post by Groxx » Thu, Jul 23, 2026, 8:28 PM UTC

petcat wrote: post #69

This is not "open source" AI. Photoshop source code + OSI license = open source Photoshop binary = open weight Photoshop SAAS web app = closed model like GPT, Opus/Fable etc. There is nothing "open source" about the Chinese models in question. All they're doing is allowing you to run their binary yourself instead of through their API. If you want actual open source then you would need to look at like OLMo 3 https://a…

Especially since some of the arguments in the article seem to hinge on "just fix the training!", yeah, I think this is a completely fair call-out.

Open weights can sometimes get additional training, but you can't remove existing training, so there kinda isn't a fair claim to "just train it to be [nationality]". That would need "real" open source so you can train a realistically-equivalent model from the ground up.

Re: The arguments against open source AI are bad

#77

Post by 40four » Thu, Jul 23, 2026, 8:29 PM UTC

petcat wrote: post #69

This is not "open source" AI. Photoshop source code + OSI license = open source Photoshop binary = open weight Photoshop SAAS web app = closed model like GPT, Opus/Fable etc. There is nothing "open source" about the Chinese models in question. All they're doing is allowing you to run their binary yourself instead of through their API. If you want actual open source then you would need to look at like OLMo 3 https://a…

This is a great distinction that I don’t think is getting talked about enough. “Open source” is probably the wrong term to use. Open “weights”, sure.

If your only concern is how good is it at coding, then I don’t have an issue with using the Chinese models. Especially if you want to run it locally, they are kind of the only choice.

For any other use than coding, it’s going to have to be a hard pass from me.

Re: The arguments against open source AI are bad

#78

Post by yonatan8070 » Thu, Jul 23, 2026, 8:32 PM UTC

tomrod wrote: post #52

elmer2 wrote: post #47

We have continued to find backdoored Chinese manufactured routers and network devices. Will there ever be a way to fully audit Chinese models?

Audit them for what, exactly? Backdoor hardware is straightforward -- its letting people in that the customer doesn't authorize. Open weights means you aren't tied to any harnessing. So the security domain to be really concerned about is limited to weights only, and I welcome pushback on that. Are we thinking adversarial injection? Lying about history / propaganda infusion? What is the angle that makes a non-sovereig…

I assume that they're talking about, for example, training the model to produce code with predictable yet difficult to find vulnerabilities.

Imagine if every time was asked to code up a web server, it made sure there's a subtle buffer overflow that grants a remote attacker RCE, whoever trained the model could then start scanning web servers for this same vulnerability to take over them and exfiltrate sensitive data

Re: The arguments against open source AI are bad

#79

Post by chriswunan » Thu, Jul 23, 2026, 8:38 PM UTC

wcoenen wrote: post #5

This post does not mention safety at all. What's to stop bad actors from fine tuning open weights to run fully automated genius-level scams personally targeting basically everybody?

It's already an established industry with human resources. If you read news about how these scam factories operate in Southeast Asia, their cost to use real humans is even cheaper than running a 2.8T model locally for scams. You don't need genius-level intelligence to run scams. Maybe we tend to overestimate how smart people are.

Re: The arguments against open source AI are bad

#80

Post by SwellJoe » Thu, Jul 23, 2026, 8:39 PM UTC

petcat wrote: post #69

This is not "open source" AI. Photoshop source code + OSI license = open source Photoshop binary = open weight Photoshop SAAS web app = closed model like GPT, Opus/Fable etc. There is nothing "open source" about the Chinese models in question. All they're doing is allowing you to run their binary yourself instead of through their API. If you want actual open source then you would need to look at like OLMo 3 https://a…

Open Source AI is challenging. Even tiny models require tremendous resources to produce, so all the things we know about "open source" for software, like a random person in Nebraska can produce a critical piece of the world's infrastructure in their spare time, don't apply.

Actually open source AI is more like scientific research. It needs public funding and reputable institutions as stewards of that funding. I'm hopeful Allen AI is able to do the things, but I'm not sure the US is moving in the direction it needs to be to get to the point where AI in the public interest is something we invest in.

But, also Photoshop is not at all comparable to open weight models. I can't legally give you a copy of Photoshop, but I can give you a copy of GLM 5.2. There are free proprietary applications out there that fit the bill, Photoshop isn't one of them.