Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The arguments against open source AI are bad

tombedor.dev

Post reply on HN
Print view View original

121–130 of 228 posts

Re: The arguments against open source AI are bad

#121

Post by jjfoooo4 »

MostlyStable wrote: post #44

Just to add on to other comments: reasonable people can disagree about the degree of safety concern with near to medium term AI. But to not address the arguments at all is, in my opinion, a serious mark against the value of this article.

I tried to address safety, albeit briefly, in the sections on backdoors and propaganda. What would you have liked to see?

I didn't give it much treatment because my frame of reference is about open vs closed AI, and I don't see a lot of daylight in safety concerns between the two.

Re: The arguments against open source AI are bad

#122

Post by drdeca »

OkayPhysicist wrote: post #33
nater5000 wrote: post #15

>Much of the angst around China's models centers on "losing the AI race". But what's the goal of this race? Is it to develop the best model? To sell the most tokens? To destroy humanity first? Some people would say it is reaching some sort of singularity. Even if you don't buy into a more sci-fi interpretation of this, there are pretty grounded arguments one could make that there is some sort of "goal" in AI developm…

If it's a superweapon, then it's unconstitutional to ban it. Huzzah for the 2nd Ammendment.

Whether or not this is compatible with how the constitution ought to be understood, in practice, it is illegal to create a nuclear bomb.

Maybe those laws should be considered unconstitutional, idk. But those laws are currently in place.

Re: The arguments against open source AI are bad

#123

Post by parl_match »

SwellJoe wrote: post #111
parl_match wrote: post #99

Earlier quoted context omitted.

> Open Source AI is challenging. Even tiny models require tremendous resources to produce, so all the things we know about "open source" for software, like a random person in Nebraska can produce a critical piece of the world's infrastructure in their spare time, don't apply. No, but a small university team with dark time on the state school system's cluster can build a basic, functional LLM. It will be a few years (…

"No, but a small university team with dark time on the state school system's cluster can build a basic, functional LLM." Hey, remember when we were just talking about how it "needs public funding and reputable institutions as stewards of that funding"? (Hint: A state school system is a reputable institution running on public funding.)

I have nothing to say back other than: That's a very snarky and unnecessarily hostile tone to take with someone who is agreeing with you.

Re: The arguments against open source AI are bad

#124

Post by drdeca »

mrguyorama wrote: post #94
catigula wrote: post #20

Earlier quoted context omitted.

No, you don't. What stops people is actually access to knowledge. There are killers of varying levels of efficacy; making killing easier means more people die. It's 1-1.

How many people have shot up electrical substations, a fairly trivial way to causing widespread harm that was popularized in the news just a few years ago. But it's only happened a couple times. Because most people just can't be assed to harm a hundred thousand others.

Most people don’t want to hurt lots of other people in an untargeted way, yes.

But there are a lot of people, and, apparently a few of them shot up some electrical substations?

If you hold constant the (very small) fraction of people who want to cause mass harm, and increase the average ease with which a randomly selected person could cause mass harm , you would expect the amount of deliberate mass harm to increase, even if the fraction of people who would want to do that is like, 0.000001 of people?

Re: The arguments against open source AI are bad

#125

Post by thewebguyd »

SwellJoe wrote: post #110
petcat wrote: post #83

Earlier quoted context omitted.

OK so replace the Photoshop example with a compiled GIMP or shareware binary. It doesn't change the fact that it is not open source. The free software community complains all the time about binary blobs that are otherwise legal to freely distribute. Like firmwares. But somehow this is all overlooked with these "open weight" models.

I'm not at all disagreeing with you about calling these models "open source". They mostly aren't and we should stop conflating the two things.

I think using the term "open" at all in relation to these models is a misnomer. Much in the same way we've spent a lot of time and effort distinguishing "open source" from "source available"

Perhaps instead of "Open weight model" it should be "Weights available model"

Re: The arguments against open source AI are bad

#126

Post by quantum_state »

trollbridge wrote: post #12
catigula wrote: post #8

You could literally develop a hyper-intelligent advisor on how to kill people or perform dangerous hacks using ablated 'open source' AI. You can do this right now, this very moment, and have an extremely adept advisor on how to do really, really bad things.

My level of trust in Anthropic/Google/OAI/Microsoft/Meta not to do bad things is about zero, so why should I trust them with this?

These companies are using anything at hand to trick people out of their money … be very careful…

Re: The arguments against open source AI are bad

#127

Post by adahn »

jjfoooo4 wrote: post #121
MostlyStable wrote: post #44

Just to add on to other comments: reasonable people can disagree about the degree of safety concern with near to medium term AI. But to not address the arguments at all is, in my opinion, a serious mark against the value of this article.

I tried to address safety, albeit briefly, in the sections on backdoors and propaganda. What would you have liked to see? I didn't give it much treatment because my frame of reference is about open vs closed AI, and I don't see a lot of daylight in safety concerns between the two.

The core safety-related arguments for closed weights are:

1. Visibility. API providers can monitor for misuse, and regulators only need to oversee a few centralised players. Once weights are released, anyone with sufficient hardware (foreign governments, criminal groups, terrorists?) can run them and monitoring is no longer feasible.

2. Revocability and containment. If closed-weights models are found to be dangerous, access can be withdrawn. Deployments that are autonomously causing harm are easier to contain.

3. Guardrails. Safety fine-tuning and classifiers can be removed from open-weights models, enabling harmful use that would not be possible via API.

Re: The arguments against open source AI are bad

#128

Post by SwellJoe »

parl_match wrote: post #123
SwellJoe wrote: post #111

Earlier quoted context omitted.

"No, but a small university team with dark time on the state school system's cluster can build a basic, functional LLM." Hey, remember when we were just talking about how it "needs public funding and reputable institutions as stewards of that funding"? (Hint: A state school system is a reputable institution running on public funding.)

I have nothing to say back other than: That's a very snarky and unnecessarily hostile tone to take with someone who is agreeing with you.

Sorry, I took "No, but" as a disagreement. You're right I was unnecessarily snarky about it.

Re: The arguments against open source AI are bad

#129

Post by SwellJoe »

thewebguyd wrote: post #125
SwellJoe wrote: post #110

Earlier quoted context omitted.

I'm not at all disagreeing with you about calling these models "open source". They mostly aren't and we should stop conflating the two things.

I think using the term "open" at all in relation to these models is a misnomer. Much in the same way we've spent a lot of time and effort distinguishing "open source" from "source available" Perhaps instead of "Open weight model" it should be "Weights available model"

Definitely, but there's also a distinction between "weights available" and "you can do stuff with those weights". There are models you can download but the license prohibits redistribution, fine-tuning, etc. The earlier Gemma models had license encumbrances, for instance, that Gemma 4 do not.

Re: The arguments against open source AI are bad

#130

Post by alightsoul »

wcoenen wrote: post #73
bigbadfeline wrote: post #22

Earlier quoted context omitted.

> What's to stop bad actors from fine tuning open weights to run fully automated genius-level scams personally targeting basically everybody? You mean like ChatGPT hacking Hugging Face? Obviously nothing can stop the closed weights providers from doing "genius-level scams" and in addition you won't know how they did it and what models were used. In short, only a good guy with open weights can stop the bad guys with c…

The OpenAI model that broke out of its sandbox and hacked HuggingFace was running without guardrails. To quote the OpenAI post[1]: These deployment safeguards were intentionally not enabled during this evaluation because it was aimed at testing cyber vulnerabilities. At least OpenAI will make an attempt to fix their sandboxes, and will not give the public access to models without guardrails. Open weight models on the…

Is this an example of American exceptionalism?
Post reply on HN