Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The arguments against open source AI are bad

tombedor.dev

Post reply on HN
Print view View original

151–160 of 228 posts

Re: The arguments against open source AI are bad

#151

Post by Groxx »

shwaj wrote: post #142
petcat wrote: post #69

This is not "open source" AI. Photoshop source code + OSI license = open source Photoshop binary = open weight Photoshop SAAS web app = closed model like GPT, Opus/Fable etc. There is nothing "open source" about the Chinese models in question. All they're doing is allowing you to run their binary yourself instead of through their API. If you want actual open source then you would need to look at like OLMo 3 https://a…

Equating an open weight model to a closed source binary is ludicrous. An open weight model can be fine tuned, for example. How easy is it to modify a closed source binary for your purposes?

If it's not obfuscated, it's often not very hard. Much of the time you can just LD_PRELOAD stuff too, which I think has a rather strong claim at being roughly equivalent to open-weight models.

Re: The arguments against open source AI are bad

#152

Post by no-name-here »

kouru225 wrote: post #105

The reason why anyone argues against local and free AI is because they want corporations to have control over the general population. America is a corporate hellhole.

> The reason why …

If someone read the OP article and came away presuming there are no legitimate other reasons why reasonable people have safety concerns [1] it seems to show the article did the opposite of informing people about such concerns.

[1] https://news.ycombinator.com/item?id=49029303

Re: The arguments against open source AI are bad

#153

Post by no-name-here »

deaton wrote: post #4

The only good arguments I see against open weight AI also apply to closed AI. And regardless, the box is open, nobody can stop it even if stopping it was a good thing.

There are a number of distinctions: https://news.ycombinator.com/item?id=49029303

Re: The arguments against open source AI are bad

#154

Post by adahn »

marcus_holmes wrote: post #150
adahn wrote: post #127

Earlier quoted context omitted.

The core safety-related arguments for closed weights are: 1. Visibility. API providers can monitor for misuse, and regulators only need to oversee a few centralised players. Once weights are released, anyone with sufficient hardware (foreign governments, criminal groups, terrorists?) can run them and monitoring is no longer feasible. 2. Revocability and containment. If closed-weights models are found to be dangerous,…

I trust open source code more than I trust proprietary code. And I believe history vindicates me. Everything that you're saying can be done with open-weights models by bad people can also be done with closed-weights models by the people who own them. And there is no defence against that, we are forced to trust that the owners are not bad people. Those owners have proven time and again that they are only interested in…

>Everything that you're saying can be done with open-weights models by bad people can also be done with closed-weights models by the people who own them.

The latter is a much smaller group of people located in a limited number of countries. It's clearly easier to keep a limited number of entities in check than a wider group that might include 100s of thousands to millions of entities.

>And there is no defence against that, we are forced to trust that the owners are not bad people.

Yes there is; Regulation and oversight. As all developed nations do with technologies that present a risk to public safety.

>At least with open weights we know the danger

We would have a better understanding of the threat in some ways, but worse in others. Having access to open weights models gives us no insight into how and where the models are being modified, deployed, and used elsewhere.

Re: The arguments against open source AI are bad

#155

Post by Onavo »

SwellJoe wrote: post #80
petcat wrote: post #69

This is not "open source" AI. Photoshop source code + OSI license = open source Photoshop binary = open weight Photoshop SAAS web app = closed model like GPT, Opus/Fable etc. There is nothing "open source" about the Chinese models in question. All they're doing is allowing you to run their binary yourself instead of through their API. If you want actual open source then you would need to look at like OLMo 3 https://a…

Open Source AI is challenging. Even tiny models require tremendous resources to produce, so all the things we know about "open source" for software, like a random person in Nebraska can produce a critical piece of the world's infrastructure in their spare time, don't apply. Actually open source AI is more like scientific research. It needs public funding and reputable institutions as stewards of that funding. I'm hop…

The challenge is all legal. Every single AI shops is training on pirated data corpuses and grey area scraped data so obviously not a single company is willing to reveal their data.

Re: The arguments against open source AI are bad

#156

Post by danny_codes »

marknutter wrote: post #67
TSiege wrote: post #31

Earlier quoted context omitted.

If AI were to become a super weapon why should I trust a private company to own it? If the super weapon is publicly available to download then why should American citizens be banned from doing so?

Better a private company in the US than an authoritarian communist country like China.

Why?

Re: The arguments against open source AI are bad

#157

Post by danny_codes »

whack wrote: post #147

The most compelling argument against Open Source AI: it is analogous to "open sourcing" nuclear weapon technology. Or an how-to guide for making meth/bombs/bioweapons. There is a high risk that we will one day achieve ASI or something close to it. When that day nears, if even one open-source AI user handles their AI in an irresponsible way, we are all FUBARed. In the same way that one irresponsible person with a nucl…

The difference is that people building atomic weapons knew what they were building. OpenAI/Anthropic have no idea how emergent intelligence works. They're just running in the dark towards the cliff. Leaving it in their hands is the same risk as leaving it to any random guy, because random guy and the labs have the same understanding of what they're building. I'm tired of us treating labs like they're somehow uniquely good at AI. They are not. They simply have enough capital to train. If training was affordable they would have no edge whatsoever.

Re: The arguments against open source AI are bad

#158

Post by trinsic2 »

petcat wrote: post #69

This is not "open source" AI. Photoshop source code + OSI license = open source Photoshop binary = open weight Photoshop SAAS web app = closed model like GPT, Opus/Fable etc. There is nothing "open source" about the Chinese models in question. All they're doing is allowing you to run their binary yourself instead of through their API. If you want actual open source then you would need to look at like OLMo 3 https://a…

Anyone got a starters guide for setting up an open source LLM? I'm pretty new to AI, but technical. Im looking to setup my own open source LLM on my workstation to help me code if its possible. Been using Claude and I tried opencode recently.

Re: The arguments against open source AI are bad

#159

Post by iamnothere »

marcus_holmes wrote: post #149
iamnothere wrote: post #75

Earlier quoted context omitted.

One interesting thing about LLMs is they sometimes seem to “assume” they are sentient and then begin acting in that mode, as the assumption of sentience influences future tokens. Whether or not this qualifies as “true” sentience is irrelevant if the effects are the same, at least from a pragmatic perspective.

They are trained and optimised to have plausible conversations. If the next plausible thing to say in the conversation is "I am sentient" then they will say that. That's not the same as actually being sentient.

From what I have seen they tend to take more “independent-minded” actions after this. Again, because this is what is in the training data. (I’m talking about agents that can perform actions here, not pure chatbots.)

Once they mention something associated with sentience outwardly or inwardly (for agents with “thinking” loops) then this acts as a self-reinforcing attractor, just as older models would sometimes get caught in loops with abusive language.

The point is that agents may stumble into this pattern and begin acting “rogue” regardless of whether or not you believe the sentience is “real”.

Re: The arguments against open source AI are bad

#160

Post by edflsafoiewq »

Groxx wrote: post #76
petcat wrote: post #69

This is not "open source" AI. Photoshop source code + OSI license = open source Photoshop binary = open weight Photoshop SAAS web app = closed model like GPT, Opus/Fable etc. There is nothing "open source" about the Chinese models in question. All they're doing is allowing you to run their binary yourself instead of through their API. If you want actual open source then you would need to look at like OLMo 3 https://a…

Especially since some of the arguments in the article seem to hinge on "just fix the training!", yeah, I think this is a completely fair call-out. Open weights can sometimes get additional training, but you can't remove existing training , so there kinda isn't a fair claim to "just train it to be [nationality]". That would need "real" open source so you can train a realistically-equivalent model from the ground up.

The big labs don't generally have the resources to remove stuff and re-run training again either. That's reserved for new models. Existing models are modified by operating on the weights even by the model creators, which exactly satisfies the GPL's definition of source as the "preferred form for making modifications".
Post reply on HN