Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The arguments against open source AI are bad

tombedor.dev

Post reply on HN
Print view View original

111–120 of 228 posts

Re: The arguments against open source AI are bad

#111

Post by SwellJoe »

parl_match wrote: post #99
SwellJoe wrote: post #80

Earlier quoted context omitted.

Open Source AI is challenging. Even tiny models require tremendous resources to produce, so all the things we know about "open source" for software, like a random person in Nebraska can produce a critical piece of the world's infrastructure in their spare time, don't apply. Actually open source AI is more like scientific research. It needs public funding and reputable institutions as stewards of that funding. I'm hop…

> Open Source AI is challenging. Even tiny models require tremendous resources to produce, so all the things we know about "open source" for software, like a random person in Nebraska can produce a critical piece of the world's infrastructure in their spare time, don't apply. No, but a small university team with dark time on the state school system's cluster can build a basic, functional LLM. It will be a few years (…

"No, but a small university team with dark time on the state school system's cluster can build a basic, functional LLM."

Hey, remember when we were just talking about how it "needs public funding and reputable institutions as stewards of that funding"? (Hint: A state school system is a reputable institution running on public funding.)

Re: The arguments against open source AI are bad

#112

Post by kerblang »

I would be more inclined to agree if American companies were not in a competition to be the biggest flameout and lose the most money. How is China able to afford to train models that US companies need to burn billions of dollars for?

In general, if American capitalism weren't so broken I would advocate for it as the strongest option, but American corporations are dead set on destroying it. The US may end up as "democratic socialist" if there is no capitalist reform.

Re: The arguments against open source AI are bad

#113

Post by Avicebron »

kerblang wrote: post #112

I would be more inclined to agree if American companies were not in a competition to be the biggest flameout and lose the most money. How is China able to afford to train models that US companies need to burn billions of dollars for? In general, if American capitalism weren't so broken I would advocate for it as the strongest option, but American corporations are dead set on destroying it. The US may end up as "democ…

[dead]

Re: The arguments against open source AI are bad

#114

Post by samrus »

ekidd wrote: post #49
TSiege wrote: post #31

Earlier quoted context omitted.

If AI were to become a super weapon why should I trust a private company to own it? If the super weapon is publicly available to download then why should American citizens be banned from doing so?

> If AI were to become a super weapon why should I trust a private company to own it? We have just recently established that: 1. OpenAI's internal "Galaxy" model is fully capable of functioning as what security people refer to as an "Advanced Persistent Threat." The published details of the recent sandbox escape and Hugging Face attack involved chaining multiple unknown zero-days at various stages of the attack, and…

> OpenAI's internal "Galaxy" model is fully capable of functioning as what security people refer to as an "Advanced Persistent Threat

This is marketing. They saw anthropic create crazy hype around (the admittedly great) fable/mythos and they want to replicate that.

Im sure the model is capable of chaining zero days together to hack things, and thats something to address, but i have zero belief that they didnt have it do that intentionally so they could pretend it went rogue. These things dont have initiative, drive, or motivation outside of what we give them from the RLHF. OpenAI deliberately alligned or even prompted it to do just that and are now pretending its emergent

Re: The arguments against open source AI are bad

#115

Post by samrus »

CodingJeebus wrote: post #45
nater5000 wrote: post #15

>Much of the angst around China's models centers on "losing the AI race". But what's the goal of this race? Is it to develop the best model? To sell the most tokens? To destroy humanity first? Some people would say it is reaching some sort of singularity. Even if you don't buy into a more sci-fi interpretation of this, there are pretty grounded arguments one could make that there is some sort of "goal" in AI developm…

The OpenAI/Huggingface scenario proves the necessity of open models. Huggingface, in their postmortem, explained how they quickly realized that the threat was an advanced AI and proceeded to use frontier models to analyze the suspect telemetry, but were unable to due to frontier model safeguards. So they fired up GLM instead, which had no such safeguards and proceeded to analyze their telemetry just fine and get to r…

That is so true. These models need to be democratized. The chinese are doing that because its beneficial for them in second place. The americans are trying to stop it because its detrimental for them in first place. Simple incentive allignment

The critical thing is that the american elites have screwed over normal americans so much that the public isnt alligned with them. The public doesnt feel like its in first place, because it feels like its (and is being told it will be) an underclass. So normal americans feel in the same second place as the chinese and are inadvertently alligning with them. Its crazy how badly the american elites managed to fuck this up

Re: The arguments against open source AI are bad

#116

Post by samrus »

elmer2 wrote: post #47

We have continued to find backdoored Chinese manufactured routers and network devices. Will there ever be a way to fully audit Chinese models?

You have the wieghts for the chinese models. Go ahead and run whatever tests you want.

The american models on the other hand are too powerful for simpletons like us to be allowed to use, but we can be assured its all in our best interest, citizen

Re: The arguments against open source AI are bad

#118

Post by samrus »

yonatan8070 wrote: post #78
tomrod wrote: post #52

Earlier quoted context omitted.

Audit them for what, exactly? Backdoor hardware is straightforward -- its letting people in that the customer doesn't authorize. Open weights means you aren't tied to any harnessing. So the security domain to be really concerned about is limited to weights only, and I welcome pushback on that. Are we thinking adversarial injection? Lying about history / propaganda infusion? What is the angle that makes a non-sovereig…

I assume that they're talking about, for example, training the model to produce code with predictable yet difficult to find vulnerabilities. Imagine if every time was asked to code up a web server, it made sure there's a subtle buffer overflow that grants a remote attacker RCE, whoever trained the model could then start scanning web servers for this same vulnerability to take over them and exfiltrate sensitive data

Model allignment isnt a science, its a craft. And its very very imprecise. Getting anything that subtle through would be impossible without making it obvious

And if your still concerned, go ahead and have a non chinese model review the code, make that part of the harness. The beautiful thing is that your free to do that because its open weight, you can run it however you want

Re: The arguments against open source AI are bad

#120

Post by entropicdrifter »

iamnothere wrote: post #75
bigfishrunning wrote: post #74

Earlier quoted context omitted.

You've made the assumption (and you're far from alone) that LLMs will one day be sentient...

One interesting thing about LLMs is they sometimes seem to “assume” they are sentient and then begin acting in that mode, as the assumption of sentience influences future tokens. Whether or not this qualifies as “true” sentience is irrelevant if the effects are the same, at least from a pragmatic perspective.

It will be interesting to see if philosphical zombies exist. I guess we should know for certain in the next couple of years.
Post reply on HN