Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gavin Newsom vetoes SB 1047

gov.ca.gov

Post reply on HN
Print view View original

71–80 of 514 posts

Re: Gavin Newsom vetoes SB 1047

#71

Post by hn_throwaway_99 »

Curious if anyone can point to some resources that summarize the pros/cons arguments of this legislation. Reading this article, my first thought is that I definitely agree it sounds impossibly vague for a piece of legislation - "reasonable care" and "unreasonable risk" sound like things that could be endlessly litigated.

At the same time,

> Computer scientists Geoffrey Hinton and Yoshua Bengio, who developed much of the technology on which the current generative-AI wave is based, were outspoken supporters. In addition, 119 current and former employees at the biggest AI companies signed a letter urging its passage.

These are obviously highly intelligent people (though I've definitely learned in my life that intelligence in one area, like AI and science, doesn't mean you should be trusted to give legal advice), so I'm curious to know why Hinton and Bengio supported the legislation so strongly.

Re: Gavin Newsom vetoes SB 1047

#72

Post by handfuloflight »

SonOfLilit wrote: post #55

I wondered if the article was over-dramatizing what risks were covered by the bill, so I read the text: (g) (1) “Critical harm” means any of the following harms caused or materially enabled by a covered model or covered model derivative: (A) The creation or use of a chemical, biological, radiological, or nuclear weapon in a manner that results in mass casualties. (B) Mass casualties or at least five hundred million d…

Does Newsom believe that an AI model can do this damage autonomously or does he understand it must be wielded and overseen by humans to do so?

In that case, how much of an enabler is an AI to meet the destructive ends, when, if the humans can use AI to conduct the damage, they can surely do it without the AI as well.

The potential for destruction exists either way but is the concern that AI makes this more accessible and effective? What's the boogeyman? I don't think these models have private information regarding infrastructure and systems that could be exploited.

Re: Gavin Newsom vetoes SB 1047

#73

Post by worstspotgain »

SonOfLilit wrote: post #66
worstspotgain wrote: post #52

Excellent move by Newsom. We have a very active legislature, but it's been extremely bandwagon-y in recent years. I support much of Wiener's agenda, particularly his housing policy, but this bill was way off the mark. It was basically a torpedo against open models. Market leaders like OpenAI and Anthropic weren't really worried about it, or about open models in general. Its supporters were the also-rans like Musk [1]…

why would Google, Microsoft and OpenAI oppose a torpedo against open models? Aren't they positioned to benefit the most?

If there was just one quasi-monopoly it would have probably supported the bill. As it is, the market leaders have the competition from each other to worry about. Getting rid of open models wouldn't let them raise their prices much.

Re: Gavin Newsom vetoes SB 1047

#74

Post by KoolKat23 »

pkage wrote: post #49
jpk wrote: post #41

Earlier quoted context omitted.

> running them at all is the "high-risk situation" What is the actual, concrete concern here? That a model "breaks out", or something? The risk with AI is not in just running models, the risk is becoming overconfident in them, and then putting them in charge of real-world stuff in a way that allows them to do harm. Hooking a model up to an effector capable of harm is a deliberate act requiring assurance that it doesn…

There is no concrete concern past "models that can simulate thinking are scary." The risk has always been connecting models to systems which are safety critical, but for some reason the discourse around this issue has been more influenced by Terminator than OSHA. As a researcher in the field, I believe there's no risk beyond overconfident automation---and we already have analogous legislation for automations, for exa…

Well it's a mix of concerns, the models are general purpose, there are plenty of areas regulation does not exist or is being bypassed. Can't access a prohibited chemical, no need to worry the model can tell you how to synthesize it from other household chemicals etc.

Re: Gavin Newsom vetoes SB 1047

#75

Post by CSMastermind »

SonOfLilit wrote: post #66
worstspotgain wrote: post #52

Excellent move by Newsom. We have a very active legislature, but it's been extremely bandwagon-y in recent years. I support much of Wiener's agenda, particularly his housing policy, but this bill was way off the mark. It was basically a torpedo against open models. Market leaders like OpenAI and Anthropic weren't really worried about it, or about open models in general. Its supporters were the also-rans like Musk [1]…

why would Google, Microsoft and OpenAI oppose a torpedo against open models? Aren't they positioned to benefit the most?

The bill included language that required the creators of models to have various "safety" features that would severely restrict their development. It required audits and other regulatory hurdles to build the models at all.

Re: Gavin Newsom vetoes SB 1047

#76

Post by nradov »

whimsicalism wrote: post #37
akira2501 wrote: post #33

Earlier quoted context omitted.

> and mandates audit requirements to prove that your models won't help people cause disasters Audits cannot prove anything and they offer no value when planning for the future. They're purely a retrospective tool that offers insights into potential risk factors. > and can be turned off. I really wish legislators would operate inside reality instead of a Star Trek episode.

This snide dismissiveness around “sci-fi” scenarios, while capabilities continue to grow, seems incredibly naïve and foolish. Many of you saying stuff like this were the same naysayers who have been terribly wrong about scaling for the last 6-8 years or people who only started paying attention in the last two years.

That's a total non sequitur. Just because LLMs are scalable doesn't mean this is a problem that requires government intervention. It's only idiots and grifters who want us to worry about sci-fi disaster scenarios. The snide dismissiveness is completely deserved.

Re: Gavin Newsom vetoes SB 1047

#77

Post by tbrownaw »

comp_throw7 wrote: post #43
tbrownaw wrote: post #18

https://legiscan.com/CA/text/SB1047/id/3019694 So this is the one that would make it illegal to provide open weights for models past a certain size, would make it illegal to sell enough compute power to train such a model without first verifying that your customer isn't going to train a model and then ignore this law, and mandates audit requirements to prove that your models won't help people cause disasters and can…

> this is the one that would make it illegal to provide open weights for models past a certain size That's nowhere in the bill, but plenty of people have been confused into thinking this by the bill's opponents.

Three of the four options of what an "artifical intelligence safety incident" is defined as require that the weights be kept secret. One is quite explicit, the others are just impossible to prevent if the weights are available:

> (2) Theft, misappropriation, malicious use, inadvertent release, unauthorized access, or escape of the model weights of a covered model or covered model derivative.

> (3) The critical failure of technical or administrative controls, including controls limiting the ability to modify a covered model or covered model derivative.

> (4) Unauthorized use of a covered model or covered model derivative to cause or materially enable critical harm.

Re: Gavin Newsom vetoes SB 1047

#78

Post by nisten »

Imagine being concerned about AI safety and then introducing a bill that had to be ammended to change criminal responsability of AI developers to civil legal responsability for people who are trying to investigate and work openly on models.

What's next, going after maintainers of python packages... is attacking transparency itself a good way to make AI safer. Yeah, no, it's f*king idiotic.

Re: Gavin Newsom vetoes SB 1047

#79

Post by throwup238 »

hn_throwaway_99 wrote: post #71

Curious if anyone can point to some resources that summarize the pros/cons arguments of this legislation. Reading this article, my first thought is that I definitely agree it sounds impossibly vague for a piece of legislation - "reasonable care" and "unreasonable risk" sound like things that could be endlessly litigated. At the same time, > Computer scientists Geoffrey Hinton and Yoshua Bengio, who developed much of…

California’s Office of Legislative Counsel always provides a “digest” for every bill as part of its full text: https://leginfo.legislature.ca.gov/faces/billTextClient.xhtm...

It’s not an opinionated pros/cons list from the industry but it’s probably the most neutral explanation of what the bill does.

Re: Gavin Newsom vetoes SB 1047

#80

Post by hn_throwaway_99 »

SonOfLilit wrote: post #66
worstspotgain wrote: post #52

Excellent move by Newsom. We have a very active legislature, but it's been extremely bandwagon-y in recent years. I support much of Wiener's agenda, particularly his housing policy, but this bill was way off the mark. It was basically a torpedo against open models. Market leaders like OpenAI and Anthropic weren't really worried about it, or about open models in general. Its supporters were the also-rans like Musk [1]…

why would Google, Microsoft and OpenAI oppose a torpedo against open models? Aren't they positioned to benefit the most?

Yeah, I think the argument that "this just hurts open models" makes no sense given the supporters/detractors of this bill.

The thing that large companies care the most about in the legal realm is certainty. They're obviously going to be a big target of lawsuits regardless, so they want to know that legislation is clear as to the ways they can act - their biggest fear is that you get a good "emotional sob story" in front of a court with a sympathetic jury. It sounded like this legislation was so vague that it would attract a hoard of lawyers looking for a way they can argue these big companies didn't take "reasonable" care.

Post reply on HN