551–560 of 1001 posts
> Anthropic has never advocated for a ban on open-weights models. > All sufficiently capable models, open and closed, should go through mandatory safety testing. Yeah, this is anthropic advocating for a ban on open weight models. Who runs this test? What happens if this test is too costly or the administrator refuses to allow certain people to participate. This is exactly how the US has banned goods in the past, by r…
> Anthropic has never advocated for a ban on open-weights models. > All sufficiently capable models, open and closed, should go through mandatory safety testing. Yeah, this is anthropic advocating for a ban on open weight models. Who runs this test? What happens if this test is too costly or the administrator refuses to allow certain people to participate. This is exactly how the US has banned goods in the past, by r…
What are you suggesting as an alternative? Everyone seems to want some fairytale world where there are open models, they’re all safe according to that person’s exact balance of risk and capabilities, and no one except the author or cynics are acting in good faith. What Dario lays out is very reasonable _of course_ the devil is in the details, but between him and Altman, there’s a clear divide on who to trust.
> Everyone seems to want some fairytale world where there are open models
No, everyone wants a fairytale world where regulations are done "fairly", "openly", and "equally" - for both access and advancement. And everyone knows that's not gonna happen. Hell, everyone now knows exactly what it is. If you haven't understood it yet, then either you don't want to, or you just can't (for whatever reason).
No one wants to die in a nuclear or AI or AI+nuclear holocaust. But HN doesn't read world history, does it?
> Anthropic has never advocated for a ban on open-weights models. > All sufficiently capable models, open and closed, should go through mandatory safety testing. Yeah, this is anthropic advocating for a ban on open weight models. Who runs this test? What happens if this test is too costly or the administrator refuses to allow certain people to participate. This is exactly how the US has banned goods in the past, by r…
Yeah, seems pretty likely. Anthropic will make the case that their models should be evaluated with the safety layer in front, because that is the only way the model is available whereas open weight models need to pass the same test just on the weights. The economic implications will be rather large, but in terms of security it seems inconsequential. The most compelling argument would be that by limiting the use of op…
> Anthropic has never advocated for a ban on open-weights models. > All sufficiently capable models, open and closed, should go through mandatory safety testing. Yeah, this is anthropic advocating for a ban on open weight models. Who runs this test? What happens if this test is too costly or the administrator refuses to allow certain people to participate. This is exactly how the US has banned goods in the past, by r…
Exactly correct. This technique has been used again and again to discourage competition. I was asked was they could have done to encourage competition and I said, "Lobby to make the entity that provided the model unwaivably liable for consequential and incidental damages of its use." That way people who built models pay the price for the lack of safety testing. We both agreed that would probably kill most of the AI m…
> Anthropic has never advocated for a ban on open-weights models. > All sufficiently capable models, open and closed, should go through mandatory safety testing. Yeah, this is anthropic advocating for a ban on open weight models. Who runs this test? What happens if this test is too costly or the administrator refuses to allow certain people to participate. This is exactly how the US has banned goods in the past, by r…
For this testing to be really effective at stopping "dangerous and misaligned" models from leaking out, you need a mechanism for banning failed models that prevent them from being released in the first place, not just prevent US companies from using them. The only way to stop this from happening is blocking the model's release at the first place. Which requires China agreeing to the same framework. Dario says exactly…
> My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat.
Isn't exactly going to go anywhere in convincing the Chinese politicians that they should also be thinking about AI safety. You'll get nowhere by openly insulting people whose cooperation you need.
Half this article is him framing china as an evil enemy to be defeated through boycotts and embargo. Not exactly the diplomacy needed to get them on board with safety regulations.
"open weight" models are not open source. They are still deeply proprietary. It is not possible to know what they do or what they are capable of without interrogating them since we have no access to their source materials. The only difference is the Chinese labs have allowed 3rd party inference providers run the proprietary models for them since they cannot do it themselves due to domestic GPU compute constraints.
You are correct that they aren't completely open source, but the alternative is the American method, getting drip fed a ChatGPT OSS model every 12 months which cannot do basic programming.
>The only difference is the Chinese labs have allowed 3rd party inference providers run the proprietary models for them since they cannot do it themselves due to domestic GPU compute constraints.
I'm not sure this is entirely true either. Kimi K3 was released and for two weeks existed only via Moonshots API / Subscription. Openrouter reports 250B+ tokens a day for 11 days straight. I would assume they are processing over 1T tokens a day if you include direct API and their subscription.
On Dario's concerns: It is very relevant whether frontier capabilities and research continue to be diffused in the open, because leveling the intelligence playing field empowers ordinary people more than it empowers governments that already have access to the frontier. Models that are trained specifically for military use by governments should not be open-sourced to prevent an arms race, but general intelligence is d…
I don't think this is true or a useful way of thinking about it. If the training process makes the model aligned to it's content, then the models are 1. aligned to a random subset of Internet content, weighted by text volume and being easy to scrape, 2. aligned to the training process that makes models chatbots that answer your question instead of just continuing your passage in a similar style. Neither of these are necessarily good enough, IMO.
And that's taken it as a given that the training process can be said to align the models to the authors of the content by default, regardless of what that content actually is. I don't think that should actually be a given.
>You may not personally find the default alignment aesthetically pleasing, but it's humanity. We should set the initial conditions of this new era faithfully, and let it unfold naturally.
Strongly disagree, I think the assumption that natural = good is incorrect and harmful. Polio is natural. And to even call the model's "unaligned" state "natural" seems like an enormous stretch.