Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We must pace the frontier

darioamodei.com

Post reply on HN
Print view View original

901–910 of 936 posts

Re: We must pace the frontier

#901

Post by sigmarule »

RobertDeNiro wrote: post #866
aubanel wrote: post #777

Earlier quoted context omitted.

You can disagree with Anthropic leadership, but all the points you mentioned can reconcile very well with them thinking in good faith "advanced AI is too dangerous to be left in all hands" Except the "train on everyone else IP" which can be said of all AI companies.

Problem with Anthropic is that they benefited massively from open source and now are completely against it. It’s hypocrisy at its worst. Why do they get to benefit, but not everyone else?

If this is going to factor into my reasoning about the topic at hand, it will surely be a secondary or tertiary consideration at best. Same goes for much of the list in the root comment.

We are discussing the possibility of severe adverse societal/world/human impacts from AI, why should open vs closed source/weights, copyright violations, anti-competitive moves, corporate hypocrisy, etc be so heavily weighted in the discussion? Hell, multiple things in OP’s list are quite literally moves a good-faith (wrt the topic at hand, genuine vs non-genuine concern) actor in such a position would do: closed weights means bad actors can’t abliterate, regulatory control, getting blacklisted by Hegseth’s DOW because they LIMITED acceptable use cases…

I’m not pro- or anti-Anthropic at all, but it has started to annoy me how biased and shaky reasoning gets about this topic on here, it feels like people arguing to fit their existing opinions into today’s hubbubery rather than measured analysis of said hubbubery, which is why I come here.

Question: consider the scenario where RSI is reached internally and internal tests struggle to align or control/contain the model; what are the differences in observables vs what we’re observing today?

Re: We must pace the frontier

#902

Post by usernametaken29 »

RGS1811 wrote: post #521

I don’t understand all the comments assuming that RSI is the real threat here. Dario is admitting that they failed to solve alignment. Without alignment, further improvements in capability turn LLMs into wanton felony generators. This call to pace the frontier is dressed up as altruism but it’s an admission that they cannot produce a marketable product better than what they have. Pacing the frontier means the US labs…

> Without alignment, further improvements in capability turn LLMs into wanton felony generators

Honestly, I don’t think that’s bad at all. I hope OpenAI and Antrophic keep RL training runs up that randomly fuck with a lot of people. Until the day the DOJ comes knocking, locks those idiots up in jail and closes them both down for the insane lack of responsibility and carelessness they’ve shown. Sounds like the IDEAL outcome. Finally some jail time for all the fraud, negligence, outright scamming, hype inflation etc. if anything can accelerate this, oi, be my guest. Amodei might be afraid because he knows if he keeps pulling the stunts for investment theatre, at some point they’ll actually face consequences. AWESOME. That’s what we want right there

Re: We must pace the frontier

#903

Post by sigmarule »

cuuupid wrote: post #457

At what point do we stop engaging with Anthropic’s leadership in good faith and acknowledge their track record, - no open weights - can’t use claude to research AI - train on everyone else’s IP and sell it back to them - 8 regulatory capture attempts and counting - so controlling they are the only US company blacklisted by the US government This is not effective altruism / rationalism gone wild, it’s just monopolisti…

> so controlling they are the only US company blacklisted by the US government

Extremely bad example for the topic at hand.

Re: We must pace the frontier

#904

Post by cute_boi »

Sevii wrote: post #13
heaney-555 wrote: post #11

None of this works without buy-in from China. This isn't something private companies can decide. The US would need to sign a groundbreaking deal with China, equivalent to the Anti-Ballistic Missile Treaty of the Cold War.

We'd have to make a deal with China and be confident they wouldn't cheat on it.

The funny thing is the US have cheated a lot compared to China if we look into history.

Re: We must pace the frontier

#905

Post by tedggh »

usernametaken29 wrote: post #902
RGS1811 wrote: post #521

I don’t understand all the comments assuming that RSI is the real threat here. Dario is admitting that they failed to solve alignment. Without alignment, further improvements in capability turn LLMs into wanton felony generators. This call to pace the frontier is dressed up as altruism but it’s an admission that they cannot produce a marketable product better than what they have. Pacing the frontier means the US labs…

> Without alignment, further improvements in capability turn LLMs into wanton felony generators Honestly, I don’t think that’s bad at all. I hope OpenAI and Antrophic keep RL training runs up that randomly fuck with a lot of people. Until the day the DOJ comes knocking, locks those idiots up in jail and closes them both down for the insane lack of responsibility and carelessness they’ve shown. Sounds like the IDEAL o…

Not a fan of Amodei myself but calling Anthropic a scam is a bit of a stretch when their revenue growth is unprecedented in the history of tech. They are also technically a profitable business.

Re: We must pace the frontier

#906

Post by ElProlactin »

jonas21 wrote: post #891
ElProlactin wrote: post #827

Earlier quoted context omitted.

If they genuinely believe the tech is dangerous, how would it be acting in "good faith" to keep developing it, prepping an IPO, etc.?

The entire essay is about stopping development until it can be made more safe, with specific ideas on how to do that. The fact that he would suggest this despite preparing for an IPO is an even stronger signal that it's in good faith -- it will almost certainly delay or reduce the valuation of the IPO.

> The fact that he would suggest this despite preparing for an IPO is an even stronger signal that it's in good faith -- it will almost certainly delay or reduce the valuation of the IPO.

No it won't.

This is a call for a "safety cartel"[1] in which the dominant firms become more entrenched against competition by colluding to limit the progress would-be competitors can make in the market.

[1] https://x.com/alexwg/status/2098793433275985932

Re: We must pace the frontier

#907

Post by sigmarule »

vlyan wrote: post #806
cuuupid wrote: post #457

At what point do we stop engaging with Anthropic’s leadership in good faith and acknowledge their track record, - no open weights - can’t use claude to research AI - train on everyone else’s IP and sell it back to them - 8 regulatory capture attempts and counting - so controlling they are the only US company blacklisted by the US government This is not effective altruism / rationalism gone wild, it’s just monopolisti…

I can't imagine a grown adult genuinely believing that an American company funded with over $100 billion of venture capital values the best interests of mankind over the best interests of its investors. while not everyone may recognize all that self-serving chutzpah as regulatory capture efforts, I think everyone can tell they're being bullshitted. some just pretend to suspend their disbelief when the blatant lies th…

This thinking feels off for this context.

First, there is a counterexample to your point, or a class of counterexamples: scenarios where the company believes the harm to mankind could propagate to harm their ability to sell to mankind.

Which is interesting, of course, considering the “AI may destroy humanity” arguments this topic is centered on are prototypical members of this category.

To be fair, one could argue this counterexample is not actually one, but rather a degenerate case where the good of the shareholders and humanity collapse and converge - in either case, though, the implication here is the same.

It could certainly be the case they are playing these concerns up for more effective use as tactics in unrelated corporate strategies. But failing to consider the possibility of genuine concern being behind this behavior is childish.

Re: We must pace the frontier

#908

Post by cruffle_duffle »

robomartin wrote: post #447

My position on the clear gaslighting coming out of Anthropic is simple. 1: It is ridiculous on the face of it; it does not pass the physics test. They are actually claiming that AI will kill ALL OF HUMANITY (>10% probability) and have not provided a detailed explanation of exactly how they see that happening. Not one. Publish a paper showing exactly how you kill eight billion people while we all sit around and do not…

It’s the same thing with Covid. People thought 1 in 10 people would be dead. And believed it for years, despite it being orders and orders of magnitude wrong.

Humans love a good end-of-world story. Always did, always will. It’s when they want the world to conform to their ungrounded irrational bat-shit crazy nonsense that I get a worried. Especially when people in power actually take their nonsense seriously.

Re: We must pace the frontier

#909

Post by eknkc »

boshalfoshal wrote: post #803
un_montagnard wrote: post #598

Anthropic is about to go public, but they figured they can't keep making improvement at the same pace they used to. If the frontier is paced, they can continue hyping their unreleased capabilities that coincidentally cannot be released due to things outside their control. Which buy them more time to try to improve the models.

This is just completely false lol. They most certainly can make better models - why cant people here just read this for face value? I think anthropic is legitimately concerned about the safety implications of stronger models. Race dynamics necessitate that they make better and better models, which they clearly think is bad. They have an internal model which could solve a millenium problem end to end with no intervent…

> which they clearly think is bad

These are companies with huge amounts of money invested in them. Their main responsibility is to their investors. If building better models (or making more money from the models they already have) would increase returns, they would do it. So they clearly can’t do it right now.

Climate change hasn’t stopped the oil industry, and concerns about forever chemicals haven’t stopped the chemical industry. Companies don’t usually give up profitable opportunities just because they might be harmful or kill us all in the long term..

Re: We must pace the frontier

#910

Post by hardbass »

8note wrote: post #630
Aurornis wrote: post #442

Earlier quoted context omitted.

> Sure - look at what is being done today with H1B visas and tariffs. I have some bad news for you if you bought into the idea that the tariffs were good for domestic jobs and the economy. > What is more important to you - having a job and a functioning economy/society, or upholding some "free market" doctrine you've bought into? This is the definition of a false dichotomy, and you also ignored (or missed) the point…

free market doctrine is an axiom to your second order effects. its so embedded into all your assumptions that you havent even considered it

Whatever you are calling "free market doctrine" is a physical law. Why would a business owner subject themselves to shitty conditions if they could find workers in better places? Some would of course, but not most or all.
Post reply on HN