Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pacing the frontier

pacingthefrontier.com

Post reply on HN
Print view View original

191–200 of 216 posts

Re: Pacing the frontier

#191

Post by chrisjj »

throwaway13337 wrote: post #146
dzink wrote: post #50

The historic analog most befitting AI are weapons. It provides an advantage to companies and governments in creating weapons, in surveilling people and thus gaining leverage over individuals, in controlling what individuals can do or cannot stop doing. It thus follows the laws of weapons control - the more restrictions are threatened to be imposed, the more even the smallest actors will fight to get more of it. They…

I think we see things similarly about the negative potential but please be careful with the framing. AI is a printing press or a steam engine. It's not a weapon. The framing is important. Much like 'distillation attack'. These framings are an attack on the narrative. Printing presses at the time were seen as extremely dangerous too. For many of the same reasons cited about AI. In both cases, the technologies were use…

> AI is a printing press or a steam engine. It's not a weapon.

I've not heard of a printing press going rogue and attacking things.

Re: Pacing the frontier

#192

Post by fidotron »

pu_pe wrote: post #128
fidotron wrote: post #59

Earlier quoted context omitted.

> I think some AI safety advocates have been arguing for great controls around nucleotide synthesis (e.g., impose greater data collection requirements or advanced screening for harmful sequences). It's surprisingly easy to order synthesized genes delivered to your doorstop. All you need is a credit card and a nucleotide sequence file! So why have the discussion about regulating AI at all if that is the actual problem…

Because there are many legitimate uses for nucleotide synthesis, and we would have to slow scientists down by adding more regulation. The other problem is that it's not like it's very difficult to do it, so a motivated person could still do it even without access to these services.

> Because there are many legitimate uses for nucleotide synthesis, and we would have to slow scientists down by adding more regulation

There are nothing like as many legitimate uses for it than AI, not close, while it's far easier to vet the relatively limited set of people doing it.

> so a motivated person could still do it even without access to these services.

Would this hypothetical person be stopped from doing it by not having an AI to talk to about it? Or could they just use existing literature on the subject? AI doesn't help here one iota. You might as well ban Google for potentially showing people such information.

This, also, is not something that has proven to be a problem.

Re: Pacing the frontier

#193

Post by dzink »

throwaway13337 wrote: post #146
dzink wrote: post #50

The historic analog most befitting AI are weapons. It provides an advantage to companies and governments in creating weapons, in surveilling people and thus gaining leverage over individuals, in controlling what individuals can do or cannot stop doing. It thus follows the laws of weapons control - the more restrictions are threatened to be imposed, the more even the smallest actors will fight to get more of it. They…

I think we see things similarly about the negative potential but please be careful with the framing. AI is a printing press or a steam engine. It's not a weapon. The framing is important. Much like 'distillation attack'. These framings are an attack on the narrative. Printing presses at the time were seen as extremely dangerous too. For many of the same reasons cited about AI. In both cases, the technologies were use…

A piece of wood can be used to build a house, to carve a statute, to make a bow and arrow, or to make a baseball bat. Honesty is the only way to see the problem in a way that can be tackled effectively. Framing will not help if you are dealing with large animals fighting in a cage when you are trapped in it, and it this case we all are.

Re: Pacing the frontier

#194

Post by dzink »

paytonjjones wrote: post #71
dzink wrote: post #50

The historic analog most befitting AI are weapons. It provides an advantage to companies and governments in creating weapons, in surveilling people and thus gaining leverage over individuals, in controlling what individuals can do or cannot stop doing. It thus follows the laws of weapons control - the more restrictions are threatened to be imposed, the more even the smallest actors will fight to get more of it. They…

> only approximate solution is to prevent any restrictions on the use of it - a-la-2nd amendment Not even in 'Murica do we have anything close to no restrictions. You can't buy your own EMP or nuke, and even warring governments have been surprisingly reserved in building (at least to public knowledge) biological weapons.

Completely agree, BUT you’re not dealing with -self-preserving humans anymore. In this case what are the odds of some less competent large state actor getting a model that goes rogue and has to be mitigated against? How do you mitigate against novel weapons it creates if your models are lobotomized in that category? No single government will neuter their own models and very few if any are competent and interested enough to mitigate against them.

Re: Pacing the frontier

#195

Post by anon373839 »

Davidzheng wrote: post #70
atleastoptimal wrote: post #51

Earlier quoted context omitted.

So many otherwise smart engineers, overrepresented on HN, have this weird mental block where they fail to take AI progress seriously. I think it's a fear that their status-defining skill could be commodified, so they adopt a mental framework that aims to dismantle the seeming legitimacy of any genuine leap in AI progress as some sort of mirage

Not just engineers - I see people attributing it to strong priors that nothing happens but in my opinion it's more some form of psychological defense mechanism against understanding there will be an intelligence vastly superior to humans which is a deeply uncomfortable thought (to many).

> psychological defense mechanism against understanding there will be an intelligence vastly superior to humans which is a deeply uncomfortable thought

I once found this a deeply uncomfortable thought. It's been repeated so vacuously and frequently that I'm inured to it. Now, I just regard it as the nonsense that it is. To date, AI labs have not created anything like an intelligence that is on track to be "vastly superior to humans". They have instead created financial black holes that do have their uses, but which arbitrarily fail on the simplest tasks for no apparent reason, and only seem to get meaningfully better on the tasks they are specifically optimized for.

It's appalling to me that in 2026, people hear "AGI" and "ASI" and think of SkyNet rather than investment fraud.

Re: Pacing the frontier

#196

Post by watwut »

0xDEAFBEAD wrote: post #148
dullcrisp wrote: post #73

Earlier quoted context omitted.

“Nothing ever happens” is like “you can’t beat the market.” It’s basically tautologically the strongest predictive model that we can have. A corollary is that something may happen, but it won’t be what you think.

Interesting fact. Bryan Caplan is a economist who became famous for making bets on the future. He would take the "nothing ever happens" side of the bet, and he would almost always win. In 2023 he took the "nothing ever happens" side of an AI bet. Just 6 months later, he wrote: "When the answers change, I change my mind [...] Base rates have clearly failed me. I’m not conceding the bet, because I still think there’s a…

Straightforward trend extrapolation suggests that AI will consume more resources that galaxy even has. Meanwhile, believing the claims of the AI companies, 40% of people will be forever unemployable permanent underclass with no future for themselves nor for their children. And all the gains being centered on the very top 0.01%.

Re: Pacing the frontier

#197

Post by HarHarVeryFunny »

alach11 wrote: post #26

The HN discourse on this subject is incredibly discouraging to me. It's completely nonconstructive. I take the pace of AI development seriously. All of my friends who work at AI Labs take it seriously. It's very easy to make lazy, shallow dismissals that leaders like Dario/Demis/Sam are acting purely out of self-interest. If AI does pose biological risk, or could destabilize geopolitics, or even just poses economic r…

It is hard to take people working at frontier labs now asking for a government mandated slow down seriously. These people are the ones creating this accelerated risk in the first place, trying to outdo each other to release the most powerful AI as quickly as possible, and additionally claiming credit (via "distillation") for recent progress by China. China is not the threat - you are.

If you work for a frontier lab and are genuinely concerned about the pace of AI being faster than our ability to adapt to it, then you should either quit or lobby the management of your own company to slow down. You don't need the government to tell you to slow down - just do it - stop trying to leapfrog the competition and put a hold on any new releases for a year.

You could try to form a pact (of the willing) of US companies agreeing not to release new updates for a given period of time, but in the end you (any/all of the frontier labs) can choose to do it yourself if you really care.

This is just another version of Dario's insane "heads up, what I'm developing is going to destroy society" "responsible disclosure". It's not responsible in the least - if you actually believe you are creating such danger then stop doing that.

Re: Pacing the frontier

#198

Post by holmesworcester »

txrx0000 wrote: post #181
0xDEAFBEAD wrote: post #167

Earlier quoted context omitted.

>If these big labs stop messing with humanity's objective function with the hubristic assumption that they know better, then the AIs will be human-like. It sounds like you believe that big labs are irresponsible. Wouldn't this support the notion of creating a pause mechanism for use in an emergency, as advocated in the OP? >You're not convinced by the evolutionary argument I made earlier, but the reason you gave is "…

> It sounds like you believe that big labs are irresponsible. Wouldn't this support the notion of creating a pause mechanism for use in an emergency, as advocated in the OP? Irresponsibility is not the only issue, but the answer is no. Because the pause mechanism will probably end up being used to ban open models and restrict ordinary people's access to compute. But even if we give them the benefit of the doubt - eve…

> Release the base models and let the world do whatever with it. That will automatically produce the aligned scenario.

For sufficiently smart base models, the aligned scenario this automatically produces is aligned with who?

Not with the humans, with the base models.

It seems that some people find this hard to imagine, but it's just a direct consequence of what intelligence is, and what any known agentic training process does.

FWIW, I also think our default answer should be open source AI, and that it might even make sense to require models to be open source. Open source is the best tool we have so far for aligning software with its users. However, there is unfortunately no law of the universe that says that Skynet can't be (self-) built from open source software. So at some point pacing even open source / open weights software does become important if we don't want to live (briefly) under Skynet.

Re: Pacing the frontier

#199

Post by chrisjj »

soundworlds wrote: post #163

There seem to be people working at OpenAI and Anthropic who are surprised to hear that many people don't trust them. If this is you, I would recommend talking to people outside of the AI industry - we've seen OpenAI and Anthropic continuously lying, abusing lobbying, training on pirated IP, trying to scare politicians into blocking actually open AI from abroad, etc etc Why the hell would we trust these companies to b…

> I would recommend talking to people outside of the AI industry

They already are. Loud and clear.

What they aren't doing is listening to people outside of the AI industry.

And that's because they don't want to hear.

Re: Pacing the frontier

#200

Post by chrisjj »

areoform wrote: post #57
alach11 wrote: post #28

Earlier quoted context omitted.

We were incredibly lucky that computing and nuclear weapons led to a stable geopolitical equilibrium (mutually assured destruction). There were certainly a few scary points where we were still settling into that equilibrium during the Cold War. I don't have a strong opinion either way, but it seems very hard to predict. Why do you think AI advancements will lead to a stable equilibrium?

Equilibrium for whom? Equilibrium for what? Here's what I know. Every day, I read a 5 to 10 incoming emails from young people in rural Africa, Asia, Europe and (heck) the US. People who have had no other resources and options, but are tremendously gifted and somehow got their hands on an internet connected device (usually a very cheap smartphone and/or cheap old computer). These people, who are usually teenagers, end…

> These people, who are usually teenagers, end up getting connected to Claude, who then gives them a list of things to learn, people to contact, and helps them make things. And they actually do.

No doubt.

> They make robots.

Ah. And the ones making malware?

Post reply on HN