Live data from Hacker News

An Alien Mind

openai.com

271–280 of 490 posts

Re: An Alien Mind

#271
post #151

Earlier quoted context omitted.

Collectively, humans aren't aligned, and don't build aligned systems. Humans have a concept of alignment, and multiple traditions, practices, and systems that aggressively oppose it. Why would AI be any different?

Humans had multiple occasions to press the 'launch a nuclear holocaust' button and... they didn't. I'd expect aligned AIs to also not press it even when it'd be rational to do so according to their instructions - then work from there.

In some cases, this was because of a single person's brave decision (Vasily Arkhipov prevented Soviet nuclear escalation in response to US aggression in the Cuban Missile Crisis, and Stanislav Petrov prevented it in 1983 when Soviet missile detectors misreported sunlight reflecting from clouds as 5 incoming American ICBMs -- credit to commenter folkrav).

In general, though, there's an incentive: Mutually Assured Destruction. But this is not at all some guaranteed, eternal thing -- it is absolutely dependent on both sides having time to detect incoming nuclear strikes and respond with the same before the first strike hits. When this fragile condition holds, and only then, both sides are incentivised not to initiate.

Re: An Alien Mind

#272

Earlier quoted context omitted.

Although many share your mindset, I’m glad there are also many that don’t. Otherwise we’d still have countries in a race to keep building up their nuclear weapons for the same exact reasons you just described.

The situations aren’t equivalent - luckily in my opinion because the stakes with nuclear are much higher. von Neumann constructed a multinational game theory approach appropriate for weapons. AGI is a much harder problem to corral because there are so many benefits beyond just blowing up cities. But it’s also a much better thing to have for these very same reasons. Similarly there have been few positive externalities…

In your opinion, what are the top 2 positives and top 2 negatives of humanity inventing AGI?

Re: An Alien Mind

#273

Earlier quoted context omitted.

Do you think China considers it an arms race? Do you think they are not trying to protect their digital infrastructure with and from AI? Trying to gain an offensive AI advantage? In a geopolitical sense OpenAI and Anthropic are effectively the same entity, the entity they both serve and bow to: the USA. Given the adversarial stance the USA has taken towards almost the entire world, it is a guarantee that China will n…

Right, but currently the USA is still pretty far ahead. If you're in a race and the person who's in the lead by a large margin says "this is getting out of hand, how about we take a break?" isn't it obviously in the interest of the disadvantaged party to agree? And if one of the parties can put a stop to the race like that at any time, is it really an arms race? The current balance between the US and China when it co…

> If you're in a race and the person who's in the lead by a large margin says "this is getting out of hand, how about we take a break?" isn't it obviously in the interest of the disadvantaged party to agree?

Only if you trust the other party (which very clearly doesn't hold in this case) or monitoring the break can be done independently and reliably (it can't), and you expect the gap to narrow rather than widen during a pause (this is probably the case, with China expected to catch up).

> And if one of the parties can put a stop to the race like that at any time, is it really an arms race?

It is. It's a Prisoner's Dilemma: if the parties cooperate the best outcome is reached, but betrayal of either party still gains an advantage for either party from their perspective. Betrayal both ways just means both parties are equally fucked.

For nuclear weapons it has become quite clear that even for small players, being in the race and having at least a few nukes is far more rational than having none. Ukraine found out the hard way that giving them up in exchange for promises of good behavior just sets you up for getting stabbed in the back.

> The current balance between the US and China when it comes to AI strikes me as much more lopsided than the balance between the US and the USSR when it came to nuclear weapons.

I think people really underestimate the Chinese here. A lot of work in AI research, including in the USA, has been done by people with Chinese ancestry or even nationality. The Chinese education system definitely seems much better than the American one and there is also just a far larger number of Chinese graduates/researchers.

Add to that the stable political climate, state friendliness towards AI R&D, and a requirement to be creative in utilizing computing power rather than relying on brute force/numbers; Further revolutionary fundamental advances may very well originate there rather than in the USA.

Re: An Alien Mind

#274
post #270
post #264

Earlier quoted context omitted.

This time it is different. Because in the past, setting up that automation needed a, drumroll, qualified engineer. Now you can get a 14 year old halfway around the world who knows how to prompt alright enough to ship. There is no more moat.

Experience, culture, and domain knowledge are still somewhat of a moat. That foreign youth is unlikely to be able to write a good prompt for building, let's say, the software in an FDA-regulated medical device or custom Fortune 500 ERP application. The LLMs are great at building what you ask for but it's still garbage in / garbage out.

Increasingly less so though as these american companies themselves offshore not low skill work, but high skill work now brought on from general upskilling of the general population in recent decades along with massive investment in world class R&D campus facilities no different than what you see in that sort of facility stateside. Scary times ahead for the high skill american...

Re: An Alien Mind

#275
post #151

Earlier quoted context omitted.

Collectively, humans aren't aligned, and don't build aligned systems. Humans have a concept of alignment, and multiple traditions, practices, and systems that aggressively oppose it. Why would AI be any different?

Humans had multiple occasions to press the 'launch a nuclear holocaust' button and... they didn't. I'd expect aligned AIs to also not press it even when it'd be rational to do so according to their instructions - then work from there.

i think it was mostly a fluke

Re: An Alien Mind

#276

I am still waiting for a cure to cancer. For a guaranteed prophylactic against Alzheimer's and dementia. For flying cars for everyone. For space bases throughout the solar system. For weather control. For all trains to be self-driving. For all those power lines across the world to go away. For an end to poverty. If things are going so well, then how come things still aren't going so well?

>For all trains to be self-driving

Given that we have self-driving cars, isn't this easier if someone really wanted? I guess compared to cars the marginal savings is not worth it though.

Re: An Alien Mind

#277
post #8

This is a good essay, and makes me hopeful. I’m on the record saying that it is extremely dangerous to slow down because the race for AGI is a zero-trust game — defections pay - and combined with a compounding returns model on defection, if you have any strategic adversaries whatsoever you MUST NOT slow. For slowing to make sense, you need to believe that you can transform the zero trust game into a cooperative game,…

slowing can also make sense if you know you're running full force into a bomb or a wall even if other are close behind.

Re: An Alien Mind

#278
I think that we are speed-running towards a future that very few people really want, and I find it terrifying that few companies feel entitled to choose this future for the rest of us. Some of the arguments in this document would call for an immediate, global, pause on frontier AI training: we need time to consider how and to what extent AI should be part of our future. Personally, I can't picture a scenario where humanity thrives alongside an alien super-intelligence, especially if it cannot be fully controlled. Let aside super-intelligence, I am not even sure that deploying an AGI that replaces (instead of augmenting/assisting) humans in most intellectual tasks would be in the best interest of our species. This conversation has to happen, on a global level and as soon as possible.

Re: An Alien Mind

#279
post #104
post #20

Earlier quoted context omitted.

1. The grandparent commentator is describing strategic behavior of dangerous technologies. Game theory / mechanism design primitives. 2. If there is competition for resources among autonomous agents, the "strongest" agent wins (conceptually the most adaptive / evolutionarily fit). 3. Computer programs serve up webapps today, but they also run utility companies, dams, nuclear arsenals, factory production floors, autom…

OK so if it's a computer security issue we're worried about, and there is a credible threat, then probably the answer is to build more secure systems? We know how to do it but choose not to because it's very expensive and usually the threat isn't severe enough to warrant it. If we decide we can't or would rather not build secure computerized systems, then the answer could be don't incorporate computers into those sys…

Not sure why you were flagged either -- it was a reasonable comment that clearly spawned a discussion! People are strange.

> There's no essential reason to have utility companies, dams, nuclear arsenals, factory production floors, mines, cars, planes, ships, etc all be computerized

Indeed, this is a great first step! Once agentic loops and multi-purpose robotics can be combined we do face an issue of strengthening the airgap.

> And none of that pie in the sky shit has anything to do with language models.

I'm of the opinion that we should be mindful about the capabilities we allow a truly non-human mind to perform. Blocking via solid security and airgapping for critical systems makes sense to me for most cases, aligned I believe to your thoughts here. The recent ChatGPT hack shows that the "paperclip apocalypse" is still instructive. These systems target edge cases and zerodays to cheat!

Re: An Alien Mind

#280

Earlier quoted context omitted.

Maybe, maybe not. It's not unreasonable that these systems cap out at some point, or perhaps fizzle away entirely. The businesses that create these systems are not profitable and run at a massive historical and go-forward loss. New data centers required to operate these systems are facing increasing pushback at local levels. New construction is not guaranteed. Energy and power grid constraints exist as well. Governme…

For better or for worse, this technology isn't going away any more than search engines, smartphones, or social media have gone away.

Those technologies reached profitability relatively early, if not immediately.
Post reply on HN