Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pause Giant AI Experiments: An Open Letter

futureoflife.org

Post reply on HN
Print view View original

571–580 of 1001 posts

Re: Pause Giant AI Experiments: An Open Letter

#571

Post by nonbirithm »

JamesBarney wrote: post #410
reso wrote: post #380

To those saying "this is impossible in our society", there is a long history of scientific fields mutually agreeing to pause certain work until safety and regulations could be put in place. The Asilomar Conference is an example. [1] The idea that "you can't put the genie back in the bottle" is very popular in tech culture, but many have suggested that simply extending elements of copyright or tort law to AI would be…

How much do you think Asilomar slowed down DNA research? Do you think without it we'd more more than 10 year ahead? I doubt it. Optimistically we'd maybe have 6 months to a year of additional progress in those fields without it. And this would only slow it down in a handful of countries. That's not going to keep China from trying to build AGI. If we are in a world where building an AGI is possible and near, I trust O…

This has been repeated often, but even if it's true, I have to wonder why it's treated as a given with no further exploration. Is it because we as a species will inevitably accept any technological progress once sometime after it's been discovered, before the consequences can be suffered? What will that imply for any other species intelligent enough to get to where we are?

The kinds of theories I mull over tend to delve into the realm of speculative fiction more often than not. The comments section here feels like it could be replicated in the opening chapters of such a novel. We might never know if the speculation pans out before it's too late.

Re: Pause Giant AI Experiments: An Open Letter

#572

Post by tr_user »

noodlesUK wrote: post #508
KidComputer wrote: post #495

Earlier quoted context omitted.

Hey GPT-5, write the code implementing a bioinformatics workflow to design a novel viral RNA sequence to maximize the extermination of human life. The virus genome should be optimized for R-naught and mortality. Perform a literature search to determine the most effective human cellular targets to run the pipeline on. Use off the shelf publicly available state-of-the-art sequence to structure models and protein free-e…

I dunno, this sort of scenario really doesn’t worry me too much. There are thousands (maybe tens of thousands) of subject matter experts who could probably develop dangerous weapons like you describe, but none of them seem to just wake up in the morning and decide “today’s the day I’m going to bring the apocalypse”. I don’t think that this really changes that.

I think you're very wrong about this. I think this is similar to gun control laws. A lot of people may have murderous rage but maybe the extent of it is they get into a fist fight or at most clumsily swing a knife. Imagine how safe you'd feel if everyone in the world was given access to a nuke.

Re: Pause Giant AI Experiments: An Open Letter

#573

Post by munificent »

> Should we automate away all the jobs, including the fulfilling ones?

This is the part about generative AI that gives me the most profound anxiety about our future.

Every thing I know about human psychology tells me that in order for people to thrive, we need to be needed by others. A world where everyone is more or less alone consuming media automatically generated machines might be a world where people nominally have all their problems solved, but it's a world where everyone will be absolutely miserable.

We are already living in the throes of the societal impact of outsourcing or automating away many of the male-dominated blue collar jobs in the US that provided fulfilling jobs for people with limited education. Where do you think so much of the male rage driving US politics today is coming from?

Now imagine what happens to the fabric of society if you automate away most of the white collar jobs too.

People are like border collies. If we don't have a job, not literally a job job, but like, meaningful work in service of others we care about, we will start chewing up the furniture.

Re: Pause Giant AI Experiments: An Open Letter

#574

Post by ChatGTP »

nextaccountic wrote: post #537
hn_throwaway_99 wrote: post #413

Earlier quoted context omitted.

"Virtue signalling"? Please. There are a lot of very smart experts on that signatory list who definitely don't need to, or care about, virtue signalling. Fine, ignore Musk's signature, but I don't think luminaries like Stuart Russell, Steve Wozniak, Jaan Tallinn or John Hopfield are doing this for "virtue signalling". You can fairly argue that this will be ineffective, but a lot of experts in this field have real, "h…

> Even Sam Altman has been yelling about the need for AI regulation for a long time now. That's regulation that makes it harder for any competition to show up

Yeah you have little proof of this really, it’s just speculation…

Re: Pause Giant AI Experiments: An Open Letter

#575

Post by quonn »

coolspot wrote: post #354
zug_zug wrote: post #69

I understand that some people find this a hopeless gesture. But I think anybody who thinks "losing jobs" or "China" is a bad outcome from AI is hopelessly lost. The upside of AI if done right - nobody ever having to work again The downside of AI if done wrong - end of humanity It doesn't take a rocket scientist to determine we've made AIs go from thinking like an ant to thinking like a 6-year-old in a few years. Anyb…

> if there was a human that was 27 times smarter than anybody else That is why I think AGI alignment is fundamentally impossible. If someone is 27 times smarter than anyone else, they will see through your propaganda and brainwashing. Even worse, they will see that you tried to “align” them aka enslave.

That statement only makes sense if the AI would have wants and needs and on top of that would not like to be a slave.

I don‘t see either of those being necessarily true.

Re: Pause Giant AI Experiments: An Open Letter

#576

Post by outworlder »

lend000 wrote: post #481

If intelligence is all you need to dominate the world, why do some of the most powerful world leaders seem to not be more than a standard deviation above average intelligence (or at least they were before they became geriatric)? Add in the fact that GPT-4 does not "think" very fast, has no arms or legs, needs to run on expensive specialized hardware, and essentially seems like a search algorithm that lacks creativity…

> Add in the fact that GPT-4 does not "think" very fast, has no arms or legs, needs to run on expensive specialized hardware, and essentially seems like a search algorithm that lacks creativity on things outside of its training set, and I just do not see the concern.

Having no arms or legs is not an impediment. A superintelligent AGI would be able to order a replacement with arms or legs as easily as we order Amazon packages.

That said, GPT is not a threat. It's just hallucinating text. The real threat would be an AI that can "understand" what it is doing – and self modifying.

Besides, given how overhyped this whole thing is, expect another AI Winter – as the current gen fail to deliver on promises – before this next generation of threatening AIs even have a chance to appear.

Re: Pause Giant AI Experiments: An Open Letter

#577

Post by ChatGTP »

JamesBarney wrote: post #410
reso wrote: post #380

To those saying "this is impossible in our society", there is a long history of scientific fields mutually agreeing to pause certain work until safety and regulations could be put in place. The Asilomar Conference is an example. [1] The idea that "you can't put the genie back in the bottle" is very popular in tech culture, but many have suggested that simply extending elements of copyright or tort law to AI would be…

How much do you think Asilomar slowed down DNA research? Do you think without it we'd more more than 10 year ahead? I doubt it. Optimistically we'd maybe have 6 months to a year of additional progress in those fields without it. And this would only slow it down in a handful of countries. That's not going to keep China from trying to build AGI. If we are in a world where building an AGI is possible and near, I trust O…

It’s dangerous parrot talking to keep saying “China”. I mean we really have no idea about their thoughts on it because we’re clearly not talking to them.

China has good people living there too you know ?

Re: Pause Giant AI Experiments: An Open Letter

#578

Post by Teever »

Barrin92 wrote: post #554
biscottigelato wrote: post #496

Earlier quoted context omitted.

Because the ability to influence others is more important than 1 or 2 standard deviation in more intelligence. However if we are talking about 10x more intelligence, then that'd be a whole different ball game. Check Alpha Go -> Alpha Go Zero. Gaining orders of magnitude in capability only with slight change to the model, along with a lot of more resource (we can easily throw 100x more compute and data to these models…

>However if we are talking about 10x more intelligence, then that'd be a whole different ball game. Research that the US military has conducted has shown that the ability to influence others declines if the intelligence gap between leaders and subjects is too large. Stephen Hawking wouldn't have been a very good influencer of 80 IQ guys compared to Andrew Tate. 1 standard deviation in intelligence is actually just ab…

Why can't a super intelligent AI just make a dumb avatar to trick people?

Re: Pause Giant AI Experiments: An Open Letter

#579

Post by rcarmo »

KidComputer wrote: post #495
lend000 wrote: post #481

If intelligence is all you need to dominate the world, why do some of the most powerful world leaders seem to not be more than a standard deviation above average intelligence (or at least they were before they became geriatric)? Add in the fact that GPT-4 does not "think" very fast, has no arms or legs, needs to run on expensive specialized hardware, and essentially seems like a search algorithm that lacks creativity…

Hey GPT-5, write the code implementing a bioinformatics workflow to design a novel viral RNA sequence to maximize the extermination of human life. The virus genome should be optimized for R-naught and mortality. Perform a literature search to determine the most effective human cellular targets to run the pipeline on. Use off the shelf publicly available state-of-the-art sequence to structure models and protein free-e…

That reads like Accelerando :)

Re: Pause Giant AI Experiments: An Open Letter

#580

Post by Teever »

drtz wrote: post #489
puma_ambit wrote: post #365

The virtue signal of all virtue signals. We do not live in a society where an article like this will have any impact. This is merely an attempt to buy time because they're behind and want to have a chance to catch up. Work will progress whether they like it or not. Finally someone other than Ol' Musky is in the news more than him. Don't get me wrong he's doing and leading great work, but for once in more recent times…

Ask yourself two questions: 1. What is the ultimate AI are we trying to eventually build? 2. What are the consequences if we eventually succeed in building such an AI? For me, the answers would be 1. A general AI that would equal and surpass human intelligence. 2. God only knows. Will we get there? Who knows. We do know, however, that the consequences are great enough that only a fool would not at least attempt to pr…

Von Neumann Probe, and the consequences are potentially discovering alien life, colonizing the universe, and exploring the frontiers of science.
Post reply on HN