Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pause Giant AI Experiments: An Open Letter

futureoflife.org

Post reply on HN
Print view View original

511–520 of 1001 posts

Re: Pause Giant AI Experiments: An Open Letter

#511

Post by lend000 »

KidComputer wrote: post #495
lend000 wrote: post #481

If intelligence is all you need to dominate the world, why do some of the most powerful world leaders seem to not be more than a standard deviation above average intelligence (or at least they were before they became geriatric)? Add in the fact that GPT-4 does not "think" very fast, has no arms or legs, needs to run on expensive specialized hardware, and essentially seems like a search algorithm that lacks creativity…

Hey GPT-5, write the code implementing a bioinformatics workflow to design a novel viral RNA sequence to maximize the extermination of human life. The virus genome should be optimized for R-naught and mortality. Perform a literature search to determine the most effective human cellular targets to run the pipeline on. Use off the shelf publicly available state-of-the-art sequence to structure models and protein free-e…

I've been trying to use GPT-4 for my hard science startup, and it really has nothing to offer when you push the boundaries of what has been done by even a little, but it's great for speeding up coding.

Once we do have an AI capable of extraordinary innovation (hopefully in 10 years! But probably a lot longer), it will be obvious, and it will unfortunately be removed from the hands of the plebs based on fearmongering around scenarios like what you mentioned (despite the enormous resources and practical hurdles that would be necessary for a mentally unhinged individual to execute such instructions, even if an AI were capable of generating them and it made it past its filters / surveillance).

Re: Pause Giant AI Experiments: An Open Letter

#512

Post by worldsayshi »

richardw wrote: post #444
xianshou wrote: post #355

Two-party iterated prisoner's dilemma is hard enough. Sensible players will coordinate with something like tit-for-tat, but that only works when both parties start off on the right foot. Regardless of initial strategy, the chances of degenerating towards the mutual-defection Nash equilibrium increase with the number of parties. The only prior example of world coordination at this level would be nuclear disarmament ac…

And add in just one peer-level war where one side has their back against the wall. Then give it 100 years where anyone can create such a model on their phone. We’d need a constantly evolving inoculation function to compete. And it would probably lose because the other side has fewer restrictions. In my darker thoughts about this, this is why we see no aliens. To get this to work we need a far smarter entity with no p…

> In my darker thoughts about this, this is why we see no aliens.

If AI would be a common great filter we'd expect at least one of them to expand outwards after being the filter?

Re: Pause Giant AI Experiments: An Open Letter

#513

Post by dumbaccount123 »

m_ke wrote: post #41

GPT-4 might be close to the best we'll get on the general LLM model front for a while since they trained on a huge chunk of web text. Next real advances will probably be in tuning them for specific applications in medicine, law, accounting, marketing, coding and etc. As someone running a one man company I can't wait for the cost of accounting, legal and copywriting to approach 0. Cost of shipping products will also g…

Wrong https://www.digitaltrends.com/computing/gpt-5-artificial-gen...

Re: Pause Giant AI Experiments: An Open Letter

#514

Post by mewpmewp2 »

jmull wrote: post #430

Not sure why people think AI development can’t be paused. A letter alone probably won’t do it, but a movement — perhaps coalescing around the principles laid out in letter signed by influential people — can. Governments can regulate it. Consumers and companies can, if they want, punish companies that keep moving forward on AI. People can keep making AIs but would have difficulty releasing them, which greatly reduces…

> People can keep making AIs but would have difficulty releasing them, which greatly reduces their impact.

And it would then only increase impact of bad actors, who can and will keep using the AI.

Re: Pause Giant AI Experiments: An Open Letter

#515

Post by jmull »

WXLCKNO wrote: post #463
jmull wrote: post #430

Not sure why people think AI development can’t be paused. A letter alone probably won’t do it, but a movement — perhaps coalescing around the principles laid out in letter signed by influential people — can. Governments can regulate it. Consumers and companies can, if they want, punish companies that keep moving forward on AI. People can keep making AIs but would have difficulty releasing them, which greatly reduces…

> governments can regulate it Sure let me regulate it and slow down development of US based OpenAI while China doesn't and takes over the world. There is no stopping it now, it's an uncertain negative outcome whereas a nuke blowing up is quite certain. For that reason alone the arguments won't be unanimous enough for development to be paused.

> …while China doesn't and takes over the world

First of all, China will regulate as much as anyone, probably more.

Setting that aside though (if not China then India or Russia or Iran, etc.), you’re presuming an unregulated approach will out-compete a regulated one, which seems rather unlikely to me. Think banking systems, medical research, transportation systems. A lot of things work better with a good framework of organization.

Re: Pause Giant AI Experiments: An Open Letter

#516

Post by nice_byte »

JW_00000 wrote: post #500
nice_byte wrote: post #492

Earlier quoted context omitted.

> In the next 10 years, AI/robots will generate wealth at an unprecedented scale. Food, clothing and shelter will be plentiful. Anyone who believes in the possibility of post-scarcity society must be either naive or trolling. Something cannot be made from nothing, therefore scarcity cannot be overcome, even assuming that all planning and execution is performed by superhuman synthetic minds. Assuming that it's theoret…

> Education beyond basic reading and writing will become unattainable (and, frankly, unnecessary) Is this possible in democracies with free (i.e. state-provided) education? Why would a majority vote to abolish education?

i didn't say it would get abolished, more like wither away. here's a possible path to it:

1) school program getting progressively dumbed down; 2) fewer and fewer people choosing to continue education after school.

continuing education is mostly necessary to become specialized and perform knowledge work, it would be a waste of time to do so with emergence of a.i. agents capable of performing these functions even at a level _comparable_ to humans (let alone exceeding).

repeat this for a few decades and you have a new dark age.

Re: Pause Giant AI Experiments: An Open Letter

#518

Post by xiphias2 »

lend000 wrote: post #511
KidComputer wrote: post #495

Earlier quoted context omitted.

Hey GPT-5, write the code implementing a bioinformatics workflow to design a novel viral RNA sequence to maximize the extermination of human life. The virus genome should be optimized for R-naught and mortality. Perform a literature search to determine the most effective human cellular targets to run the pipeline on. Use off the shelf publicly available state-of-the-art sequence to structure models and protein free-e…

I've been trying to use GPT-4 for my hard science startup, and it really has nothing to offer when you push the boundaries of what has been done by even a little, but it's great for speeding up coding. Once we do have an AI capable of extraordinary innovation (hopefully in 10 years! But probably a lot longer), it will be obvious, and it will unfortunately be removed from the hands of the plebs based on fearmongering…

I guess it will get more interesting for your work when it integrates with BioTech startup apis as plugins (I imagine not too cheap ones)

Re: Pause Giant AI Experiments: An Open Letter

#519

Post by mewpmewp2 »

jmull wrote: post #447
throwthrowuknow wrote: post #442

Earlier quoted context omitted.

The publicly released AIs are not the ones we need to worry about. Pushing research underground will worsen the situation

It’s wide AI use that will disrupt society. Underground AI will not be widely used. Let’s not imbue individual underground AI with magical abilities.

> Underground AI will not be widely used.

It will 100% be used and improved for nefarious purposes.

By criminal groups or competing countries themselves.

It's worse if bad actors are using it in secret to manipulate the masses rather than it being completely transparent to everyone.

Imagine only bad actors being able to do deep fakes and regular people not knowing it was possible.

Re: Pause Giant AI Experiments: An Open Letter

#520

Post by mewpmewp2 »

nopinsight wrote: post #166
juancn wrote: post #156

Stopping now would be extremely dangerous and borderline stupid. If you stop now, you're just left behind, because there's no way everyone will stop. At this point the only logical course of action in an adversarial situation is to double down and keep researching, otherwise some other country or culture with different (and possibly worse) values ends up dominating the technology and you're left behind in the dust. T…

They are calling for a pause, not a stop. It’s quite clear that OpenAI has a significant lead over everyone else. The only other country outside the west that even has a chance at developing something better than GPT-4 soon is China. China has a pretty cautious culture as well so it’s quite possible that a bilateral moratorium can be negotiated with them. ADDED: Even without considering X-risks, China’s rulers cannot…

If OpenAI as a start up was able to do it, certainly full countries can do it if they see the evidence and will invest in it.
Post reply on HN