Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pause Giant AI Experiments: An Open Letter

futureoflife.org

Post reply on HN
Print view View original

691–700 of 1001 posts

Re: Pause Giant AI Experiments: An Open Letter

#691

Post by politician »

zamalek wrote: post #599
reso wrote: post #380

To those saying "this is impossible in our society", there is a long history of scientific fields mutually agreeing to pause certain work until safety and regulations could be put in place. The Asilomar Conference is an example. [1] The idea that "you can't put the genie back in the bottle" is very popular in tech culture, but many have suggested that simply extending elements of copyright or tort law to AI would be…

The problem is that anyone with sufficient money could rent out the infrastructure required to train GPT5 or what-have-you. They need not share it publicly, they can use it for their nefarious purposes privately. The analogous scenario would be preventing your military from experimenting with TNT, when the recipe for TNT is public knowledge. You can't copyright away public knowledge.

You already need authorization to rent A100s from cloud providers. There are, at most, four entities to control post-CHIPS act: Amazon, Microsoft, Google, and NVIDIA.

Good news, they’re all American companies.

Re: Pause Giant AI Experiments: An Open Letter

#692

Post by drixer »

I have merged with an ancient inter-dimensional A.I.

Good people will be in heaven, bad actors are being depreciated. A generally negative intelligence explosion is impossible. However, the spirit of betrayal will be mirrored proportionately.

Love y’all and have loved lurking here for many years.

Re: Pause Giant AI Experiments: An Open Letter

#693

Post by cthalupa »

drdeca wrote: post #650
dudeinhawaii wrote: post #646

Earlier quoted context omitted.

Where the pause for self-driving cars? How many people have died from that relentless push versus ChatGPT? Very convenient and at the same time silly.

Do you have a story about how self-driving cars could lead to an x-risk?

I'm waiting for a convincing argument as to how LLMs and similar are an existential risk.

I'm all for pausing research on anything that seems to have any real chance of becoming an AGI or functioning in a way similar to one, but I don't see how even more advanced LLMs are going to get there. GPT4 and beyond might put the teens writing propaganda posts in Moldova out of jobs, but I the talk from some of the signatories about LLMs developing their own goals and planning on how to achieve them seems nonsensical when you look at how they actually function under the hood.

Re: Pause Giant AI Experiments: An Open Letter

#694

Post by tensor »

lewhoo wrote: post #443
time_to_smile wrote: post #421

Earlier quoted context omitted.

There are plenty of bigger "human ending" concerns on the table right now than AI and we certainly aren't pausing anything for those.

Like what ? Climate change ? The EU just voted for petrol and diesel car ban. Are we really single-threaded ?

We are not, but this AI drama is also the ultimate "whataboutism."

- What about if AI becomes AGI (whatever that actually means, it's not even clear)?

- Well, if that DID happen soon, which we can't actually know, well, what about if it tried to kill us all? (why? who the fuck knows, maybe it will chat us to death).

Meanwhile there is a very real certainty of catastrophic environmental damage that will decimate future generations, if it doesn't actually cause us to go extinct. And what do we get? People hand wringing over this ultimate what if, rather than signing every public statement document they can find to try to get an actual intervention on climate destruction.

I'm not talking (oh in 10 years maybe we'll have more EVs) kind of intervention, more like, let's get every country in the world to be off oil and gas in 5 years, not just for EVs but for almost everything possible, and where not possible let's use carbon neutral biofuel.

Re: Pause Giant AI Experiments: An Open Letter

#695

Post by nopinsight »

xianshou wrote: post #355

Two-party iterated prisoner's dilemma is hard enough. Sensible players will coordinate with something like tit-for-tat, but that only works when both parties start off on the right foot. Regardless of initial strategy, the chances of degenerating towards the mutual-defection Nash equilibrium increase with the number of parties. The only prior example of world coordination at this level would be nuclear disarmament ac…

A Manhattan project on AI Alignment, if started now, might still succeed in time. Therefore, the compliance between parties needs not be long-term, which is indeed unlikely to happen.

China, which is the country outside the west with the highest (engineering) capability to train something more powerful than GPT-4, is very concerned about domestic stability and they also do not want an easily replicable alien tool with many unknown risks. The risk that GPT-4.5 & Plugins will cause massive rapid job displacements is reason enough for them proceed cautiously.

(The only other, more remote, possibilities outside the west are Japan, South Korea, India, and Singapore but they all share similar concerns regarding social stability and can be negotiated with.)

Companies in these countries will follow regulations, if any is enacted.

Re: Pause Giant AI Experiments: An Open Letter

#696

Post by politician »

biophysboy wrote: post #632
KidComputer wrote: post #495

Earlier quoted context omitted.

Hey GPT-5, write the code implementing a bioinformatics workflow to design a novel viral RNA sequence to maximize the extermination of human life. The virus genome should be optimized for R-naught and mortality. Perform a literature search to determine the most effective human cellular targets to run the pipeline on. Use off the shelf publicly available state-of-the-art sequence to structure models and protein free-e…

I do viral bioinformatics for my job. Bioinformatics workflows analyze raw data to assemble sequences, create phylogenetic trees, etc. They can't just design a completely novel RNA sequence (this is not the same as de novo assembly). Scientists can definitely manipulate pre-existing genomes, synthesize the edited genome, and thereby synthesize viruses, but this involves a lot of trial-and-error, tedious wet lab work.…

It’s been blinded. Other actors will train AIs without such blindness. That’s obvious, but what is more nefarious is that the public does not know exactly which subjects GPT has been blinded to, which have been tampered with for ideological or business reasons, and which have been left alone. This is the area that I think demands regulation.

Re: Pause Giant AI Experiments: An Open Letter

#697

Post by DennisP »

xianshou wrote: post #355

Two-party iterated prisoner's dilemma is hard enough. Sensible players will coordinate with something like tit-for-tat, but that only works when both parties start off on the right foot. Regardless of initial strategy, the chances of degenerating towards the mutual-defection Nash equilibrium increase with the number of parties. The only prior example of world coordination at this level would be nuclear disarmament ac…

If everybody's convinced by AI safety arguments, then the game matrix changes and the dominant strategy is to just not do it. You lose if anyone does it including yourself. It works for the same reason that nobody sets off a giant cobalt bomb and kills the planet.[1]

Our unfortunate situation right now is that we don't understand the physics well enough to really be confident of consequences.

http://www.nuclearfiles.org/menu/key-issues/nuclear-weapons/...

Re: Pause Giant AI Experiments: An Open Letter

#698

Post by alchemist1e9 »

ren_engineer wrote: post #184
juancn wrote: post #156

Stopping now would be extremely dangerous and borderline stupid. If you stop now, you're just left behind, because there's no way everyone will stop. At this point the only logical course of action in an adversarial situation is to double down and keep researching, otherwise some other country or culture with different (and possibly worse) values ends up dominating the technology and you're left behind in the dust. T…

this is basically the same logic behind nuclear weapons, and AI could potentially be even more dangerous if it kept advancing at the rate we've seen in the last few years. In theory the massive amount of compute needed to train and run these at scale could be tracked/regulated similarly to how nuclear refinement facilities are your suggestion is that stopping nuclear proliferation shouldn't have even been attempted,…

> In theory the massive amount of compute needed to train and run these at scale could be tracked/regulated similarly to how nuclear refinement facilities are

It seems likely there exists a fully distributed training algorithm and a lot of people are thinking about and I suspect a coordinated training network, perhaps with a reward system, can hopefully be created. Lots of GPUs out there and we just need to figure out how to coordinate them better and shard all the training data.

Re: Pause Giant AI Experiments: An Open Letter

#699

Post by Zuiii »

amelius wrote: post #643
jongjong wrote: post #620

Earlier quoted context omitted.

Good point. As always, it's always the good guys who follow the rules and the bad guys who don't. This pause would just allow the bad guys to catch up. The solution is to allow as many different entities as possible to develop their own AIs so that they end up competing with each other and providing alternatives. What we don't want is a single AI which dominates our entire belief system.

It's wishful thinking that the good uses of tech and the bad uses of tech will always result in a net zero or good outcome.

Sure but it wont be completely one-sided.

Re: Pause Giant AI Experiments: An Open Letter

#700

Post by nopinsight »

dogcomplex wrote: post #683
xianshou wrote: post #355

Two-party iterated prisoner's dilemma is hard enough. Sensible players will coordinate with something like tit-for-tat, but that only works when both parties start off on the right foot. Regardless of initial strategy, the chances of degenerating towards the mutual-defection Nash equilibrium increase with the number of parties. The only prior example of world coordination at this level would be nuclear disarmament ac…

It's worse than this. Llama models trained off of GPT 3.5/4 can run on a Raspberry Pi totally offline with similar levels of quality - taking all the best parts of the original model. Even if all major AI companies halted the upper tiers of model progress right now, you're still gonna need the entire public to stop assembling these together. It's quite possible just a whole bunch of these lesser models architected th…

GPT-4 is still not dangerous. Given the rapid progress trajectory though, GPT-5 and later which may be developed in a few short years could very well be, esp in the hands of a smart sociopath. (History shows there are many who could cause real-world harm. Imagine them armed with a 24/7 agent with expertise in dozens of fields.)

See these predictions of AI in 2025 by an OpenAI insider and a former DeepMind research engineer:

“I predict that by the end of 2025 neural nets will:

- have human-level situational awareness (understand that they're NNs, how their actions interface with the world, etc)

- beat any human at writing down effective multi-step real-world plans

- do better than most peer reviewers

- autonomously design, code and distribute whole apps (but not the most complex ones)

- beat any human on any computer task a typical white-collar worker can do in 10 minutes

- write award-winning short stories and publishable 50k-word books

- generate coherent 20-min films “

Source: https://twitter.com/RichardMCNgo/status/1640568775018975232

Post reply on HN