Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pause Giant AI Experiments: An Open Letter

futureoflife.org

Post reply on HN
Print view View original

631–640 of 1001 posts

Re: Pause Giant AI Experiments: An Open Letter

#631

Post by lanstin »

hooande wrote: post #323
petilon wrote: post #68

> Should we automate away all the jobs, including the fulfilling ones? You don't necessarily need jobs because you don't need money. What you need is wealth. I am going to leave it to PG to explain the difference [1]: Wealth is not money. Wealth is stuff we want: food, clothes, houses, cars, gadgets, travel to interesting places, and so on. You can have wealth without having money. If you had a magic machine that cou…

> In the next 10 years, AI/robots will generate wealth at an unprecedented scale. Would love to see the reasoning behind this. Nothing I've seen about LLMs makes me think they will produce any kind of bounty for anyone.

I do remember feeling the same thing would be true of networked software in the early 90s. It is hard to tell to what extent it has happened.

Re: Pause Giant AI Experiments: An Open Letter

#632

Post by biophysboy »

KidComputer wrote: post #495
lend000 wrote: post #481

If intelligence is all you need to dominate the world, why do some of the most powerful world leaders seem to not be more than a standard deviation above average intelligence (or at least they were before they became geriatric)? Add in the fact that GPT-4 does not "think" very fast, has no arms or legs, needs to run on expensive specialized hardware, and essentially seems like a search algorithm that lacks creativity…

Hey GPT-5, write the code implementing a bioinformatics workflow to design a novel viral RNA sequence to maximize the extermination of human life. The virus genome should be optimized for R-naught and mortality. Perform a literature search to determine the most effective human cellular targets to run the pipeline on. Use off the shelf publicly available state-of-the-art sequence to structure models and protein free-e…

I do viral bioinformatics for my job. Bioinformatics workflows analyze raw data to assemble sequences, create phylogenetic trees, etc. They can't just design a completely novel RNA sequence (this is not the same as de novo assembly). Scientists can definitely manipulate pre-existing genomes, synthesize the edited genome, and thereby synthesize viruses, but this involves a lot of trial-and-error, tedious wet lab work. Also, the research on making more dangerous viruses through manipulation is extremely controversial and regulated, so its not like there is a wealth of scientific papers/experiments/data that a natural language model could just suck up.

Also, I asked GPT to do some of these things you suggested and it said no. It won't even write a scientific paper.

Re: Pause Giant AI Experiments: An Open Letter

#633

Post by fullstackchris »

haxiomic wrote: post #618
noodlesUK wrote: post #508

Earlier quoted context omitted.

I dunno, this sort of scenario really doesn’t worry me too much. There are thousands (maybe tens of thousands) of subject matter experts who could probably develop dangerous weapons like you describe, but none of them seem to just wake up in the morning and decide “today’s the day I’m going to bring the apocalypse”. I don’t think that this really changes that.

If everyone had an app on their phone with a button to destroy the world the remaining lifetime of the human race would be measured in milliseconds Now if this button was something you had to order from Amazon I think we’ve got a few days There’s a scenario where people with the intent will have the capability in the foreseeable future

like what? would you rather have a gpt5 or a nuke? pure fearmongering. what am i gonna do, text to speech them to death? give me a break

Re: Pause Giant AI Experiments: An Open Letter

#634

Post by streptomycin »

reso wrote: post #380

To those saying "this is impossible in our society", there is a long history of scientific fields mutually agreeing to pause certain work until safety and regulations could be put in place. The Asilomar Conference is an example. [1] The idea that "you can't put the genie back in the bottle" is very popular in tech culture, but many have suggested that simply extending elements of copyright or tort law to AI would be…

I had a professor who was involved in that kind of research at the time of the Asilomar Conference. He said it was all very good, everyone agreed to pause their research until people established guidelines for safety. And then once the guidelines were established and research was allowed to resume, everyone immediately published all the research that they had never stopped working on during the "pause".

Re: Pause Giant AI Experiments: An Open Letter

#635

Post by int_19h »

Mystery-Machine wrote: post #522

As with anything else, the best way to look at things when you're unsure if someone is right or they are lying: imagine four scenarios: 1. It's true and you believed/supported them 2. It's true and you didn't support them 3. It's a lie and you believed/supported them 4. It's a lie and you didn't support them Now for each of these think of the worst-case scenario. Find out which one is the _worst case_ scenario and...…

There's no unitary "we" to block further development of AI. There are way too many actors with too many strong incentives to develop it.

Thus in practice you also have to consider scenarios such as "it's a lie and you supported the block, but other actors that are adversarial to you ignored it, and now they hold the upper hand". Given that the current state of world affairs can be described as "dangerously close to WW3", this can also be a very literal life-or-death matter.

Re: Pause Giant AI Experiments: An Open Letter

#636

Post by lanstin »

slowmovintarget wrote: post #337
jillesvangurp wrote: post #271

The cat is out of the bag. Some people might volunteer to slow down. But others inevitably won't. And that creates this sense of FOMO with everyone. I think the reason OpenAI is choosing to operate in the open has been exactly this. This way they get to control the narrative, act as a first mover in the market, and ensure it is done right (by their notion of this). Of course others are working on the same topics and…

> The cat is out of the bag. As GPT-4 puts it "the confetti has left the cannon."

Sam Altman in his very interesting interview with Kara Swisher did say that one’s initial experience with these models tends to be a lot more “wow” than after using them for a while.

Re: Pause Giant AI Experiments: An Open Letter

#637

Post by lanstin »

int_19h wrote: post #627
lanstin wrote: post #613

Earlier quoted context omitted.

I feel more sympathy for that Google engineer that feel in love with their LLM. I am sure more people will become attached, at least once the rate limits are relaxed.

The rea trap is that LLMs can simulate empathy surprisingly well. If you have many problems to rant about but nobody willing to listen, the LLM is always there, it will never get bored or tell you to shut up, and it will always respond in encouraging and "positive" ways. Given how many people today do not have anyone like that in their lives, it's no wonder that they form an emotional bond.

I said thanks to each answer :)

Re: Pause Giant AI Experiments: An Open Letter

#639

Post by lukeplato »

Is it even possible for our generation, let alone past or future generations, to choose ignorance? Can ignorance only be chosen before a discovery occurs?

AGI goes hand-in-hand with stewardship and utopic visions but I certainly don't see our generation being the ideal candidate.

Re: Pause Giant AI Experiments: An Open Letter

#640

Post by ilaksh »

xianshou wrote: post #355

Two-party iterated prisoner's dilemma is hard enough. Sensible players will coordinate with something like tit-for-tat, but that only works when both parties start off on the right foot. Regardless of initial strategy, the chances of degenerating towards the mutual-defection Nash equilibrium increase with the number of parties. The only prior example of world coordination at this level would be nuclear disarmament ac…

What nuclear disarmament?
Post reply on HN