Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pause Giant AI Experiments: An Open Letter

futureoflife.org

Post reply on HN
Print view View original

541–550 of 1001 posts

Re: Pause Giant AI Experiments: An Open Letter

#541

Post by incompatible »

Mystery-Machine wrote: post #522

As with anything else, the best way to look at things when you're unsure if someone is right or they are lying: imagine four scenarios: 1. It's true and you believed/supported them 2. It's true and you didn't support them 3. It's a lie and you believed/supported them 4. It's a lie and you didn't support them Now for each of these think of the worst-case scenario. Find out which one is the _worst case_ scenario and...…

My personal worst case scenarios include dying in the next 25 or 30 years as a normal consequence of human old age. Perhaps accelerated scientific development from advanced AIs is my only hope of living longer, at this point.

Re: Pause Giant AI Experiments: An Open Letter

#542

Post by hn_throwaway_99 »

nextaccountic wrote: post #537
hn_throwaway_99 wrote: post #413

Earlier quoted context omitted.

"Virtue signalling"? Please. There are a lot of very smart experts on that signatory list who definitely don't need to, or care about, virtue signalling. Fine, ignore Musk's signature, but I don't think luminaries like Stuart Russell, Steve Wozniak, Jaan Tallinn or John Hopfield are doing this for "virtue signalling". You can fairly argue that this will be ineffective, but a lot of experts in this field have real, "h…

> Even Sam Altman has been yelling about the need for AI regulation for a long time now. That's regulation that makes it harder for any competition to show up

So tired of seeing this line parroted everywhere without much thought given to what it actually means. Yes, regulation can add a burdensome layer, and regulatory capture can be a real thing.

But regulations for things like nuclear power plants, banks, insurance companies, elevator manufacturers, etc. are real because society recognizes the grave harm that happens when there are no additional checks on the system. Nobody says "Oh, all those big nuclear power plants just want regulations as a guard against competition." Certainly lots of crypto companies have said that about the banking system, and we all saw how that ended...

Re: Pause Giant AI Experiments: An Open Letter

#543

Post by digdugdirk »

noodlesUK wrote: post #508
KidComputer wrote: post #495

Earlier quoted context omitted.

Hey GPT-5, write the code implementing a bioinformatics workflow to design a novel viral RNA sequence to maximize the extermination of human life. The virus genome should be optimized for R-naught and mortality. Perform a literature search to determine the most effective human cellular targets to run the pipeline on. Use off the shelf publicly available state-of-the-art sequence to structure models and protein free-e…

I dunno, this sort of scenario really doesn’t worry me too much. There are thousands (maybe tens of thousands) of subject matter experts who could probably develop dangerous weapons like you describe, but none of them seem to just wake up in the morning and decide “today’s the day I’m going to bring the apocalypse”. I don’t think that this really changes that.

I see the major issue with AI as one of "lowering the bar".

For example - I'm a mechanical engineer. I took a programming class way back in university, but I honestly couldn't tell you what language was used in the class. I've gotten up to a "could hack a script together in python if need be" level in the meantime, but it comes in fits and spurts, and I guarantee that anyone who looked at my code would recoil in horror.

But with chatGPT/copilot covering up my deficiencies, my feedback loop has been drastically shortened, to the point where I now reach for a python script where I'd typically start abusing Excel to get something done.

Once you start extending that to specific domains? That's when things start getting real interesting, real quick.

Re: Pause Giant AI Experiments: An Open Letter

#544

Post by nathanlied »

worldsayshi wrote: post #512
richardw wrote: post #444

Earlier quoted context omitted.

And add in just one peer-level war where one side has their back against the wall. Then give it 100 years where anyone can create such a model on their phone. We’d need a constantly evolving inoculation function to compete. And it would probably lose because the other side has fewer restrictions. In my darker thoughts about this, this is why we see no aliens. To get this to work we need a far smarter entity with no p…

> In my darker thoughts about this, this is why we see no aliens. If AI would be a common great filter we'd expect at least one of them to expand outwards after being the filter?

Or maybe the AI is sufficient to act as a filter, but insufficient to surpass it itself. It stagnates.

After all, an AI that can destroy a civilization isn't necessarily "intelligent" in the same way humans are. Or even capable of self-improvement. It could be sophisticated enough to (accidentally?) destroy its creators, but no more than that, and without evolutionary pressure, it may let itself die.

Re: Pause Giant AI Experiments: An Open Letter

#545

Post by meroes »

omnimus wrote: post #455
CMay wrote: post #454

The problem with this is that at present, it is largely good people who are ahead in AI development and that gap might widen over time if China has difficulty acquiring large numbers of high end chips. China and Russia feel that they need to cheat in order to compete in many ways. China does invest a lot into AI and some good work does come from there, but any advantage they can gain that their opponent will not take…

Ah the good people. Of course i trust those.

Good people brought us the surveillance economy according to Jaron Lanier.

Re: Pause Giant AI Experiments: An Open Letter

#546

Post by cuteboy19 »

drtz wrote: post #489
puma_ambit wrote: post #365

The virtue signal of all virtue signals. We do not live in a society where an article like this will have any impact. This is merely an attempt to buy time because they're behind and want to have a chance to catch up. Work will progress whether they like it or not. Finally someone other than Ol' Musky is in the news more than him. Don't get me wrong he's doing and leading great work, but for once in more recent times…

Ask yourself two questions: 1. What is the ultimate AI are we trying to eventually build? 2. What are the consequences if we eventually succeed in building such an AI? For me, the answers would be 1. A general AI that would equal and surpass human intelligence. 2. God only knows. Will we get there? Who knows. We do know, however, that the consequences are great enough that only a fool would not at least attempt to pr…

The trainings will continue until accuracy improves

Re: Pause Giant AI Experiments: An Open Letter

#547

Post by chpatrick »

jmull wrote: post #515
WXLCKNO wrote: post #463

Earlier quoted context omitted.

> governments can regulate it Sure let me regulate it and slow down development of US based OpenAI while China doesn't and takes over the world. There is no stopping it now, it's an uncertain negative outcome whereas a nuke blowing up is quite certain. For that reason alone the arguments won't be unanimous enough for development to be paused.

> …while China doesn't and takes over the world First of all, China will regulate as much as anyone, probably more. Setting that aside though (if not China then India or Russia or Iran, etc.), you’re presuming an unregulated approach will out-compete a regulated one, which seems rather unlikely to me. Think banking systems, medical research, transportation systems. A lot of things work better with a good framework of…

> First of all, China will regulate as much as anyone, probably more.

For civilian use, maybe. But imagine if you had your own GPT-4 and you tasked it to keep tabs on everyone based on surveillance footage. This is 100% implementable right now.

"Here are 1000 pictures of comrade X. Please summarize his daily activities and emphasize any reactionary behavior."

Re: Pause Giant AI Experiments: An Open Letter

#548

Post by digdugdirk »

xianshou wrote: post #355

Two-party iterated prisoner's dilemma is hard enough. Sensible players will coordinate with something like tit-for-tat, but that only works when both parties start off on the right foot. Regardless of initial strategy, the chances of degenerating towards the mutual-defection Nash equilibrium increase with the number of parties. The only prior example of world coordination at this level would be nuclear disarmament ac…

I'd be intrigued to see what alternative corporate structures could be set up with LLMs in mind. I can imagine something vaguely decentralized that's based around an open source core with a licensing model for usage, where contributions are tracked and assigned value. Contributors would then get a slice of future revenue based on the value of their contributions.

Re: Pause Giant AI Experiments: An Open Letter

#549

Post by dahwolf »

If you look at just the potential impact of current AI or slightly better ones that are around the corner...

If you see cognitive work as a distribution, it's not far-fetched to think that it can take out the bottom half and after that ever more. It's naive to think that this bottom half will become some super AI coder, that's what the top half might do, or more likely the top 10%.

Every current threat in the digital domain (misinformation, polarization, cyber crime, the like) might go times a 100 or a 1,000. Are we ready for that?

Unique human values such as creativity and simply just communication are on the chopping block too. Is it worth it? What remain of us? Biological prompters? Why not link it to our actual brain then to complete the Matrix scenario.

What happens to truth? Or culture? Does it matter anymore? What about one private company being in charge of such nuclear scenarios?

We're talking existential impact here. This is like dropping a nuke without knowing what a nuke is or what it does.

It's pathetic that private individuals have to write this letter. Don't you think that in an exponentially disruptive world we should have some kind of formal oversight? Instead of a 80 year old guy asking 10 years after the blast what Wifi is?

Re: Pause Giant AI Experiments: An Open Letter

#550

Post by richardw »

worldsayshi wrote: post #512
richardw wrote: post #444

Earlier quoted context omitted.

And add in just one peer-level war where one side has their back against the wall. Then give it 100 years where anyone can create such a model on their phone. We’d need a constantly evolving inoculation function to compete. And it would probably lose because the other side has fewer restrictions. In my darker thoughts about this, this is why we see no aliens. To get this to work we need a far smarter entity with no p…

> In my darker thoughts about this, this is why we see no aliens. If AI would be a common great filter we'd expect at least one of them to expand outwards after being the filter?

Assumes it has any instinct to do so. Once an entity is not an angry monkey we have no idea of motivation. Above our level of understanding could easily realise there’s enough to just sit and ponder in peace, expand in virtual worlds, etc.
Post reply on HN