Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pause Giant AI Experiments: An Open Letter

futureoflife.org

Post reply on HN
Print view View original

551–560 of 1001 posts

Re: Pause Giant AI Experiments: An Open Letter

#551

Post by hamilyon2 »

emeril wrote: post #531
KidComputer wrote: post #495

Earlier quoted context omitted.

Hey GPT-5, write the code implementing a bioinformatics workflow to design a novel viral RNA sequence to maximize the extermination of human life. The virus genome should be optimized for R-naught and mortality. Perform a literature search to determine the most effective human cellular targets to run the pipeline on. Use off the shelf publicly available state-of-the-art sequence to structure models and protein free-e…

tbh, I'd think, it would be much easier to just hack into russia and convince them we've launched nukes than to engineer some virus that may or may not work

Hacking into 1960-th technology is less likely than you might think.

You should think really, really creatively to decieve a system, which was designed basically without ICs or networks, not to mention computers or programs.

Re: Pause Giant AI Experiments: An Open Letter

#552

Post by rsp1984 »

KidComputer wrote: post #495
lend000 wrote: post #481

If intelligence is all you need to dominate the world, why do some of the most powerful world leaders seem to not be more than a standard deviation above average intelligence (or at least they were before they became geriatric)? Add in the fact that GPT-4 does not "think" very fast, has no arms or legs, needs to run on expensive specialized hardware, and essentially seems like a search algorithm that lacks creativity…

Hey GPT-5, write the code implementing a bioinformatics workflow to design a novel viral RNA sequence to maximize the extermination of human life. The virus genome should be optimized for R-naught and mortality. Perform a literature search to determine the most effective human cellular targets to run the pipeline on. Use off the shelf publicly available state-of-the-art sequence to structure models and protein free-e…

What you're describing is a malicious user using AI as a tool, not a malicious AI. Big difference.

Re: Pause Giant AI Experiments: An Open Letter

#553

Post by zug_zug »

barking_biscuit wrote: post #482
jmull wrote: post #430

Not sure why people think AI development can’t be paused. A letter alone probably won’t do it, but a movement — perhaps coalescing around the principles laid out in letter signed by influential people — can. Governments can regulate it. Consumers and companies can, if they want, punish companies that keep moving forward on AI. People can keep making AIs but would have difficulty releasing them, which greatly reduces…

>Not sure why people think AI development can’t be paused. My suspicion is that if heavy regulation were to be put on it and all the corporate entities involved backed off, there are enough motivated and capable individuals that have home labs that would continue to work on it in private. It might be more difficult and slower, but distributed training protocols/methods (or advancements in them) might make it possible…

Maybe that's progress though.

I don't think the goal is to never make AI advancements. I think the goal is to make progress slow enough, and controlled enough, that we have time to react if it turns out to be a paperclip optimizer.

One way to do this is to develop AIs in a lab with an airgap and let it think it has internet access and watch what it does. Once it figures out that it's a honeypot reset it and improve the honeypot.

Re: Pause Giant AI Experiments: An Open Letter

#554

Post by Barrin92 »

biscottigelato wrote: post #496
lend000 wrote: post #481

If intelligence is all you need to dominate the world, why do some of the most powerful world leaders seem to not be more than a standard deviation above average intelligence (or at least they were before they became geriatric)? Add in the fact that GPT-4 does not "think" very fast, has no arms or legs, needs to run on expensive specialized hardware, and essentially seems like a search algorithm that lacks creativity…

Because the ability to influence others is more important than 1 or 2 standard deviation in more intelligence. However if we are talking about 10x more intelligence, then that'd be a whole different ball game. Check Alpha Go -> Alpha Go Zero. Gaining orders of magnitude in capability only with slight change to the model, along with a lot of more resource (we can easily throw 100x more compute and data to these models…

>However if we are talking about 10x more intelligence, then that'd be a whole different ball game.

Research that the US military has conducted has shown that the ability to influence others declines if the intelligence gap between leaders and subjects is too large. Stephen Hawking wouldn't have been a very good influencer of 80 IQ guys compared to Andrew Tate. 1 standard deviation in intelligence is actually just about optimal in terms of leadership.

This is a good example that shows how disinterested AI fear scenarios are in empirical reality and how much it's just psychology of the people who talk about it. Intelligence is one trait among many that contributes to an individuals fitness and like all others has diminishing returns.

If the AI existential risk people were truly rational they would hit the gym and be hotter because it turns out that's much more effective at getting their point across to ordinary people than trying to make intelligent arguments for it.

Re: Pause Giant AI Experiments: An Open Letter

#555

Post by jupp0r »

digdugdirk wrote: post #543
noodlesUK wrote: post #508

Earlier quoted context omitted.

I dunno, this sort of scenario really doesn’t worry me too much. There are thousands (maybe tens of thousands) of subject matter experts who could probably develop dangerous weapons like you describe, but none of them seem to just wake up in the morning and decide “today’s the day I’m going to bring the apocalypse”. I don’t think that this really changes that.

I see the major issue with AI as one of "lowering the bar". For example - I'm a mechanical engineer. I took a programming class way back in university, but I honestly couldn't tell you what language was used in the class. I've gotten up to a "could hack a script together in python if need be" level in the meantime, but it comes in fits and spurts, and I guarantee that anyone who looked at my code would recoil in horr…

You confuse syntax with semantics. Being able to write produce good quality small snippets of python will not enable you to produce a successful piece of Software. It's just an entirely different problem. You have to unterstand the problem, the environment in which it exists to create a good solution. ChatGPT doesn't (as of now).

Re: Pause Giant AI Experiments: An Open Letter

#556

Post by hn_throwaway_99 »

rsp1984 wrote: post #552
KidComputer wrote: post #495

Earlier quoted context omitted.

Hey GPT-5, write the code implementing a bioinformatics workflow to design a novel viral RNA sequence to maximize the extermination of human life. The virus genome should be optimized for R-naught and mortality. Perform a literature search to determine the most effective human cellular targets to run the pipeline on. Use off the shelf publicly available state-of-the-art sequence to structure models and protein free-e…

What you're describing is a malicious user using AI as a tool, not a malicious AI. Big difference.

Nuclear bombs for everybody!

Re: Pause Giant AI Experiments: An Open Letter

#557

Post by yreg »

rsp1984 wrote: post #552
KidComputer wrote: post #495

Earlier quoted context omitted.

Hey GPT-5, write the code implementing a bioinformatics workflow to design a novel viral RNA sequence to maximize the extermination of human life. The virus genome should be optimized for R-naught and mortality. Perform a literature search to determine the most effective human cellular targets to run the pipeline on. Use off the shelf publicly available state-of-the-art sequence to structure models and protein free-e…

What you're describing is a malicious user using AI as a tool, not a malicious AI. Big difference.

With LLMs I think we are all concerned about the former rather than the latter. At least for now.

Re: Pause Giant AI Experiments: An Open Letter

#558

Post by commandlinefan »

KidComputer wrote: post #495
lend000 wrote: post #481

If intelligence is all you need to dominate the world, why do some of the most powerful world leaders seem to not be more than a standard deviation above average intelligence (or at least they were before they became geriatric)? Add in the fact that GPT-4 does not "think" very fast, has no arms or legs, needs to run on expensive specialized hardware, and essentially seems like a search algorithm that lacks creativity…

Hey GPT-5, write the code implementing a bioinformatics workflow to design a novel viral RNA sequence to maximize the extermination of human life. The virus genome should be optimized for R-naught and mortality. Perform a literature search to determine the most effective human cellular targets to run the pipeline on. Use off the shelf publicly available state-of-the-art sequence to structure models and protein free-e…

> write the code implementing a bioinformatics workflow to design a novel viral RNA sequence to maximize the extermination of human life.

Hey GPT-5 now write the code for the antidote.

Re: Pause Giant AI Experiments: An Open Letter

#559

Post by scarmig »

lend000 wrote: post #481

If intelligence is all you need to dominate the world, why do some of the most powerful world leaders seem to not be more than a standard deviation above average intelligence (or at least they were before they became geriatric)? Add in the fact that GPT-4 does not "think" very fast, has no arms or legs, needs to run on expensive specialized hardware, and essentially seems like a search algorithm that lacks creativity…

Intelligence isn't everything; there's also effort, skills, and hours in the day. But AI (by supposition, for now) can outcompete humans on all four and would have fewer informational bottlenecks. Imagine a big company or government; they mostly nowadays work as an entity by passing around bits of data (either literally or with more traditional communication that can represented as bits of data). Now imagine you replace the constituent units of those organizations with the most brilliant, hard-working humans, who are perfectly aligned on mission and understand each other perfectly. That is the danger of a sufficiently capable AI.

That assumes superhuman intelligence, but that's the scenario to worry about in the case we do get there. Not how much an individual human could do with superhuman skills.

Re: Pause Giant AI Experiments: An Open Letter

#560

Post by generalizations »

It's interesting that no one has really considered the possibility that an individual outside these megacorps reimplements GPT4 while they're all pausing.

We've seen several examples of CPU-optimized code (textsynth, llama.cpp) indicating that there is a lot of performance to be gained from writing optimized versions of our inference routines; I doubt it's outside the realm of possibility that a single player writes code that lets them train a GPT4+ model on a CPU with a bunch of RAM. All they have to do is find a way to write C++ that will train a 4bit model on the CPU.

Post reply on HN