Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pause Giant AI Experiments: An Open Letter

futureoflife.org

Post reply on HN
Print view View original

751–760 of 1001 posts

Re: Pause Giant AI Experiments: An Open Letter

#751

Post by zone411 »

NhanH wrote: post #747
JoshuaDavid wrote: post #745

Earlier quoted context omitted.

https://openai.com/blog/chatgpt-plugins

I'm gonna request more explanations and proof, or at least theoretical path on using Expedia, Zapier, Instacart, Kayak etc. to dominate the world and kill every single human on earth.

Not endorsing the arguments either way but let's say DNA printing (https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-no...) or something like Stuxnet or crashing a nuclear-power country's stock market or currency through trading while making trades appear to come from another country or by causing bank runs through hacking social media or something like WhatsApp or through deep fakes or by having human helpers do stuff for the AI voluntarily in order to get very rich...

Re: Pause Giant AI Experiments: An Open Letter

#752

Post by janalsncm »

I might be on board with this if the things they hope to accomplish in 6 months were tractable, let alone in such a short amount of time. The concepts in the letter are basically ill-defined puffery.

For example, take the word “safe”. The letter suggests making AI that is “safe”. Great idea, but what does it mean? Safe at all costs? Safe within certain bounds? Who gets to decide? What do I do if I disagree? They should probably start asking people tomorrow if they hope to get a consensus on the goal of safety, let alone realize it. Needless to say, no such consensus exists.

Re: Pause Giant AI Experiments: An Open Letter

#753

Post by arpowers »

dmz73 wrote: post #737

Best response to the current "AI" fad driven fear I've seen so far (not my words): These AI tools cannot do things. They create text (or images or code or what-have-you) in response to prompts. And that's it! It is impressive, and it is clearly passing the Turing Test to some degree, because people are confusing the apparent intelligence behind these outputs with a combination of actual intelligence and "will." Not o…

Honestly, you haven’t thought this through deeply enough.

Bad actors can actually do a ton w ai. Hacking is a breeze. I could train models to hack at 10k the efficiency of the worlds best.

I could go on… every process that can’t scale cuz manual, has been invalidated

Re: Pause Giant AI Experiments: An Open Letter

#754

Post by Jensson »

arpowers wrote: post #753
dmz73 wrote: post #737

Best response to the current "AI" fad driven fear I've seen so far (not my words): These AI tools cannot do things. They create text (or images or code or what-have-you) in response to prompts. And that's it! It is impressive, and it is clearly passing the Turing Test to some degree, because people are confusing the apparent intelligence behind these outputs with a combination of actual intelligence and "will." Not o…

Honestly, you haven’t thought this through deeply enough. Bad actors can actually do a ton w ai. Hacking is a breeze. I could train models to hack at 10k the efficiency of the worlds best. I could go on… every process that can’t scale cuz manual, has been invalidated

> I could train models to hack at 10k the efficiency of the worlds best.

What?

Re: Pause Giant AI Experiments: An Open Letter

#755

Post by JoshuaDavid »

ImaCake wrote: post #739
lukeplato wrote: post #709

Earlier quoted context omitted.

This take is a little outdated. Many people actively working on this technology no longer view it this simplistically. ex: https://twitter.com/bio_bootloader/status/164051244495839641...

There is a big difference between extending the chatbot explanation from "compressed word guesser" to "recursive compressed word guesser" and the argument being made here that it "understands" context. The former can be explained with the concept of attention etc, the latter is nebulous and would presumably require a lot more stuff. The 2 + 2 = 5 stuff is just an illustration that LLMs are fairly simple probability m…

> The 2 + 2 = 5 stuff is just an illustration that LLMs are fairly simple probability models and not, yet, understanding anything.

See for example this research paper[1]. The researchers trained the model on sequences of moves in the game of Othello[2]. The model started with no knowledge of the game, and was fed a bunch of move sequences (e.g. "c4 c3 d3 e3"). The researchers then were able to look at the model activations and figure out what it thought the board state was. When they updated those activations so that they represented a different board state, the model made moves that made sense with the altered board state but not the original board state.

See also this post[3], which demonstrates that not only does that language model have an internal model of the board state, that internal model is pretty simple. Specifically, for each square on the board, there is a dimension that corresponds to "my color" vs "opponent color" and a dimension that corresponds to whether that square is blank. Changing the activations in the corresponding directions leads to the outputs you would expect.

Recall that this model has never seen an 8x8 board, just sequences of moves. It derived an accurate model of board geometry and the rules from that data. If that doesn't count as "understanding" I'm not sure what would.

[1] https://arxiv.org/pdf/2210.13382.pdf

[2] https://en.wikipedia.org/wiki/Reversi

[3] https://www.alignmentforum.org/posts/nmxzr2zsjNtjaHh7x/actua...

Re: Pause Giant AI Experiments: An Open Letter

#756

Post by sagarpatil »

I see their point and I can tell it’s coming from a good place but I don’t agree with it. My life changed when ChatGPT was introduced, as an IT engineer who couldn’t code, using ChatGPT, I was able to increase my productivity 10x. Everyone will have to adapt to these changes, it’s like evolution (adapt or die). I don’t believe in artificially stopping the progress.

Re: Pause Giant AI Experiments: An Open Letter

#757

Post by joefourier »

dmz73 wrote: post #737

Best response to the current "AI" fad driven fear I've seen so far (not my words): These AI tools cannot do things. They create text (or images or code or what-have-you) in response to prompts. And that's it! It is impressive, and it is clearly passing the Turing Test to some degree, because people are confusing the apparent intelligence behind these outputs with a combination of actual intelligence and "will." Not o…

LLMs certainly can “will” and “do things” when provided with the right interface like LangChain: https://github.com/hwchase17/langchain

See also the ARC paper where the model was capable of recruiting and convincing a TaskRabbit worker to solve captchas.

I think many people make the mistake to see raw LLMs as some sort of singular entity when in fact, they’re more like a simulation of a text based “world” (with multimodal models adding images and other data). The LLM itself isn’t an agent and doesn’t “will” anything, but it can simulate entities that definitely behave as if they do. Fine-tuning and RLHF can somewhat force it into a consistent role, but it’s not perfect as evidenced by the multitude of ChatGPT and Bing jailbreaks.

Re: Pause Giant AI Experiments: An Open Letter

#758

Post by AbrahamParangi »

nopinsight wrote: post #695
xianshou wrote: post #355

Two-party iterated prisoner's dilemma is hard enough. Sensible players will coordinate with something like tit-for-tat, but that only works when both parties start off on the right foot. Regardless of initial strategy, the chances of degenerating towards the mutual-defection Nash equilibrium increase with the number of parties. The only prior example of world coordination at this level would be nuclear disarmament ac…

A Manhattan project on AI Alignment, if started now, might still succeed in time. Therefore, the compliance between parties needs not be long-term, which is indeed unlikely to happen. China, which is the country outside the west with the highest (engineering) capability to train something more powerful than GPT-4, is very concerned about domestic stability and they also do not want an easily replicable alien tool wit…

This analysis is completely fact-free.

"A Manhattan project on AI Alignment, if started now, might still succeed in time. Therefore, the compliance between parties needs not be long-term, which is indeed unlikely to happen."

On what grounds do you base this? You have 3 hypotheticals stacked one on top of the other:

1) AI Alignment is possible

2) AI Alignment is a specific project that may be accomplished before [bad thing happens] if we start now

3) Solving AI Alignment is an actual problem and not just dumb extrapolation from science fiction

Each of these things is totally improbable and the joint probability of them is so astronomically low that you should reconsider your position.

Re: Pause Giant AI Experiments: An Open Letter

#759

Post by nagonago »

dmz73 wrote: post #737

Best response to the current "AI" fad driven fear I've seen so far (not my words): These AI tools cannot do things. They create text (or images or code or what-have-you) in response to prompts. And that's it! It is impressive, and it is clearly passing the Turing Test to some degree, because people are confusing the apparent intelligence behind these outputs with a combination of actual intelligence and "will." Not o…

Yes. The real danger of AI tools is people overestimating them, not underestimating them. We are not in danger of AI developing intelligence, we are in danger of humans putting them in charge of making decisions they really shouldn't be making.

We already have real-world examples of this, such as algorithms erroneously detecting welfare fraud.[0][1]

The "pause" idea is both unrealistic and unhelpful. It would be better to educate people on the limitations of AI tools and not let governments put them in charge of important decisions.

[0] https://archive.is/ZbgRw [1] https://archive.is/bikFx

Re: Pause Giant AI Experiments: An Open Letter

#760

Post by Uhhrrr »

KidComputer wrote: post #495
lend000 wrote: post #481

If intelligence is all you need to dominate the world, why do some of the most powerful world leaders seem to not be more than a standard deviation above average intelligence (or at least they were before they became geriatric)? Add in the fact that GPT-4 does not "think" very fast, has no arms or legs, needs to run on expensive specialized hardware, and essentially seems like a search algorithm that lacks creativity…

Hey GPT-5, write the code implementing a bioinformatics workflow to design a novel viral RNA sequence to maximize the extermination of human life. The virus genome should be optimized for R-naught and mortality. Perform a literature search to determine the most effective human cellular targets to run the pipeline on. Use off the shelf publicly available state-of-the-art sequence to structure models and protein free-e…

An LLM will happily hallucinate a plausible-looking answer for you, with correct spelling and grammar.
Post reply on HN