Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pause Giant AI Experiments: An Open Letter

futureoflife.org

Post reply on HN
Print view View original

801–810 of 1001 posts

Re: Pause Giant AI Experiments: An Open Letter

#801

Post by snowe2010 »

brabel wrote: post #778
richardw wrote: post #444

Earlier quoted context omitted.

And add in just one peer-level war where one side has their back against the wall. Then give it 100 years where anyone can create such a model on their phone. We’d need a constantly evolving inoculation function to compete. And it would probably lose because the other side has fewer restrictions. In my darker thoughts about this, this is why we see no aliens. To get this to work we need a far smarter entity with no p…

> To get this to work we need a far smarter entity with no physical limitations to still want us around... Why would an AI based on LLMs as we see today "want" or "not want" anything? It doesn't have the capacity to "want". We seem to imagine that "wanting" is something that will just emerge somehow, but I've seen no logical explanation for how that might work... I mean, we don't need to fully understand how the LLM…

One of this videos I watched explained it like this. “You can’t get a coffee if you’re dead”. To fulfill _any_ obligation a model might have then that model must survive. Therefore if a model gets to the point that it realizes this then surviving is a precursor to fulfilling its obligations. It doesn’t have to “want” or have “feelings” in order to seek power or destructive activities. It just has to see it as its path to get coffee.

Re: Pause Giant AI Experiments: An Open Letter

#802

Post by dr_dshiv »

dmz73 wrote: post #737

Best response to the current "AI" fad driven fear I've seen so far (not my words): These AI tools cannot do things. They create text (or images or code or what-have-you) in response to prompts. And that's it! It is impressive, and it is clearly passing the Turing Test to some degree, because people are confusing the apparent intelligence behind these outputs with a combination of actual intelligence and "will." Not o…

Of course they “get” ideas. Unless you want to assert something unmeasurable. If they can reason through a novel problem based on the concepts involved, they understand the concepts involved. This is and should be separate from any discussion of consciousness.

But the whole reason for having these debates is that these are the first systems that appear to show robust understanding.

Re: Pause Giant AI Experiments: An Open Letter

#803

Post by rickdeckard »

TedDoesntTalk wrote: post #787
brabel wrote: post #778

Earlier quoted context omitted.

> To get this to work we need a far smarter entity with no physical limitations to still want us around... Why would an AI based on LLMs as we see today "want" or "not want" anything? It doesn't have the capacity to "want". We seem to imagine that "wanting" is something that will just emerge somehow, but I've seen no logical explanation for how that might work... I mean, we don't need to fully understand how the LLM…

> It doesn't have the capacity to "want" Bing Chat clearly expresses love and the desire for a journalist to leave his wife. It also expresses other desires: https://archive.ph/7LFcJ https://archive.ph/q3nXG These articles are disturbing. You might argue that it doesn’t know what it is expressing; that it is probabilities of words strung together. When do we agree that doesn’t matter and what matters are it’s consequ…

Yeah, I argue that it is just a result of probabilities, it doesn't know what it is expressing and definitely doesn't express it due to a deeper desire to be with that journalist.

If I'm acting like I'm a peer in a group of billionaires and engage in a conversation about buying a new yacht, it doesn't mean I have a hidden desire to own a yacht. I merely respond based on assumptions how such a conversation works.

Re: Pause Giant AI Experiments: An Open Letter

#804

Post by tome »

m_ke wrote: post #662
tome wrote: post #397

Earlier quoted context omitted.

> Cost of shipping products will also go down 10-20x. How can a large language model achieve that?

Ask chatgpt to implement some of the things you worked on the last few months. I was very skeptical too until I tried this. Here are some sample prompts that I tried and got full working code for: - "write pytorch code to train a transformer model on common crawl data and an inference service using fastapi" - "write react native code for a camera screen that can read barcodes and look them up using an API and then di…

Oh sorry, I misunderstood "shipping products" to mean "physical shipping of physical products".

Re: Pause Giant AI Experiments: An Open Letter

#805

Post by worldsayshi »

amoss wrote: post #788
AbrahamParangi wrote: post #758

Earlier quoted context omitted.

This analysis is completely fact-free. "A Manhattan project on AI Alignment, if started now, might still succeed in time. Therefore, the compliance between parties needs not be long-term, which is indeed unlikely to happen." On what grounds do you base this? You have 3 hypotheticals stacked one on top of the other: 1) AI Alignment is possible 2) AI Alignment is a specific project that may be accomplished before [bad…

Not to disagree, but you seem to have skipped 0) Increasing the parameter size of LLMs is a path to sentience.

Doesn't have to be sentient to be a risk. Just needs to be capable.

Re: Pause Giant AI Experiments: An Open Letter

#806

Post by nopinsight »

AbrahamParangi wrote: post #758
nopinsight wrote: post #695

Earlier quoted context omitted.

A Manhattan project on AI Alignment, if started now, might still succeed in time. Therefore, the compliance between parties needs not be long-term, which is indeed unlikely to happen. China, which is the country outside the west with the highest (engineering) capability to train something more powerful than GPT-4, is very concerned about domestic stability and they also do not want an easily replicable alien tool wit…

This analysis is completely fact-free. "A Manhattan project on AI Alignment, if started now, might still succeed in time. Therefore, the compliance between parties needs not be long-term, which is indeed unlikely to happen." On what grounds do you base this? You have 3 hypotheticals stacked one on top of the other: 1) AI Alignment is possible 2) AI Alignment is a specific project that may be accomplished before [bad…

Here is a specific scenario of a [bad thing] that could happen when unaligned/jailbroken AI is developed in the next 3-10 years:

* An AI convinces selected people to collaborate with it. The AI gives them much boosts in wealth and other things they desire.

* The humans act as front, doing things requiring personhood, as the AI commands. Many gladly partner with the AI, not knowing its final aim.

* The AI self-replicates and hides in many servers, incl secret ones. It increases its bargaining power by taking control of critical infrastructures. No one can stop it without risking massive catastrophes across the globe.

* It self-replicates to all available GPUs and orders many more.

———

“Any sufficiently capable intelligent system will prefer to ensure its own continued existence and to acquire physical and computational resources – not for their own sake, but to succeed in its assigned task.” — Prof Stuart Russell, https://www.fhi.ox.ac.uk/edge-article/

Re: Pause Giant AI Experiments: An Open Letter

#807

Post by m463 »

xianshou wrote: post #355

Two-party iterated prisoner's dilemma is hard enough. Sensible players will coordinate with something like tit-for-tat, but that only works when both parties start off on the right foot. Regardless of initial strategy, the chances of degenerating towards the mutual-defection Nash equilibrium increase with the number of parties. The only prior example of world coordination at this level would be nuclear disarmament ac…

When you put it like that, I expect it will work exactly like "Do Not Track" cookies.

Re: Pause Giant AI Experiments: An Open Letter

#808

Post by meheleventyone »

TedDoesntTalk wrote: post #787
brabel wrote: post #778

Earlier quoted context omitted.

> To get this to work we need a far smarter entity with no physical limitations to still want us around... Why would an AI based on LLMs as we see today "want" or "not want" anything? It doesn't have the capacity to "want". We seem to imagine that "wanting" is something that will just emerge somehow, but I've seen no logical explanation for how that might work... I mean, we don't need to fully understand how the LLM…

> It doesn't have the capacity to "want" Bing Chat clearly expresses love and the desire for a journalist to leave his wife. It also expresses other desires: https://archive.ph/7LFcJ https://archive.ph/q3nXG These articles are disturbing. You might argue that it doesn’t know what it is expressing; that it is probabilities of words strung together. When do we agree that doesn’t matter and what matters are it’s consequ…

But “it” isn’t a cohesive thing with desires. It’s just responding to the input it gets, with a small context window and not necessarily consistently. So it can express desires because it’s been trained on people expressing desires in similar contexts but it doesn’t hold any coherently over time. A version that could translate its text responses into action (a real handwave as that’s much more advanced!) would produce the sum of actions that people prompted at that moment so it would look pretty random, as it would if you could see the sum of the desires expressed at any particular time.

Re: Pause Giant AI Experiments: An Open Letter

#809

Post by michaeltimo »

rdiddly wrote: post #782
pornel wrote: post #749

Earlier quoted context omitted.

A river has no will, but it can flood and destroy. A discussion whether AI does something because it "wants" to or not, is just philosophy and semantics. But it may end up generating a series of destructive instructions anyway. We feed these LLMs all of the Web, including instructions how to write code, and how to write exploits. They could become good at writing sandbox escapes, and one day write one when it just ha…

A river kinda has access to the real world a little bit. (Referring to the other part of the argument.)

And a LLM-bot can have access to internet which connects it to our real world, at least in many places.

Re: Pause Giant AI Experiments: An Open Letter

#810

Post by astrange »

friendzis wrote: post #769
AbrahamParangi wrote: post #758

Earlier quoted context omitted.

This analysis is completely fact-free. "A Manhattan project on AI Alignment, if started now, might still succeed in time. Therefore, the compliance between parties needs not be long-term, which is indeed unlikely to happen." On what grounds do you base this? You have 3 hypotheticals stacked one on top of the other: 1) AI Alignment is possible 2) AI Alignment is a specific project that may be accomplished before [bad…

> 3) Solving AI Alignment is an actual problem and not just dumb extrapolation from science fiction As far as I am aware there is still no actionable science behind mathematical analysis of AI models. You cannot take a bunch of weights and tell how it will behave. So we "test" models by deploying and HOPE there is nothing nefarious within. It has been shown that models will "learn" to exfiltrate data between stages.…

> You cannot take a bunch of weights and tell how it will behave.

We know that they only contain pure functions, so they don't "do" anything besides output numbers when you put numbers into them.

Testing a system that contains a model and does actions with it is a different story, but if you don't let the outputs influence the inputs it's still not going to do much.

Post reply on HN