Pause Giant AI Experiments: An Open Letter • Hacker News Board

Re: Pause Giant AI Experiments: An Open Letter

#931

Post by jacquesm » Thu, Mar 30, 2023, 2:36 PM UTC

dekhn wrote: post #563

lend000 wrote: post #511

Earlier quoted context omitted.

I've been trying to use GPT-4 for my hard science startup, and it really has nothing to offer when you push the boundaries of what has been done by even a little, but it's great for speeding up coding. Once we do have an AI capable of extraordinary innovation (hopefully in 10 years! But probably a lot longer), it will be obvious, and it will unfortunately be removed from the hands of the plebs based on fearmongering…

My personal threshold for AGI is literally: discover something new and significant in science (preferably biology) that is almost certainly true by describing an experiment that could be replicated by a large number of scientists and whose interpretation is unambiguous. For example, the Hershey/Chase and Avery/McCleod experiments convinced the entire biological community that DNA, not protein, was almost certainly th…

That places the goalposts outside of the field though. A decade ago what we are seeing today would have been SF, much less AI. And now that it's reality it isn't even AI anymore but just 'luxury autocomplete' in spite of the massive impact that is already having.

If we get to where you are pointing then we will have passed over a massive gap between today and then, and we're not necessarily that far away from that in time (but still in capabilities).

But likely if and when that time comes everybody that holds this kind of position will move to yet a higher level of attainment required before they'll call it truly intelligent.

So AGI vs AI may not really matter all that much: impact is what matters and impact we already have aplenty.

Re: Pause Giant AI Experiments: An Open Letter

#932

Post by crypot » Thu, Mar 30, 2023, 2:45 PM UTC

Cardinal7167 wrote: post #926

crypot wrote: post #922

Earlier quoted context omitted.

This is silly, unless you think the earth will turn into Venus. The majority of time earth has had no ice caps. Primates evolved when the polars where covered in trees, not ice. A warmer planet will support more biodiversity not less. https://www.climate.gov/news-features/climate-qa/whats-hotte...

It’s not about the lack of ice, dingus. the danger is in climate change shocking ecosystems such that they trigger a systemic collapse that we can’t contain or manage. there’s basically no timeline where food supply wont be impacted and I’d like to hear your cavalier attitude about climate change then.

There is zero evidence that a warming planet would case a systemic collapse in biodiversity. In fact, 500 millions of planetary history shows the opposite. As the planet warms, biodiversity increases. There is nothing to argue about here.

Re: Pause Giant AI Experiments: An Open Letter

#933

Post by koreanguy » Thu, Mar 30, 2023, 2:46 PM UTC

[dead]

Re: Pause Giant AI Experiments: An Open Letter

#934

Post by machiaweliczny » Thu, Mar 30, 2023, 2:50 PM UTC

scarmig wrote: post #559

lend000 wrote: post #481

If intelligence is all you need to dominate the world, why do some of the most powerful world leaders seem to not be more than a standard deviation above average intelligence (or at least they were before they became geriatric)? Add in the fact that GPT-4 does not "think" very fast, has no arms or legs, needs to run on expensive specialized hardware, and essentially seems like a search algorithm that lacks creativity…

Intelligence isn't everything; there's also effort, skills, and hours in the day. But AI (by supposition, for now) can outcompete humans on all four and would have fewer informational bottlenecks. Imagine a big company or government; they mostly nowadays work as an entity by passing around bits of data (either literally or with more traditional communication that can represented as bits of data). Now imagine you repl…

Yep, it’s probably some time before AI surpasses collective IQ of humanity but it will outcompete us in collaboration and resource needs and procreation

Re: Pause Giant AI Experiments: An Open Letter

#935

Post by machiaweliczny » Thu, Mar 30, 2023, 2:57 PM UTC

Barrin92 wrote: post #554

biscottigelato wrote: post #496

Earlier quoted context omitted.

Because the ability to influence others is more important than 1 or 2 standard deviation in more intelligence. However if we are talking about 10x more intelligence, then that'd be a whole different ball game. Check Alpha Go -> Alpha Go Zero. Gaining orders of magnitude in capability only with slight change to the model, along with a lot of more resource (we can easily throw 100x more compute and data to these models…

>However if we are talking about 10x more intelligence, then that'd be a whole different ball game. Research that the US military has conducted has shown that the ability to influence others declines if the intelligence gap between leaders and subjects is too large. Stephen Hawking wouldn't have been a very good influencer of 80 IQ guys compared to Andrew Tate. 1 standard deviation in intelligence is actually just ab…

Good point and I agree but regarding fitness:

  \* communication
  \* resource usage
  \* procreation
  \* embodiment

I think that digital agents posses very big fitness like real life viruses or malware

Re: Pause Giant AI Experiments: An Open Letter

#936

Post by ben_w » Thu, Mar 30, 2023, 3:00 PM UTC

bambax wrote: post #813

xianshou wrote: post #355

Two-party iterated prisoner's dilemma is hard enough. Sensible players will coordinate with something like tit-for-tat, but that only works when both parties start off on the right foot. Regardless of initial strategy, the chances of degenerating towards the mutual-defection Nash equilibrium increase with the number of parties. The only prior example of world coordination at this level would be nuclear disarmament ac…

> Climate change mitigation, which more closely resembles AI safety in both complexity and (lack of) barriers to entry, has been sporadic, inconsistent, and only enacted to the extent... Climate change mitigation is the perfect example. Nobody is doing anything, nobody seems to care, everyone cheats with ridiculous carbon credits or carbon offset vouchers made out of thin air, etc. It's likely the planet will become…

> Nobody is doing anything, nobody seems to care, everyone cheats

While there is indeed a lot of cheating, there's also over a terawatt (peak) of PV capacity.

It's a long way from "nothing" or "not caring".

> It's likely the planet will become hostile to (human) life long before AI will be able to do us any harm.

Depends what you mean by hostile — most of the planet already was before the industrial revolution, some will get moreso over the next few decades, but it's not a boolean.

And harm? Again, not really a boolean.

GOFAI-era had Therac-25, but if you don't like to count that and it has to be a neural net, then how about the algorithmic promotion of misinformation on (not necessarily by) YouTube, Facebook, and Google? Deaths from imperfect self driving cars? Discovery of novel deadly neurotoxins by, essentially, flipping an evil bit, which got in the news this time last year?: https://www.nature.com/articles/s42256-022-00465-9

Re: Pause Giant AI Experiments: An Open Letter

#937

Post by thomastjeffery » Thu, Mar 30, 2023, 3:12 PM UTC

sebzim4500 wrote: post #610

thomastjeffery wrote: post #302

The dumb criticize the blind. What an absurd situation! How did we get here? Here are the steps: 1. Large Language Models have been presented as "AI", which personifies them instead of describing how they work. 2. Goals for LLM development were set for the personified attributes, and not the actual functionally of the real thing. OpenAI brags about how GPT4 scores at human tests: as if that has any bearing on the mod…

This is the first time I have heard Yoshua Bengio described as "dumb". According to the Oxford English Dictionary, the definition of AI is: >the theory and development of computer systems able to perform tasks normally requiring human intelligence, such as visual perception, speech recognition, decision-making, and translation between languages. Clearly GPT-3 fits that. You are welcome to have your own definition, bu…

> This is the first time I have heard Yoshua Bengio described as "dumb".

I was being cheeky: I meant it to mean "unable to speak". By feeding the personification narrative into itself, nothing is truly being said about what the model itself is and does: only the results of using it. Anyone who has based their understanding on that personified narrative is effectively "blind" to the true nature of the tech itself.

I admit I was also being rude, and that was intentional. To spend so much effort studying a thing, and base all of your conclusions on its output...I think it's absurd to then call oneself an expert. Expert of what? A black box you never even opened? The LLM itself can only perform inference, but we aren't held back by that limitation!

> the theory and development of computer systems able to perform tasks normally requiring human intelligence, such as visual perception, speech recognition, decision-making, and translation between languages.

That is an excellent definition for the field of science that is AI. "An AI technology" "Working in AI" These statements use "AI" as a category of pursuit.

It's not a great definition for an AI. Those words already define themselves. Artificial Intelligence: That means it was created intentionally, and it thinks for itself. No one needs to search a dictionary to know what "an AI" is.

If "Intelligence" is an adjective, the definition you provided fits. If "Intelligence" is the noun, then it better be "an intelligence".

> Yeah, evaluating language models is extremely difficult. Hence OpenAI Evals, etc.

I wasn't trying to say that it's wrong to evaluate what an LLM does when prompted with the text from a test. I think that can provide valuable data and insights.

The score we would give to a human taking the test is not valuable data. The behavior being measured does not align to any part of the LLM's process.

We don't have to approach this blindly. We can have an explicit understanding to base research on. That doesn't mean making assumptions about the results, but it does mean better understanding the process that leads to them.

>> 4. An LLM is understood to be a person in a black box, and we could not possibly understand what that person will do next. Danger!

> No serious person believes this.

Not literally, but narratively many do.

Re: Pause Giant AI Experiments: An Open Letter

#938

Post by TurkishPoptart » Thu, Mar 30, 2023, 3:15 PM UTC

bambax wrote: post #813

xianshou wrote: post #355

Two-party iterated prisoner's dilemma is hard enough. Sensible players will coordinate with something like tit-for-tat, but that only works when both parties start off on the right foot. Regardless of initial strategy, the chances of degenerating towards the mutual-defection Nash equilibrium increase with the number of parties. The only prior example of world coordination at this level would be nuclear disarmament ac…

> Climate change mitigation, which more closely resembles AI safety in both complexity and (lack of) barriers to entry, has been sporadic, inconsistent, and only enacted to the extent... Climate change mitigation is the perfect example. Nobody is doing anything, nobody seems to care, everyone cheats with ridiculous carbon credits or carbon offset vouchers made out of thin air, etc. It's likely the planet will become…

As long as we are economically dependent on China to produce tons of plastic/disposable/etc consumer products, everything the US/west does it "combat climate change" is more or less in vain. We're just throwing our trash over the wall.

Re: Pause Giant AI Experiments: An Open Letter

#939

Post by soiler » Thu, Mar 30, 2023, 3:22 PM UTC

worldsayshi wrote: post #512

richardw wrote: post #444

Earlier quoted context omitted.

And add in just one peer-level war where one side has their back against the wall. Then give it 100 years where anyone can create such a model on their phone. We’d need a constantly evolving inoculation function to compete. And it would probably lose because the other side has fewer restrictions. In my darker thoughts about this, this is why we see no aliens. To get this to work we need a far smarter entity with no p…

> In my darker thoughts about this, this is why we see no aliens. If AI would be a common great filter we'd expect at least one of them to expand outwards after being the filter?

Reasonable, but not necessarily true.

1. We don't understand what the motivations of our own AI are, let alone "typical" alien AI

2. Expanding AI might be better at and/or more invested in hiding itself. It probably has no need for wasteful communications, for example.

Re: Pause Giant AI Experiments: An Open Letter

#940

Post by pjc50 » Thu, Mar 30, 2023, 3:26 PM UTC

AbrahamParangi wrote: post #758

nopinsight wrote: post #695

Earlier quoted context omitted.

A Manhattan project on AI Alignment, if started now, might still succeed in time. Therefore, the compliance between parties needs not be long-term, which is indeed unlikely to happen. China, which is the country outside the west with the highest (engineering) capability to train something more powerful than GPT-4, is very concerned about domestic stability and they also do not want an easily replicable alien tool wit…

This analysis is completely fact-free. "A Manhattan project on AI Alignment, if started now, might still succeed in time. Therefore, the compliance between parties needs not be long-term, which is indeed unlikely to happen." On what grounds do you base this? You have 3 hypotheticals stacked one on top of the other: 1) AI Alignment is possible 2) AI Alignment is a specific project that may be accomplished before [bad…

"AI alignment" is not terribly well defined, but I'd like to ask anyone with a definition how well we're doing on the "human alignment" and "corporate alignment" projects.