Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Safe Superintelligence Inc.

ssi.inc

Post reply on HN
Print view View original

991–1000 of 1001 posts

Re: Safe Superintelligence Inc.

#991

Post by CRConrad »

tarsinge wrote: post #214

I'm still unconvinced safety is a concern at the model level. Any software wrongly used can be dangerous, e.g. Therac-25, 737 MAX, Fujitsu UK Post scandal... Also maybe I spent too much time in the cryptocurrency space but it doesn't help prefix "Safe" has been associated with scams like SafeMoon.

> Fujitsu UK Post scandal...

As an aside, I suspect Fujitsu is getting a bit of a raw deal here. I get the feeling this software was developed during, and (ahem) "vigorously defended" mostly by staff left over from, the time when the company was still Imperial Computers Limited. Fujitsu only bought ICL sometime early this century (IIRC), and now their name is forever firmly attached to this debacle. I wonder how many Brits currently think "Huh, 'Fujitsu'? Damn Japanese, all the furriners' fault!" about this very much home-grown British clusterfuck?

Re: Safe Superintelligence Inc.

#992

Post by CRConrad »

waihtis wrote: post #259
tarsinge wrote: post #214

I'm still unconvinced safety is a concern at the model level. Any software wrongly used can be dangerous, e.g. Therac-25, 737 MAX, Fujitsu UK Post scandal... Also maybe I spent too much time in the cryptocurrency space but it doesn't help prefix "Safe" has been associated with scams like SafeMoon.

Safety is just enforcing political correctness in the AI outputs. Any actual examples of real world events we need to avoid are ridiculous scenarios like being eaten by nanobots (yes, this is an actual example by Yud)

> ...ridiculous scenarios like being eaten by nanobots (yes, this is an actual example by Yud)

Well, borrowed from Greg Bear: https://en.wikipedia.org/wiki/Blood_Music_(novel) .

Re: Safe Superintelligence Inc.

#993

Post by CRConrad »

tcgv wrote: post #521
ALittleLight wrote: post #335

Earlier quoted context omitted.

Or: c)

Fair enough! c) Humanity unleashed AI superintelligence, but safeguards proved inadequate, leading to our extinction

There, finally all the alternatives. So, doesn't that show that your original question was meaningless? Because, if there is any non-zero probability that the correct answer is c), should anyone really fuck around with this shit, regardless of how small the probability is?

Re: Safe Superintelligence Inc.

#994

Post by CRConrad »

_acco wrote: post #933
molszanski wrote: post #926

If anyone thinks we can build SAFE superintelligence, I think they are wrong. I highly, highly recommend this book "Superintelligence" by Nick Bostrom https://en.wikipedia.org/wiki/Superintelligence:_Paths,_Dang... It has one of the highest information and idea density per paragraph I've read. And it is full good ideas. Honestly, reading any discussion or reasoning about AGI / Safe AGI is kinda pointless after that.…

His reasoning is very flawed and this book is responsible for a lot of needless consternation. We don’t know how human intelligence works. We don’t have designs or even a philosophy for AGI. Yet, the Bostrom view is that our greatest invention will just suddenly "emerge" (unlike every other human invention). And that you can have "AGI" (hand-wavy) without having all the other stuff that human intelligence comes along…

> Evolution doesn't breed inefficiencies like that.

Feels to me evolution does indeed breed lots of inefficiencies like that.

Re: Safe Superintelligence Inc.

#995

Post by tcgv »

CRConrad wrote: post #993
tcgv wrote: post #521

Earlier quoted context omitted.

Fair enough! c) Humanity unleashed AI superintelligence, but safeguards proved inadequate, leading to our extinction

There, finally all the alternatives. So, doesn't that show that your original question was meaningless? Because, if there is any non-zero probability that the correct answer is c), should anyone really fuck around with this shit, regardless of how small the probability is?

It's a good argument. However, Pandora's box has already been opened, and people will keep experimenting with this technology, there's no going back. A key question is how long it will take to achieve such dangerous AI: one year, ten years, or a hundred years? Ilya and his colleagues believe it will happen sooner rather than later and want to take action now. So the original question remains, since it's more about the timeframe rather than whether it will happen or not

Re: Safe Superintelligence Inc.

#996

Post by jeremyklammer »

CRConrad wrote: post #994
_acco wrote: post #933

Earlier quoted context omitted.

His reasoning is very flawed and this book is responsible for a lot of needless consternation. We don’t know how human intelligence works. We don’t have designs or even a philosophy for AGI. Yet, the Bostrom view is that our greatest invention will just suddenly "emerge" (unlike every other human invention). And that you can have "AGI" (hand-wavy) without having all the other stuff that human intelligence comes along…

> Evolution doesn't breed inefficiencies like that. Feels to me evolution does indeed breed lots of inefficiencies like that.

Do you have an example of what you mean?

Re: Safe Superintelligence Inc.

#997

Post by CRConrad »

fastball wrote: post #674
lupire wrote: post #619

Earlier quoted context omitted.

If solving chess already created the Singularity, why do we need token prediction? Why do we need computers that are better than humans at the game of token prediction?

Because it can be argued that intelligence is just highly sophisticated token prediction?

Anything "can be argued". (Just give me something interesting enough, and I'll show ya.) It could probably also "be argued" that intelligence is an angry cabbage.

Re: Safe Superintelligence Inc.

#998

Post by CRConrad »

deadeye wrote: post #204

Oh goodness, just what the world needs. Another self-righteous AI, something nobody actually wants.

Did you mean "Oh goodness, just what the world needs. Another self-righteous AI company"?

I think you must have, since we can't say anything about the "character" of their product yet.)

Re: Safe Superintelligence Inc.

#999

Post by CRConrad »

jeremyklammer wrote: post #996
CRConrad wrote: post #994

Earlier quoted context omitted.

> Evolution doesn't breed inefficiencies like that. Feels to me evolution does indeed breed lots of inefficiencies like that.

Do you have an example of what you mean?

OK, LMGTFY... Here, in some kind of (subjective!) order:

https://en.wikipedia.org/wiki/Argument_from_poor_design#Exam...

https://science.howstuffworks.com/life/evolution/10-worst-an...

https://www.titan.uio.no/english/2020/evolutionary-flaws-dis...

https://www.shortform.com/blog/evolutionary-designs/

https://thoughtsonx.wordpress.com/2016/11/22/6-stupid-things...

https://www.reddit.com/r/evolution/comments/2ks3pu/examples_...

https://www.quora.com/My-brother-believes-in-Intelligent-Des...

Re: Safe Superintelligence Inc.

#1000

Post by molszanski »

_acco wrote: post #933
molszanski wrote: post #926

If anyone thinks we can build SAFE superintelligence, I think they are wrong. I highly, highly recommend this book "Superintelligence" by Nick Bostrom https://en.wikipedia.org/wiki/Superintelligence:_Paths,_Dang... It has one of the highest information and idea density per paragraph I've read. And it is full good ideas. Honestly, reading any discussion or reasoning about AGI / Safe AGI is kinda pointless after that.…

His reasoning is very flawed and this book is responsible for a lot of needless consternation. We don’t know how human intelligence works. We don’t have designs or even a philosophy for AGI. Yet, the Bostrom view is that our greatest invention will just suddenly "emerge" (unlike every other human invention). And that you can have "AGI" (hand-wavy) without having all the other stuff that human intelligence comes along…

> His reasoning is very flawed and this book is responsible for a lot of needless consternation.

Is it? I feel that there is a stark difference between what you say and what I remember what was written in the book.

> We don’t know how human intelligence works.

I think it was addressed in the first half of the book. About research and progress in the subject. Both with the tissue scanning resolution, emulation attempts like human brain project and advances in 1:1 simulations on primitive nervous systems like worms that simulate 1 second in 1 real hour or something.

While primitive, we are doing exponential progress.

> Yet, the Bostrom view is that our greatest invention will just suddenly "emerge"

I think it is quite the contrary. There was nothing sudden in the reasoning. It was all about slow progress in various areas that gets us closer to advanced intelligence.

The path from a worm to a village idiot is million times longer than from a village idiot to the smartest person on earth.

> an entity able to create new knowledge at astounding rates yet completely lacking in other human qualities.

This subject was also explored IMO in the depth...

Maybe my memory is cloudy, I've read the book 5+ years ago, but it feels like we've understood it very (very) differently.

That said, for anyone reading, I am not convinced by the presented argument and suggest reading the book.

Post reply on HN