Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Safe Superintelligence Inc.

ssi.inc

Post reply on HN
Print view View original

231–240 of 1001 posts

Re: Safe Superintelligence Inc.

#231

Post by thih9 »

brcmthrowaway wrote: post #218
nuz wrote: post #211

Quite impressive how many AI companies Daniel Gross has had a hand in lately. Carmack, this, lots of other promising companies. I expect him to be quite a big player once some of these pays off in 10 years or so.

What's Carmack?

> John Carmack, the game developer who co-founded id Software and served as Oculus’s CTO, is working on a new venture — and has already attracted capital from some big names.

> Carmack said Friday his new artificial general intelligence startup, called Keen Technologies (perhaps a reference to id’s “Commander Keen“), has raised $20 million in a financing round from former GitHub CEO Nat Friedman and Cue founder Daniel Gross.

https://techcrunch.com/2022/08/19/john-carmack-agi-keen-rais...

Re: Safe Superintelligence Inc.

#232

Post by gavin_gee »

insane_dreamer wrote: post #66

I understand the concern that a "superintelligence" will emerge that will escape its bounds and threaten humanity. That is a risk. My bigger, and more pressing worry, is that a "superintelligence" will emerge that does not escape its bounds, and the question will be which humans control it. Look no further than history to see what happens when humans acquire great power. The "cold war" nuclear arms race, which brough…

This.

Every nation-state will be in the game. Private enterprise will be in the game. Bitcoin-funded individuals will be in the game. Criminal enterprises will be in the game.

How does one company building a safe version stop that?

If I have access to hardware and data how does a safety layer get enforced? Regulations are for organizations that care about public perception, the law, and stock prices. Criminals and nation-states are not affected by these things

It seems to me enforcement is likely only possible at the hardware layer, which means the safety mechanisms need to be enforced throughout the hardware supply chain for training or inference. You don't think the Chinese government or US government will ignore this if its in their interest?

Re: Safe Superintelligence Inc.

#233

Post by raydev »

kccqzy wrote: post #123
PheonixPharts wrote: post #89

Earlier quoted context omitted.

> compensation packages that truly gifted AI researchers can make now I guess it depends on your definition of "truly gifted" but, working in this space, I've found that there is very little correlation between comp and quality of AI research. There's absolutely some brilliant people working for big names and making serious money, there's also plenty of really talented people working for smaller startups doing incred…

Two people I knew recently left Google to join OpenAI. They were solid L5 engineers on the verge of being promoted to L6, and their TC is now $900k. And they are not even doing AI research, just general backend infra. You don't need to be gifted, just good. And of course I can't really fault them for joining a company for the purpose of optimizing TC.

> their TC is now $900k

As a community we should stop throwing numbers around like this when more than half of this number is speculative. You shouldn't be able to count it as "total compensation" unless you are compensated.

Re: Safe Superintelligence Inc.

#234

Post by mdp2021 »

dougb5 wrote: post #125

> Building safe superintelligence (SSI) is the most important technical problem of our time. Call me a cranky old man but the superlatives in these sorts of announcements really annoy me. I want to ask: Have you surveyed every problem in the world? Are you aware of how much suffering there is outside of your office and how unresponsive it has been so far to improvements in artificial intelligence? Are you really sayi…

It says «technical» problem, and probably implies that other technical problems could dramatically benefit from such achievement.

Re: Safe Superintelligence Inc.

#235

Post by wizzwizz4 »

ofou wrote: post #130
wizzwizz4 wrote: post #97

Earlier quoted context omitted.

Backprop is neither commutative nor associative.

What do you mean? There's a bunch of proof-of-concepts such as Hydra, peer-nnet, Learnae, and so on.

The Wikipedia article goes into more detail. https://en.wikipedia.org/wiki/Federated_learning

Re: Safe Superintelligence Inc.

#236

Post by TideAd »

dougb5 wrote: post #125

> Building safe superintelligence (SSI) is the most important technical problem of our time. Call me a cranky old man but the superlatives in these sorts of announcements really annoy me. I want to ask: Have you surveyed every problem in the world? Are you aware of how much suffering there is outside of your office and how unresponsive it has been so far to improvements in artificial intelligence? Are you really sayi…

Yes, they see it as the top problem, by a large margin.

If you do a lot of research about the alignment problem you will see why they think that. In short it's "extremely high destructive power" + "requires us to solve 20+ difficult problems or the first superintelligence will wreck us"

Re: Safe Superintelligence Inc.

#237

Post by yowlingcat »

deegles wrote: post #225
TaupeRanger wrote: post #181

Earlier quoted context omitted.

Trying to create "safe superintelligence" before creating anything remotely resembling or approaching "superintelligence" is like trying to create "safe Dyson sphere energy transport" before creating a Dyson Sphere. And the hubris is just a cringe inducing bonus.

'Fearing a rise of killer robots is like worrying about overpopulation on Mars.' - Andrew Ng

This might have to bump out "AI is no match for HI (human idiocy)" as the pithy grumpy old man quote I trot out when I hear irrational exuberance about AI these days.

Re: Safe Superintelligence Inc.

#238

Post by klyrs »

auggierose wrote: post #207
PheonixPharts wrote: post #89

Earlier quoted context omitted.

> compensation packages that truly gifted AI researchers can make now I guess it depends on your definition of "truly gifted" but, working in this space, I've found that there is very little correlation between comp and quality of AI research. There's absolutely some brilliant people working for big names and making serious money, there's also plenty of really talented people working for smaller startups doing incred…

TC optimization being tail call optimization?

You don't get to that level by thinking about code...

Re: Safe Superintelligence Inc.

#239

Post by mdp2021 »

wffurr wrote: post #186
dougb5 wrote: post #125

> Building safe superintelligence (SSI) is the most important technical problem of our time. Call me a cranky old man but the superlatives in these sorts of announcements really annoy me. I want to ask: Have you surveyed every problem in the world? Are you aware of how much suffering there is outside of your office and how unresponsive it has been so far to improvements in artificial intelligence? Are you really sayi…

I think the idea is that a safe super intelligence would help solve those problems. I am skeptical because the vast majority are social coordination problems, and I don’t see how a machine intelligence no matter how smart can help with that.

> I am skeptical because the vast majority are social coordination problems, and I don’t see how

Leadership.

Re: Safe Superintelligence Inc.

#240

Post by moralestapia »

TaupeRanger wrote: post #181
dougb5 wrote: post #125

> Building safe superintelligence (SSI) is the most important technical problem of our time. Call me a cranky old man but the superlatives in these sorts of announcements really annoy me. I want to ask: Have you surveyed every problem in the world? Are you aware of how much suffering there is outside of your office and how unresponsive it has been so far to improvements in artificial intelligence? Are you really sayi…

Trying to create "safe superintelligence" before creating anything remotely resembling or approaching "superintelligence" is like trying to create "safe Dyson sphere energy transport" before creating a Dyson Sphere. And the hubris is just a cringe inducing bonus.

It would be akin to creating a "safe Dyson sphere", though; that's all it is.

If your hypothetical Dyson sphere (WIP) has a big chance to bring a lot of harm, why build it in the first place?

I think the whole safety proposal should be thought of from that point of view. "How do we make more beneficial than detrimental for humans?"

Congrats, Ilya. Eager to see what comes out of SSI.

Post reply on HN