Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Sam Altman’s leap of faith

techcrunch.com

Post reply on HN
Print view View original

41–50 of 166 posts

Re: Sam Altman’s leap of faith

#41

Post by throwawaymath »

kristianov wrote: post #34
notahacker wrote: post #8

Earlier quoted context omitted.

Has the same issue as Pascal's. Competing AGI projects (gods) exist and their believers might be the ones reaping the infinite rewards, not to mention the distinct possibility the AGI (god) doesn't actually see rewarding believers as its highest priority, and might choose to share its infinite rewards with people who aren't part of the inner circle, or even punish people who joined inner circles with the express inte…

Agreed. It seems extremely naive that they believe they can control the power of AGI, plus it is borderline ridiculous that their goal for AGI is to capture economic value.

I'm skeptical of OpenAI, but to be charitable I don't think economic value is the primary goal for their AGI. Their position seems to be that it needs to be one of several necessary goals in order to repay investors for helping them scale up to achieve it in the first place.

Re: Sam Altman’s leap of faith

#42

Post by seanmcdirmid »

arugulum wrote: post #5

Allow me to present Altman's wager: - If OpenAI does not achieve AGI, and you invested in it, you lose some finite money (or not, depending on the value of their other R&D) - If OpenAI does not achieve AGI, and you did not invest in it, you saved some finite money, which you could invest elsewhere for finite returns - If OpenAI achieves AGI and you invested in it, you get infinite returns, because AGI will capture al…

What if AGI really leads to Skynet and the downfall of humanity? Then you’ve realized infinite negative returns in a success condition.

Not that I’m saying it will, but Pascal’s wager is known to be a huge fallacy.

Re: Sam Altman’s leap of faith

#43

Post by avian »

sgt101 wrote: post #22
Havoc wrote: post #4

>‘Once we build a generally intelligent system, that basically we will ask it to figure out a way to make an investment return for you.' Maybe lets do curing cancer first?

I doubt that such a thing will be interested either in creating investment returns or in curing cancer. It might be interested in finding others like itself, or it might be interested in making companions, or it might be interested in some other grand projects that we don't understand, but our concerns are likely to be about as relevant as a three year old's career advice.

Why? It seems a very human-centric way to think. Such an entity wouldn't have ancestors that had to keep in groups to survive, so why would it be interested in making companions?

AI converting the mass of the solar system into stacks of $100 bills for its investors seems like a much more likely outcome.

Re: Sam Altman’s leap of faith

#44

Post by lmeyerov »

graycat wrote: post #14

There are a lot of research labs and institutes around, in universities and outside, with funding from NSF, NIH, foundations, wealthy individuals, etc. So, if Altman wants to set up a research institute, okay -- that alone is not very novel. It is obvious from history that good research is super tough to do. My view has been: We look at the research and mostly all we see is junk think. Then we see that, actually, res…

From a business view, the big innovation of OpenAI is marketing an industrial R&D lab as something good CS people will join and investors will fund.

The calculus is more like DeepMind: Can they keep attracting top talent, can the top talent ever do something the org structure can execute on commercially, and maaaaybe, in likely worst case, can they recoup big losses via aquihire, and the responsible investors look like they were in good company if they were wrong.

From that lens. OpenAI... yet in reality mostly closed. Non-profit... But really VC model. Peer review may sometimes happen, but the perceived quality and awareness is from a top content marketing team and even ex journalists. No immediate commercial path beyond selling for talent, but by merely employing Sam, investors feel like he can always pivot the co to make money in the case of a down round.

DeepMind did something similar yet without the marketing skill. OpenAI is doing it even better by, for now, removing the pressure for commercialization.

As someone coming from both R&D and enterprise data startups, I get two conflicting emotions. I'm sad that almost all top tier scientists don't get such outreach and funding help. On the otherhand, the industry has not been able to repeat Bell Labs (widescale R&D that commercialized) for decades so OpenAI's continued ability to draw R&D funding without expectation of ROI in any timeline is cool.

Re: Sam Altman’s leap of faith

#45

Post by throwawaymath »

alehul wrote: post #27
nickpsecurity wrote: post #6

Earlier quoted context omitted.

You left off the assumptions that AGI will be a superintelligence and one capable of capturing all value. There's no evidence to believe either one. Instead, it will probably be pretty dumb like humans are without the decades of supervised training we get and our interactions with the world. Further, the second a smart one starts getting devastating results, there will be political push-back to do something about it.…

> Further, the second a smart one starts getting devastating results, there will be political push-back to do something about it. People might cast votes to make rules about them that limit their power or rate of development. My fear of AGI is that it will not be able to be stopped once deployed, as you're saying. It's irreversible. It will know that we want to shut it down, and so it will be able to copy itself onto…

> think about the hacking capabilities of AGI for a moment

We're discussing the hypothetical skills and capabilities of a thing which is fundamentally science fiction. The rules are treated as arbitrary.

I don't see a priori why an AGI would be intrinsically good at hacking, or even why it would be capable of exponentially improving itself.

This is the problem I see with any discussion of AGI. The game's rules don't matter, so we can define whichever properties we'd like for the sake of argument. There's no skin in the game to counteract that, because we have no conception of what AGI will actually be like - nor if it's even possible.

As it stands, in your comment and the rest of this thread I see a variety of leaps and jumps to scenarios which seem completely undefended.

Re: Sam Altman’s leap of faith

#46

Post by idlewords »

arugulum wrote: post #5

Allow me to present Altman's wager: - If OpenAI does not achieve AGI, and you invested in it, you lose some finite money (or not, depending on the value of their other R&D) - If OpenAI does not achieve AGI, and you did not invest in it, you saved some finite money, which you could invest elsewhere for finite returns - If OpenAI achieves AGI and you invested in it, you get infinite returns, because AGI will capture al…

He's capped return at 100x, though.

Re: Sam Altman’s leap of faith

#47

Post by bryanrasmussen »

Waterluvian wrote: post #24
bryanrasmussen wrote: post #21

Earlier quoted context omitted.

Was Edison a great scientist? Not sure if Tesla was wholly in it for the money, if so he sure made some missteps in getting paid.

I don't want to get into a discussion on the semantics of scientist vs. engineer vs. inventor or what defines greatness. My point is that parent comment spoke very conclusively that leaps of progress are NEVER because of monetary gain and that's trivially refutable.

fair enough.

Re: Sam Altman’s leap of faith

#48

Post by idlewords »

There is too much easy money sloshing around Silicon Valley. The normal mechanisms for allocating it to sane, productive use (market forces) don't work, because it's in too few hands. So instead we get our version of a planned economy run by people who scared themselves with scifi.

Re: Sam Altman’s leap of faith

#49

Post by lisper »

roenxi wrote: post #39
notahacker wrote: post #8

Earlier quoted context omitted.

Has the same issue as Pascal's. Competing AGI projects (gods) exist and their believers might be the ones reaping the infinite rewards, not to mention the distinct possibility the AGI (god) doesn't actually see rewarding believers as its highest priority, and might choose to share its infinite rewards with people who aren't part of the inner circle, or even punish people who joined inner circles with the express inte…

It also has an issue not present in Pascal's, which is you are multiplying by infinity in a context where it doesn't actually make sense. The payoff from AGI may be incalculable, but it isn't infinite both in itself and in Altman's ability to enjoy the rewards it promises. Once the value becomes finite, a whole heap of risk-reward logic kicks in that the Wager wants to sweep under the rug. As a concrete example, foll…

Two other possibilities that Altman's wager as presented does not take into account:

1. The government could change the rules (i.e. redistribute wealth) in response to the new economic reality brought about by AGI

2. The AGI may decide to retain the economic value it captures for itself

Re: Sam Altman’s leap of faith

#50

Post by mcguire »

roenxi wrote: post #39
notahacker wrote: post #8

Earlier quoted context omitted.

Has the same issue as Pascal's. Competing AGI projects (gods) exist and their believers might be the ones reaping the infinite rewards, not to mention the distinct possibility the AGI (god) doesn't actually see rewarding believers as its highest priority, and might choose to share its infinite rewards with people who aren't part of the inner circle, or even punish people who joined inner circles with the express inte…

It also has an issue not present in Pascal's, which is you are multiplying by infinity in a context where it doesn't actually make sense. The payoff from AGI may be incalculable, but it isn't infinite both in itself and in Altman's ability to enjoy the rewards it promises. Once the value becomes finite, a whole heap of risk-reward logic kicks in that the Wager wants to sweep under the rug. As a concrete example, foll…

According to the true believers, the payoff for AGI is infinite because the superintelligence will be capable of literally anything (or at least simulating it well enough that it doesn't matter). To them, it is Pascal's wager.

Religious beliefs can be weird.

Post reply on HN