One day we will look back on this talk as a high water mark of the AI religion craze. This whole AGI discourse that OpenAI/Altman are evangelizing is like a giant skyscraper they are trying to build on a foundation of quicksand.
1. The foundational issue is not even that AGI "does not yet exist, with even AI's top researchers far from clear about when it might". It's way worse than that. There is a strong argument made by one of the grandfathers of AI research that AGI cannot exist, at least in the sense of common sense intelligence as attributed to humans. (see Winograd "Understanding Computers & Cognition" 1985). I was first introduced to these ideas taking a class from Winograd in undergrad.
Winograd asks why we attribute mind properties to computers but not to, say, clocks. The dominant view of mind assumes that cognition is based on systematic manipulation of representations, but there is another, non-representational way of looking at it as a form of "structural coupling" between a living organism and its environment. "The cognitive domain deals with the relevance of the changing structure of the system to behavior that is effective for its survival."
I won't try to summarize a book-length argument in a few paragraphs. I just want to point out that this whole AGI conversation rests on a premise that has been seriously challenged.
The fact that Altman can get away with saying stuff like "Once we build a generally intelligent system... we will ask it to figure out a way to make an investment return" is an indication of just how insane the mainstream AI discussion has gotten. At this point it sounds like straight-up religion being prophesied from on high.
2. The whole "capped profit" positioning at 100x return is absurd, as the author points out. Altman's argument for why it makes sense involves invoking the possibility that the AGI opportunity is so incomprehensibly enormous that if OpenAI manages to crack this particular nut, it could “maybe capture the light cone of all future value in the universe". Repent, ye sinners, for the kingdom of heaven is at hand!
3. Most troubling, perhaps, is OpenAI's transparent ploy to attempt to generate buzz and take the ethical high ground with their alarmist PR strategy. Altman's justification for OpenAI's fear-mongering, which I'll paraphrase as "look at what happened with Facebook", just doesn't hold up to scrutiny. To begin with, Facebook was a real product from day one; AGI is currently a fantasy.
But there's a deeper problem with invoking Facebook. The lesson to be learned from Facebook's failure is that the real danger with tech isn't algorithms but the people that design them. Algorithms have no agency. They just do what they're supposed to do. But hiding behind the algorithm seems to be the preferred way for tech oligarchs to avoid taking responsibility for the problems they created.
The reason why I'm so troubled by OpenAI sounding the alarm bells about destructive AGI is that they are shifting the discussion away from the real threat: people. Especially people in power with virtually unlimited technological power and massive blind spots about the consequences of their actions. Give the algorithms a break!