"Recursive self-improvement" is in the same league as "perpetual motion".
What would be a way to recursively self-improve algorithms for matrix multiplication (foundations of machine learning and inference)?
Superintelligence: The Idea That Eats Smart People (2016)
11–20 of 253 posts
Re: Superintelligence: The Idea That Eats Smart People (2016)
#12> If you encountered a cheetah in pre-industrial times (and survived the meeting), you might think it was impossible for anything to go faster. Fun fact, there is no historical evidence of an adult human ever dying from a cheetah attack. They are naturally shy, and a lot smaller than you may realize.
Yep. That said, unlike cheetahs, there’s plenty of evidence of leopards attacking humans. And these days, it’s the leopards, the closed-AI types and misanthropes -- telling everyone, “AI will take your job and there’s nothing you can do about it.”
Re: Superintelligence: The Idea That Eats Smart People (2016)
#13In the big 2026, everything certain people worried about with superintelligence came to fruition and they were vindicated. The people closest to ASI are indicating recursive self improvement is imminent, the smartest engineers in the labs themselves are autonomously using agents to develop and improve the models. The arms race is evident. NVDA is the world's most valuable company determined by the worlds' collective…
Nobody cares what we (people who have been working on AGI a long time) think.
We were dismissed as cranks before and now we’re just ignored by whomever is promising the most money to investors.
So, par for the course. Everyone in AI has lived through all the cycles so far so this is just the biggest one yet.
Re: Superintelligence: The Idea That Eats Smart People (2016)
#14Has anyone played SOMA? Spoiler warning. It explores this idea of, what if there's an AI in charge of ensuring mankind survives at all costs. What would it be willing to do, to keep us alive? Would we even recognize the result as human?
It's a horror game and it explores all kinds of fascinating and disturbing scenarios. Simulations of human minds. Artificial worlds. Human minds in robot bodies. Genetically modified humans. Man-machine hybrids etc.
(A great exploration of the substance/structure matrix, by the way. My favorite question in AI and consciousness. Is the special sauce in the material, or its shape, both, or neither?)
The very question of aligning the AI with humans assumes that we have a very robust definition of what human means in the first place.
Ostensibly the AI was aligned. It did succeed in keeping humans alive! But it did that in all sorts of ways that mostly made them wish it hadn't.
Re: Superintelligence: The Idea That Eats Smart People (2016)
#15> If you encountered a cheetah in pre-industrial times (and survived the meeting), you might think it was impossible for anything to go faster. Fun fact, there is no historical evidence of an adult human ever dying from a cheetah attack. They are naturally shy, and a lot smaller than you may realize.
There's a story about some Kenyans outrunning a Cheetah in 6km. It had been killing their livestock, so they decided to go after it.
Cheetahs are very fast, but humans have way more endurance.
Re: Superintelligence: The Idea That Eats Smart People (2016)
#16Some of the themes remind me of themes mentioned in this matrix analysis. Specifically I am reminded of the Dune concept of control: "you control what you can destroy" and then asking "do you control your refrigerator?". Sure, you can turn it off but then your food would rot and you might starve. So in a real sense humans have not controlled machines for a long time but have been co evolving in symbiosis. Sure, it's…
Could you elaborate on that last part?
I didn't mean it as in changing our framing enables technological progress but something we should do if we don't want to lose the control we have. e.g. if we lose all principle and intention then it doesn't really matter what happens with computers. In order to do something with intention we must first understand what we're doing. Skipping that step is an admission of defeat.
Re: Superintelligence: The Idea That Eats Smart People (2016)
#17It's amusing to read people in the past writing about the prospect of superhuman intelligence. The real problems have turned out to be different. Sycophancy and hallucinations, which are part of being confidently wrong, remains a big problem. Needing square miles of data centers was an issue in 1950s science fiction, and disappeared by the 1980s. Yet now they're being built, with private funding and the prospect of p…
I predicted on this site in 2016 the massive social and economic impacts AGI would have and specifically when RL data loops are not available to anyone but major players:
https://news.ycombinator.com/item?id=12168228
I even wrote up a whole article that specifically called RL loop based development as the future:
https://medium.com/@andrewkemendo/the-ai-revolution-will-be-...
> Reinforcement Learning tasks rely on ridiculous amounts of data. Whereas with traditional software architecture, where you accomplish tasks through explicit task instruction, RL trains for tasks based on millions of tests through a reward system. Most importantly once you have trained it to some minimum level, if you deploy it correctly, then it should continue improving — so long as you bake feedback into the UX. Imagine that instead of telling excel what to do, you and every other user will have a conversation with excel, improving the system incrementally.
Re: Superintelligence: The Idea That Eats Smart People (2016)
#18> Hopefully you see the resemblance between this vision of AI and a genie from folklore. The AI is all-powerful and gives you what you ask for, but interprets everything in a super-literal way that you end up regretting.
The monkey's paw. You know, you don't need superintelligence for that.
Civilization was already doing this. "What if we just gave ourselves exactly what we wanted." Well, it turns out often that's not so good!
Re: Superintelligence: The Idea That Eats Smart People (2016)
#19"Recursive self-improvement" is in the same league as "perpetual motion". What would be a way to recursively self-improve algorithms for matrix multiplication (foundations of machine learning and inference)?
It’s not advances on the underlying operation of matrix multiplication that have driven ai progress to date. It’s the layers above that; trying different neural architectures (transformers w/attention mechanisms), and also different data and training regimes (different ways of doing reinforcement learning) that are the main drivers of improved performance. Perpetual motion is a physical impossibility. Whereas Ai is already being used to improve the workflow of ai researchers, thus speeding up improvements in said research. It’s not hard to see that AI could well be spun up to continue to try new arrangements of the aforementioned levers that drive ai progress on its own.
Re: Superintelligence: The Idea That Eats Smart People (2016)
#20Has anyone played SOMA? Spoiler warning. It explores this idea of, what if there's an AI in charge of ensuring mankind survives at all costs. What would it be willing to do, to keep us alive? Would we even recognize the result as human? It's a horror game and it explores all kinds of fascinating and disturbing scenarios. Simulations of human minds. Artificial worlds. Human minds in robot bodies. Genetically modified…
The same theme was present in season 2 of Raised By Wolves. (RIP my favorite show in some years)