Live data from Hacker News

What if the singularity lies beyond a plateau we cannot cross?

jasonwillems.com

21–29 of 29 posts

Re: What if the singularity lies beyond a plateau we cannot cross?

#21

> "the cultural zeitgeist has centered on the idea of runaway progress: artificial intelligence recursively improving itself until we become subservient to it . far more plausible .. a world where the curve flattens and our rate of progress slows to a crawl." > "our mission should remain the same: accelerate to the maximum extent possible." I think you need to justify why hurrying to be AI servants should be our miss…

You're right, I should have tied that back to the opening.

The acceleration we've experienced has allowed us to "outrun" our problems. In earlier generations, that meant famine or disease. Today, it might be climate change. Tomorrow, it'll be something else entirely.

Technological progress has generally been the reason humanity should be optimistic against challenges: it gives us ever improving tools to solve our hardest problems faster than we succumb to them. Without it, that optimism becomes much harder to justify.

Even if there is a plateau we can't cross, if we believe we drive more benefit from technology than the problems it creates, it makes sense to extract as much progress as we can from the physics we have.

Re: What if the singularity lies beyond a plateau we cannot cross?

#22
post #18

Earlier quoted context omitted.

True, semi scaling is slowing down. But our software could be orders of magnitude more efficient. Am I wrong?

Yes, both software improvements and tailored hardware will continue to pay dividends (huge gains from TPUs, chips built specifically for inference, etc, even if the underlying process node is unchanged). Slowing transistor scaling just gives us one less domain through which to depend on for improvements - the others are all still valid, and will probably be something we come to invest more effort into.

More over there are probably lots of avenues we haven't even attempted because when the node scales down we get quadratic benefits (right?).

Where as tailored hardware, software improvements are unlikely to continue yielding such payoff again and again.

So the argument that cost of improvements will go up is not wrong. And maybe improvements will be more linear than exponential.

We also don't know that current semi tech stack is the best. But it's fair to argue that the cost of moving off a local optimum to a completely different technology stack would be wild.

Re: What if the singularity lies beyond a plateau we cannot cross?

#23
post #7

Earlier quoted context omitted.

I think there is still a lot we can do within the current paradigm - most software, especially for enterprise, is still quite bad. And that will continue to drive employment and growth. But w may one day have to contend with expecting fewer "new" paradigms and the ultra rapid industry growth that accompanies them (dotcom, SaaS, ML, etc). Will "software eating the world" be enough to counteract this long term? Hard to…

If the clock speed improvements had happened over a much longer stretch of time then we probably would have seen much earlier multi-core capable tooling. We are still mostly optimized for single threaded applications, extracting the maximum from a CPU is really hard work. So also consider the backlog in tooling, there is so much work that still needs to be done there.

And so much of our tech stack runs on old abstractions.

Do you even need an MMU, if you have memory safe languages?

Honestly, maybe semi scaling is a problem for AI, but for most other software today, the problem is bugs, bloat, latency.

Re: What if the singularity lies beyond a plateau we cannot cross?

#24

Explain to me how the, "Technological Singularity" isn't just Christian eschatology for dorks? As a Neon Genesis Evangelion fan this really gets me going but that's kind of why I ask.

I tend to dislike the term AGI/ASI, since it's become a marketing label more than a coherent concept (which everyone will define differently) In this case I use "singularity", by which I mean it more abstractly: a hypothetical point where technological progress begins to accelerate recursively, with heavily reduced human intervention. My point isn't theological or utopian, just that the physical limits of computation…

Thanks for the coherent response jayw_lead! I too dislike the term but I'm coming from the Noam Chomsky, "language models are a nothingburger in terms of applied linguistics" angle. As far as I can see this still seems like , "secular post-Enlightenment science culture dreams up the Millennium" and given Peter Thiel's recent commentary about the, "anti-christ" I'm terrified to the point of thinking it would be wise to buy an assault rifle.

https://wiki.evageeks.org/S%C2%B2_Engine

Re: What if the singularity lies beyond a plateau we cannot cross?

#25
The plateau you speak of is just inevitable human evolution. There will come a point where we will start to edit our genome and see what's capable of. Before we get there, though -

In time, AI and VR/AR will converge to allow us to evolve new ways to educate/entertain new generations, to distill knowledge in a faster and much more reliable way. We will experience societal upheavals before the "plateau". Our current world order will probably experience major changes.

AGI will probably be a long, long way ahead of us -- in its current state, LLMs are not going to spontaneously develop sentience. There is a massive scale (power, resources, space, etc) issue to contend with.

Re: What if the singularity lies beyond a plateau we cannot cross?

#26

The plateau you speak of is just inevitable human evolution. There will come a point where we will start to edit our genome and see what's capable of. Before we get there, though - In time, AI and VR/AR will converge to allow us to evolve new ways to educate/entertain new generations, to distill knowledge in a faster and much more reliable way. We will experience societal upheavals before the "plateau". Our current w…

I doubt that editing our genome will ever accomplish much. Maybe some minor improvements here and there, primarily with reducing the incidence of certain diseases. But there's no free lunch in biology and every advantage comes with disadvantages. For example, some people want to be taller but they don't realize that increases the risk of back problems later in life.

Most people don't even get close to their genetic limits anyway. We're capable of much more than we realize but we fall into mental traps and fool ourselves into thinking we're incapable. Some guy just set a world record by holding his breath for 29 minutes: a few years ago most people would have said that's impossible.

https://www.uow.edu.au/media/2025/the-science-behind-a-freed...

Re: What if the singularity lies beyond a plateau we cannot cross?

#27
post #23
post #7

Earlier quoted context omitted.

If the clock speed improvements had happened over a much longer stretch of time then we probably would have seen much earlier multi-core capable tooling. We are still mostly optimized for single threaded applications, extracting the maximum from a CPU is really hard work. So also consider the backlog in tooling, there is so much work that still needs to be done there.

And so much of our tech stack runs on old abstractions. Do you even need an MMU, if you have memory safe languages? Honestly, maybe semi scaling is a problem for AI, but for most other software today, the problem is bugs, bloat, latency.

MMUs allow for overcommit and make it possible to dynamically allocate memory to applications that need it.

Re: What if the singularity lies beyond a plateau we cannot cross?

#28

> "the cultural zeitgeist has centered on the idea of runaway progress: artificial intelligence recursively improving itself until we become subservient to it . far more plausible .. a world where the curve flattens and our rate of progress slows to a crawl." > "our mission should remain the same: accelerate to the maximum extent possible." I think you need to justify why hurrying to be AI servants should be our miss…

You're right, I should have tied that back to the opening. The acceleration we've experienced has allowed us to "outrun" our problems. In earlier generations, that meant famine or disease. Today, it might be climate change. Tomorrow, it'll be something else entirely. Technological progress has generally been the reason humanity should be optimistic against challenges: it gives us ever improving tools to solve our har…

AI training helps as to fight the climate change? If anything until now the opposite has been true.

Re: What if the singularity lies beyond a plateau we cannot cross?

#29

These aren't very convincing arguments; why don't aircraft factories of the 1930s or the ocean-going steamship dockyards of the 1830s count under "massive upfront investment and large and complex" and therefore predict progress stopping ages ago? Asimov's story The Last Question ends with the Multivac machine having collected all the data in the universe and still not answering the question "how can entropy be revers…

> "massive upfront investment and large and complex" and therefore predict progress stopping ages ago? Regulatory and economic barriers are probably the easiest to overcome. But they are an obstacle. All it takes is for public sentiment to turn a bit more hostile towards technology, and progress can stall indefinitely. > Opening with the recursively improving AGI and then having a section of "areas of promise for ste…

Your strongest case is that CPU design might stop getting faster, but that doesn't automatically mean progress towards AGI must stop.

Smartphones sell 1.2 billion every year. Add in server, laptop, embedded chips, GPUs, TPUS - even if transistor density and process improvements stall soon, the amount of compute power on Earth is vast and increasing rapidly until 'soon' happens, and can still increase rapidly by churning out more compute with the best available process indefinitely after that. You haven't made a case that process improvements are necessary or that they are not going to happen, you've only said that they might be neccessary and might not happen. "All it takes is for public sentiment to turn a bit more hostile towards technology, and progress can stall indefinitely" - again, true in terms of process improvements, not true in general because we could potentially make progress towards AGI with existing compute power by changing how we organise it and what we do with it; at least, you haven't given a good reason why that couldn't happen.

> "In fact, one of the strongest counterpoints to the article's thesis is molecular biology, which has a remarkable amount of momentum and a lot of room left to run."

One of the strongest counterpoints is that human brains exist - there's definitely some way to get to human-equivalent intelligence, on Earth, within the laws of Physics, the energy and temperature constraints which exist here. Handwaving "what if AGI is impossible because of the laws of physics" needs you to make a case why the laws of physics are a blocker, not just state that there are some physical limits to some things sometimes.

Yes transistors are hard to make smaller but that is not the only option - we've developed stacked 3D transistors to pack more into a small volume, hardware acceleration through design for more and more algorithms (compression, video compression, encryption) to better use existing transistor budgets, we've developed better more cache-friendly and SIMD friendly algorithms, chips where part of them can power down to allow more power for other parts without hitting thermal limits. More than one S curve involved, not just one plateau.

Post reply on HN