Live data from Hacker News

We were wrong about GPUs

fly.io

121–130 of 604 posts

Re: We were wrong about GPUs

#121

I have a timeline that I am still trying to work through but it goes like this : 2012 - moores law basically ends - nand gates do t get smaller just more cleverly wrapped. Single threaded execution more or less stops at 2 GHz and has remained there. 2012-2022 - no one notices single threaded is stalled because everything moves to VMs in the cloud - the excess parallel compute from each generation is just shared out i…

The unsung hero of early computing was Dennard scaling. Taking CPUS from 10MHz to 2GHz, all alongside massive per-clock efficiency improvements must have been a crazy time.

From a 50MHz 486 in 1990 to a 1.4GHz P3 in 2000 is a factor of 28 improvement in speed solely due to clock speed! Add on all the other multiplicative improvements from IPC...

Re: We were wrong about GPUs

#122

I really liked playing around with fly gpus, but it's just too expensive for hobby-use. Same goes for the rest of fly.io honestly. The DX is great and I wish I could move all of my homelab stuff and public websites to it, but it'd be way too expensive :(

Who do you use instead for hobby projects?

Re: We were wrong about GPUs

#123

Earlier quoted context omitted.

This is a false dichotomy. The truth is we are constantly moving further and further away from the silicon. New developers don't have as much need to understand these details because things just work; some do care because they work at a job where it's required, or because they're inherently interested (a small number). Over time we will move further away. If the cost of an easily managed solution is low enough, why d…

> The truth is we are constantly moving further and further away from the silicon. Are we? We're constantly changing abstractions, but we don't keep adding them all that often. Operating systems and high-level programming languages emerged in the 1960s. Since then, the only fundamentally new layer of abstraction were virtual machines (JVM, browser JS, hardware virtualization, etc). There's still plenty of hardware-sp…

I don’t understand how you could say something like HTTP or Cloud Functions or React aren’t abstractions that software developers take for granted.

Re: We were wrong about GPUs

#124
post #85
post #3

I spent a month setting up serverless endpoint for a custom model last year with Runpod. It was expensive and unreliable, in addition to long cold boot times. The product was unusable even as a prototype, to cover the costs, I'd have to raise money first. In a different product, I was given some Google Cloud credits, which unlocked me to put the product in front of customer. This one also needed GPU but not as expens…

Give https://modal.com a spin -- email me at deven [at] modal.com and happy to help you get set up

your fancy scrolling animation splash screen is very laggy on my M1 MBP, incase that is interesting to you.

Re: We were wrong about GPUs

#125

Earlier quoted context omitted.

I used to think this, ut it only works if the abstractions hold - it’s like if we stopped random access memory and went back to tape drives suddenly abstractions matter. My comment elsewhere goes into but more detail but basically silicon stopped being able to make single threaded code faster in about 2012 - we just have been getting “more parallel cores” since. And now at wafer scale we see 900,000 cores on a “chip”…

That's why abstractions like PyTorch exist. You can write a few lines of Python and get good utilization of all those GPU cores.

[dead]

Re: We were wrong about GPUs

#127

> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…

This is a false dichotomy. The truth is we are constantly moving further and further away from the silicon. New developers don't have as much need to understand these details because things just work; some do care because they work at a job where it's required, or because they're inherently interested (a small number). Over time we will move further away. If the cost of an easily managed solution is low enough, why d…

> because things just work

If that were true, you might be right.

What happens in reality is that things are promised to work and (at best) fulfill that promise so long as no developers or deployers or underlying systems or users deviate from a narrow golden path, but fail in befuddling ways when any of those constraints introduce a deviation.

And so what we see, year over year, is continued enshittening, with everything continuously pushing the boundaries of unreliability and inefficiency, and fewer and fewer people qualified to actually dig into the details to understand how these systems work, how to diagnose their issues, how to repair them, or how to explain their costs.

> If the cost of an easily managed solution is low enough, why do the details matter?

Because the patience that users have for degraded quality, and the luxury that budgets have for inefficiency, will eventually be exhausted and we'll have collectively led ourselves into a dark forest nobody has the tools or knowledge to navigate out of anymore.

Leveraging abstractions and assembling things from components are good things that enable rapid exploration and growth, but they come with latent costs that eventually need to be revisited. If enough attention isn't paid too understanding, maintaining, refining, and innovating on the lowest levels, the contraptions built through high-level abstraction and assempbly will eventually either collapse upon themselves or be flanked by competitors who struck a better balance and built on more refined and informed foundations.

As a software engineer who wants a long and satisfying career, you should be seeking to understand your systems to as much depth as you can, making informed, contextual choices about what abstractions you leverage, exactly what they abstract over, and what vulnerabilities and limitations are absorbed into your projects by using them. Just making naive use of the things you found a tutorial for, or that are trending, or that make things look easy today, is a poison to your career.

Re: We were wrong about GPUs

#128

Earlier quoted context omitted.

This is a false dichotomy. The truth is we are constantly moving further and further away from the silicon. New developers don't have as much need to understand these details because things just work; some do care because they work at a job where it's required, or because they're inherently interested (a small number). Over time we will move further away. If the cost of an easily managed solution is low enough, why d…

> The truth is we are constantly moving further and further away from the silicon. Are we? We're constantly changing abstractions, but we don't keep adding them all that often. Operating systems and high-level programming languages emerged in the 1960s. Since then, the only fundamentally new layer of abstraction were virtual machines (JVM, browser JS, hardware virtualization, etc). There's still plenty of hardware-sp…

Maybe fundamentally is an extremely load bearing word here, but just in the hardware itself we see far more abstraction than we saw in the 60s. The difference between what we called microcode in an 8086 and what is running in any processor you buy in 2025 is an abyss. It almost seems like hardware emulation. I could argue that the layers of memory caching that modern hardware have are themselves another layer vs the days when we sent instructions to change which memory banks to read. The fact that some addresses are very cheap and others are not, and the complexity is handled in hardware is very different than stashing data in extra registers we didn't need this loop. The virtualization any OS does for us is much deeper than even a big mainframe that was really running a dozen things at once. It only doesn't look like additional layers if you look from a mile away.

The majority of software today is written without knowing even which architecture the processor is going to be, how much of the processor we are going to have, whether anything will ever fit in memory... hell, we can write code that doesn't know not just the virtual machine it's going to run in, but even the family of virtual machine. I have written code that had no idea if it was running in a JVM, LLVM or a browser!

So when I compare my code from the 80s to what I wrote this morning, the distance from the hardware doesn't seem even remotely similar. I bet someone is writing hardware specific bits somewhere, and that maybe someone's debugging assembly might actually resemble what the hardware runs, maybe. But the vast majority of code is completely detached from anything.

Re: We were wrong about GPUs

#129

Earlier quoted context omitted.

This is a false dichotomy. The truth is we are constantly moving further and further away from the silicon. New developers don't have as much need to understand these details because things just work; some do care because they work at a job where it's required, or because they're inherently interested (a small number). Over time we will move further away. If the cost of an easily managed solution is low enough, why d…

> The truth is we are constantly moving further and further away from the silicon. Are we? We're constantly changing abstractions, but we don't keep adding them all that often. Operating systems and high-level programming languages emerged in the 1960s. Since then, the only fundamentally new layer of abstraction were virtual machines (JVM, browser JS, hardware virtualization, etc). There's still plenty of hardware-sp…

The abstraction manifest more on the language level. No memory management, simplified synchronization primitives, no need of compilation.

Not sure virtual machine are fundamentally different. In the end if you have 3 virtual or 3 physical machine the most important difference is how fast you can change their configuration. They will still have all the other concepts (network, storage, etc.). The automation that comes with VM-s is better than it was for physical (probably), but then automation for everything got better (not only for machines).

Re: We were wrong about GPUs

#130
post #87

Earlier quoted context omitted.

Plenty of non-IT applications use lots of cores, e.g. physics simulations, constraint solving, network simulation used to plan roads or electrical distribution, etc.

Yes - but the amount of code that loves Amdahls law, is tiny compared to amount code churned out each day that can never run parallel over 1M cores - no matter how clever a compiler gets. I cannot work out if we pack enough parallel problems in the world or just lack a programming language to describe them

Computers are still Von-Neumann machines, and other architectures lost out due to the great returns on investment for that architecture. However, in the AI world, this might not be the case. For instance, neuromorphic computing is one example, and there are others. Or back to analog again! Superposition is instant—no slow adders with carry bits to propagate! Who knows. Fun times!
Post reply on HN