Live data from Hacker News

We were wrong about GPUs

fly.io

171–180 of 604 posts

Re: We were wrong about GPUs

#171

> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…

> they're willing to spend a lot of (usually their employer's) money

May just be my naïveté, but I thought that something like ECS or EKS is much cheaper than an in-house k8 engineer.

Re: We were wrong about GPUs

#172
post #152

Earlier quoted context omitted.

> And when I'm not running LLMs, the memory is useful for other stuff Let's be honest, the other stuff is just Chrome: Tell me 96gb is enough?

Firefox and VS Code for me: I have 64GB and I can't run Llama 70B locally without closing a ton of windows and tabs first!

Have you tried auto tab discard?

Re: We were wrong about GPUs

#173

Earlier quoted context omitted.

> There's an (increasingly small) group of software developers who don't like "magic" and want to understand where their code is running and what it's doing. These developers gravitate toward open source solutions like Kubernetes Kubernetes is not the first thing that comes to mind when I think of "understanding where their code is running and what it's doing"...

Really? There are plenty of valid criticisms of kubernetes, but this doesn't strike me as one of them. It gives you tons of control over all of this. That's a big part of why it's so complex!

This mainframe system from the 1990s was so much simpler

https://www.ibm.com/docs/en/cics-ts/6.x?topic=sysplex-parall...

even if it wasn't as scalable as Kube. One the other hand, a cluster of 32 CMOS mainframe could handle any commercial computing job that people were doing in the 1990s.

Re: We were wrong about GPUs

#174

Earlier quoted context omitted.

This is a false dichotomy. The truth is we are constantly moving further and further away from the silicon. New developers don't have as much need to understand these details because things just work; some do care because they work at a job where it's required, or because they're inherently interested (a small number). Over time we will move further away. If the cost of an easily managed solution is low enough, why d…

> The truth is we are constantly moving further and further away from the silicon. Are we? We're constantly changing abstractions, but we don't keep adding them all that often. Operating systems and high-level programming languages emerged in the 1960s. Since then, the only fundamentally new layer of abstraction were virtual machines (JVM, browser JS, hardware virtualization, etc). There's still plenty of hardware-sp…

At the company I work for, I routinely mock the software devs for solving every problem by adding yet another layer of abstraction. The piles of abstractions these people levy is mind numbingly absurd. Half the things they are fixing, if not more, are created by the abstractions in the first place.

Re: We were wrong about GPUs

#175
post #36
post #33

Earlier quoted context omitted.

You just wrote a blog post about whats needed for a top HN post. You should have known it would do well :) https://fly.io/blog/a-blog-if-kept/

I'm a little sore the FOIA thing didn't make it up here, but Kurt's post did. ;)

Got a link? I flipped through the last dozen or so blog posts and none of them hit on a search for "FOIA".

Re: We were wrong about GPUs

#176
> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs.

I don't want GPUs, but that's not quite the reason:

- The SOTA for most use cases for most classes of models with smallish inputs is fast enough and more cost efficient on a CPU.

- With medium inputs, the GPU often wins out, but costs are high enough that a 10x markup isn't worth it, especially since the costs are still often low compared to networking and whatnot. Factor in engineer hours and these higher-priced machines, and the total cost of a CPU solution is often still lower (always more debuggable).

- For large inputs/models, the GPU definitely wins, but now the costs are at a scale that a 10x markup is untenable. It's cheaper to build your own cluster or pay engineers to hack around the deficits of a larger, hosted LLM.

- For xlarge models™ (fuzzily defined to be anything substantially bigger than the current SOTA), GPUs are fundamentally the wrong abstraction. We _can_ keep pushing in the current directions (transformers requiring O(params * seq^2) work, pseudo-transformers requiring O(params * seq) work but with a hidden, always-activated state space buried in that `params` term which has to increase nearly linearly in size to attain the same accuracy with longer sequences, ...), but the cost of doing so is exorbitant. If you look at what's provably required to do those sorts of computations, the "chuck it in a big slice of vRAM and do everything in parallel" strategy gets more expensive compared to theoretical optimality as model size increases.

I've rented a lot of GPUs. I'll probably continue to do so in the future. It's a small fraction of my overall spending though. There aren't many products I can envision which could be built on rented GPUs more efficiently than rented CPUs or in-house GPUs.

Re: We were wrong about GPUs

#177

I have a timeline that I am still trying to work through but it goes like this : 2012 - moores law basically ends - nand gates do t get smaller just more cleverly wrapped. Single threaded execution more or less stops at 2 GHz and has remained there. 2012-2022 - no one notices single threaded is stalled because everything moves to VMs in the cloud - the excess parallel compute from each generation is just shared out i…

>2012 - moores law basically ends - nand gates do t get smaller just more cleverly wrapped. Single threaded execution more or less stops at 2 GHz and has remained there. In 2012, 22nm architectures were new ( https://en.wikipedia.org/wiki/List_of_Intel_CPU_microarchite... ). Now we have 3nm architectures ( https://en.wikipedia.org/wiki/3_nm_process ). In what sense have nand gates "not gotten smaller"? My computer wa…

> In what sense have nand gates "not gotten smaller"?

Because 22nm was not actually 22nm, and 3nm is not actually 3nm.

Re: We were wrong about GPUs

#179

> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…

> There's an (increasingly small) group of software developers who don't like "magic" and want to understand where their code is running and what it's doing. These developers gravitate toward open source solutions like Kubernetes Kubernetes is not the first thing that comes to mind when I think of "understanding where their code is running and what it's doing"...

Indeed, I have to wonder how many people actually understand Kubernetes. Not just as a “user” but exactly all what it is doing behind the scenes…

Just an “idle” Kubernetes system is a behemoth to comprehend…

Re: We were wrong about GPUs

#180

I respect them for being public about this. With that said, this seems quite obvious - the type of customer that chooses Fly, seems like the last person to be spinning up dedicated GPU servers for extended periods of time. Seems much more likely they'll use something serverless which requires a ton of DX work to get right (personally I think Modal is killing it here). To compete, they would have needed to bet the com…

As someone who deploys a lot of models on rented GPU hardware, their pricing is not realistic for continous usage. They're charging hyperscaler rates, and anyone willing to pay that much won't go with Fly. For serverless usage they're only mildly overpriced compared to say Runpod, but I don't think of serverless as anything more than an onramp to renting dedicated machine, so it's not surprising to hear it's not taki…

> GPU workloads tend to have terrible cold-start performance by their nature

My Fly machine loads from turned off to first inference complete in about 35 seconds.

If it’s already running, it’s 15 seconds to complete. I think that’s pretty decent.

Post reply on HN