Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We were wrong about GPUs

fly.io

Post reply on HN
Print view View original

301–310 of 604 posts

Re: We were wrong about GPUs

#301

Post by bottega_boy »

ryuuseijin wrote: post #158

My heart stopped for a moment when reading the title. I'm glad they haven't decided to axe GPUs, because fly GPU machines are FANTASTIC! Extremely fast to start on-demand, reliable and although a little bit pricy but not unreasonably so considering the alternatives. And the DX is amazing! it's just like any other fly machine, no new set of commands to learn. Deploy, logs, metrics, everything just works out of the box…

I used the fly.io GPUs as development machines. For that, I generally launch a machine when I need it and scale it to 0 when I am finished. And this is what's really fantastic about fly.io - setting this up takes an hour... and the Dockerfile created in the process can also be used on any other machine. Here's a project where I used this setup: https://github.com/li-il-li/rl-enzyme-engineering

This is in stark contrast to all other options I tried (AWS, GCP, LambdaLabs). The fly.io config really felt like something worth being in every project of mine and I had a few occasions where I was able to tell people to sign up at fly.io and just run it right there (Btw. signing up for GPUs always included writing an email to them, which I think was a bit momentum-killing for some people).

In my experience, the only real minor flaw was the already mentioned embedding of the whole CUDA stack into your container, which creates containers that approach 8GB easily. This then lets you hit some fly.io limits as well as creating slow build times.

Re: We were wrong about GPUs

#302

Post by arijo »

nicoburns wrote: post #112
freedomben wrote: post #66

> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…

> There's an (increasingly small) group of software developers who don't like "magic" and want to understand where their code is running and what it's doing. These developers gravitate toward open source solutions like Kubernetes Kubernetes is not the first thing that comes to mind when I think of "understanding where their code is running and what it's doing"...

Great opportunity for someone ballsy to write a book about kubernetes internals for the general engineering population.

Bonus points for writing a basic implementation from first principles capturing the essence of the problem kubernetes really was meant to solve.

The 100 pages kubernetes book, Andriy Burkov style.

Re: We were wrong about GPUs

#303

Post by lifeisstillgood »

zahlman wrote: post #143
lifeisstillgood wrote: post #83

I have a timeline that I am still trying to work through but it goes like this : 2012 - moores law basically ends - nand gates do t get smaller just more cleverly wrapped. Single threaded execution more or less stops at 2 GHz and has remained there. 2012-2022 - no one notices single threaded is stalled because everything moves to VMs in the cloud - the excess parallel compute from each generation is just shared out i…

>2012 - moores law basically ends - nand gates do t get smaller just more cleverly wrapped. Single threaded execution more or less stops at 2 GHz and has remained there. In 2012, 22nm architectures were new ( https://en.wikipedia.org/wiki/List_of_Intel_CPU_microarchite... ). Now we have 3nm architectures ( https://en.wikipedia.org/wiki/3_nm_process ). In what sense have nand gates "not gotten smaller"? My computer wa…

So the size of nand gates is still roughly 28nm, but if you measure a car from above its size is fixed, if you stand the car on its nose you can measure it from above and it’s “smaller”, this is FinFet, then fold down the roof and the wheels and that’s roughly GAA. The car size stays the same - the parking density increases. It’s more marketing than reality but density is up …

As for clock speeds, yes and no, basically thermal limits stop most CPUs running full time at full speed - the problem was obvious back in the day - I would build PCs and carefully apply thermal paste to the plastic casing of a chip - thus waiting for the heat of the transistors to heat up the plastic to remove the waste. Yes they are working on thermal something something directly on the layers of silicon.

Re: We were wrong about GPUs

#305

Post by taeric »

I was at another team making a similar bet. Felt off to me at the time, but I assumed I just didn't understand the market.

I also think the call that people want LLMs is slightly off. More correct to say people want a black box that gives answers. LLMs have the advantage that nobody really knows anything about tuning them. So, it is largely a raw power race.

Taking it back to ML, folks would love a high level interface that "just worked." Dealing with the GPUs is not that, though.

Re: We were wrong about GPUs

#306

Post by raziel2p »

Rury wrote: post #219
y1n0 wrote: post #174

Earlier quoted context omitted.

At the company I work for, I routinely mock the software devs for solving every problem by adding yet another layer of abstraction. The piles of abstractions these people levy is mind numbingly absurd. Half the things they are fixing, if not more, are created by the abstractions in the first place.

Yeah, I remember watching a video of (I think?) a European professor who helped with an issue devs were having in developing The Witness. Turns out they had a large algorithm they developed in high level code (~2000 lines of code? can't remember) to place flora in the game world, which took minutes to process, and it was hampering productivity. He looked at it all, and redid almost all of it in something like Frankly…

what's your point exactly? what do you hope to achieve by "bringing it up" (I assume in your workplace)?

most programmers are not able to solve a problem like that in 20 lines of assembly or whatever, and no amount of education or awareness is going to change that. acting as if they can is just going to come across as arrogant.

Re: We were wrong about GPUs

#307

Post by lifeisstillgood »

lifeisstillgood wrote: post #83

I have a timeline that I am still trying to work through but it goes like this : 2012 - moores law basically ends - nand gates do t get smaller just more cleverly wrapped. Single threaded execution more or less stops at 2 GHz and has remained there. 2012-2022 - no one notices single threaded is stalled because everything moves to VMs in the cloud - the excess parallel compute from each generation is just shared out i…

Edit: it’s worth expanding on the data centre costs issues - if it’s fair to say we have stalled on “free”speed ups for standard software (ie C/Linux) - that is clock speeds more or less stopped and we get more 64 bit cores but each core is more or less no faster (hand wavy), then the amount of clients that can be packed into a data centre stays the same - you are renting out CPUs in a Docker VM - that’s basically one per Core(#). And while wafer scale gives you 900,000 cores in a 1u server, normal CPUs gives you what 64? 128?

Suddenly your cost for building a new data centre is something like twice the cost of the previous one (cold gets more expensive etc) and yet you only sell same amount of space. It’s not an attractive business in first place.

This was the push for lambda architecture etc - depending on usage you could have hundreds of people buying the same core. I would have put a lot of cash into making something that spins up docker instances so fast it’s like lambda - and guess what fly.io does?

I think fly.io’s obsession with engineering led them down a path of positive capital usage while AWS focused on rolling out new products on a tougher capital process.

Anyway - AI is the only thing that dense multi core data centres can use that packs in many many users compared to docker per core.

Unless we all learn how to think and code in parallel, we are leaving a huge amount of hardware gains on the table. And those gains are going to 100x in the next ten years and my bet is my janky-ass software will still not be able to use it - there will be a bifurcation of specialist software engineers who work in domains and with tooling that is embarrassingly parallel, and the rest of us will be on fly.io :-)

(#) ok so maybe 3 or 4 docker per core, with hyper visor doling out time slots, but much more than that and performance is a dog, and so the number of “virtual CPUs” you can sell is a limited number and creeps up despite hardware leaping ahead … the point I am making

Re: We were wrong about GPUs

#308

Post by zeroq »

never_inline wrote: post #296
nicoburns wrote: post #112

Earlier quoted context omitted.

> There's an (increasingly small) group of software developers who don't like "magic" and want to understand where their code is running and what it's doing. These developers gravitate toward open source solutions like Kubernetes Kubernetes is not the first thing that comes to mind when I think of "understanding where their code is running and what it's doing"...

Core kubernetes (deployments, services etc..) is fairly easy to understand. lot of other stuff in the cncf ecosystem is immature. I don't think most people need to use all the operators, admission controllers, otel, service mesh though. If you're running one team with all services trusting each other, you don't have problems solved by these things. Whenever you introduce a CNCF component outside core kubernetes, inve…

Having a team that runs the kubernetes for you and being on receiving end is indeed super easy. Need another microservice? Just add another repository, add short yaml, push it to CI and bam!, it's online.

But setting it up is not a trivial task and often a recipe for disaster.

I've seen a fair share of startups who took too much kool aid and wanted parrot FANG stacks just to discover they are burning tons of money just trying to deploy their first hello world application.

Re: We were wrong about GPUs

#309

Post by darkwater »

nicoburns wrote: post #250
igmor wrote: post #245

Earlier quoted context omitted.

These components are very different in complexity and scope. Let's be real: a seasoned developer is mostly familiar with load balancers and ingress controllers, so this will be mostly about naming and context. I agree though once you learn about k8s it becomes less mysterious but that also means the author hasn't pushed it to the limits. Outages in the control plane could be pretty nasty and it is easy to have them b…

A really simple setup for many smaller organisations wouldn't have a load balancer at all.

No load balancer means... entering one node only? Doing DNS RR over all the nodes? If you don't have a load balancer in front, why are you even using Kubernetes? Deploy a single VM and call it a day!

I mean, in my homelab I do have Kubernetes and no LB in front, but it's a homelab for fun and learn K8s internals. But in a professional environment...

Re: We were wrong about GPUs

#310

Post by mschuster91 »

nicoburns wrote: post #112
freedomben wrote: post #66

> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…

> There's an (increasingly small) group of software developers who don't like "magic" and want to understand where their code is running and what it's doing. These developers gravitate toward open source solutions like Kubernetes Kubernetes is not the first thing that comes to mind when I think of "understanding where their code is running and what it's doing"...

> Kubernetes is not the first thing that comes to mind when I think of "understanding where their code is running and what it's doing"...

In the end it's a scheduler for Docker containers on a bunch of virtual or bare metal machines. Once you get that in your head life becomes much more easy.

The only thing I'd really love to see from an ops perspective is a way to force-revive crashed containers for debugging. Yes, one shouldn't have to debug cattle, just haul the carcass off and get a new one... but I still prefer to know why the cattle died.

Post reply on HN